apache pig - How to allocate specific number of mappers to multiple job in Hadoop? -


i executing multiple pig scripts script1, script2, script3, script4. in script1 executing independently , script2,3,4 executing parallely after scripts executed. giving input file of size 7-8 gb. after executing script1, observing instead of parallely executing script 2,3,4 script2 executing consuming 33-35 mappers. other remain in queue (means script3,4 have not mapper allocation). due time requires execute scripts.

so thinking if able set limit of mapper each script may time require execute wll less scripts may allocation of mappers.

so there way allocate specific number of mappers multiple scripts?

if map number correctly set (according core/node , disks/node values), having 1 job consuming maps or having n job consuming mapnumber / n maps have same result. if want distribute maps on amount of jobs can set per job map number (mapreduce.job.maps in mapred-site.xml think).

considering still have free map slots, there config enable jobs parallel executions discussed here : running jobs parallely in hadoop

you can set map number each job (even if not sure works) if provide job.xml in set map number hadoop command.


Comments

Popular posts from this blog

javascript - DIV "hiding" when changing dropdown value -

Does Firefox offer AppleScript support to get URL of windows? -

android - How to install packaged app on Firefox for mobile? -