From owner-mpi-comm@CS.UTK.EDU Mon Nov 23 16:08:35 1992 Return-Path: Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA26326; Mon, 23 Nov 92 15:50:10 -0500 Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26317; Mon, 23 Nov 92 15:50:05 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA06935; Mon, 23 Nov 92 15:50:04 -0500 Date: Mon, 23 Nov 92 15:50:04 -0500 Message-Id: <9211232050.AA06935@thud.cs.utk.edu> To: mpi-comm@cs.utk.edu Subject: mpi committees and mailing list Status: R The following mailing lists have been set up. mpi-comm@cs.utk.edu Whole committee mpi-intro@cs.utk.edu Introduction subcommittee mpi-pt2pt@cs.utk.edu Point-to-point communication subcommittee mpi-collcomm@cs.utk.edu Collective communication subcommittee mpi-ptop@cs.utk.edu Process topology subcommittee mpi-lang@cs.utk.edu Language binding subcommittee mpi-formal@cs.utk.edu Formal language description subcommittee mpi-envir@cs.utk.edu Environment inquiry subcommittee All mail will be collected and can be retrieved by sending email to netlib@ornl.gov and in the mail message typing: send mpi-comm from mpi send mpi-intro from mpi etc. Also try: send index from mpi I'm in the process of collecting information on how the HPF committee operates and will pass along the details in the next few days. I would like to suggest January 6 - 8 for the next meeting in Dallas at the airport hotel. We should plan on starting around 1:00 on January 6th and finishing around 3:00 on January 8th. Below is a list of the subcommittees as I have them from our meeting last week. Please let me know if you have any changes. Regards, Jack Introduction Subcommittee ---------------------- Jack Dongarra - Chair dongarra@cs.utk.edu Daniel Frye danielf@kgnvma.vnet.ibm.com Tony Hey ajgh@ecs.soton.ac.uk Rusty Lusk lusk@mcs.anl.gov Steve Zenith zenith@kai.com Point-To-Point Communication Subcommittee ----------------------------------------- Marc Snir - Chair snir@watson.ibm.com Scott Berryman berryman@cs.yale.edu Jack Dongarra dongarra@cs.utk.edu Al Geist geist@msr.epm.ornl.gov Adam Greenberg moose@think.com Bill Gropp gropp@mcs.anl.gov Leslie Hart hart@fsl.noaa.gov Rolf Hempel hempel@gmd.de Tom Henderson hender@fsl.noaa.gov Bob Knighten knighten@ssd.intel.com Oliver McBryan mcbryan@piper.cs.colorado.edu Peter Rigsbee par@cray.com David Walker walker@msr.epm.ornl.gov Collective Communication Subcommittee ------------------------------------- Al Geist - Chair geist@msr.epm.ornl.gov Leslie Hart hart@fsl.noaa.gov Jon Flowers jwf@parasoft.com Adam Greenberg moose@think.com Bob Knighten knighten@ssd.intel.com Steve Otto otto@cse.ogi.edu Peter Rigsbee par@cray.com Marc Snir snir@watson.ibm.com David Walker walker@msr.epm.ornl.gov Process Topology Subcommittee ----------------------------- Rolf Hempel - Chair hempel@gmd.de Jon Flowers jwf@parasoft.com Tom Henderson hender@fsl.noaa.gov Oliver McBryan mcbryan@piper.cs.colorado.edu Steve Otto otto@cse.ogi.edu Lew Tucker tucker@think.com Language Binding Subcommittee ----------------------------- Scott Berryman - Chair berryman@cs.yale.edu Bruce Leisure bleasure@kai.com Formal Language Description Subcommittee ---------------------------------------- Steve Zenith - Chair zenith@kai.com Tony Hey ajgh@cs.ston.ac.uk Rusty Lusk lusk@mcs.anl.gov Environment Inquiry Subcommittee --------------------------------- Bill Gropp - Chair gropp@mcs.anl.gov Daniel Frye danielf@kgnvma.vnet.ibm.com From owner-mpi-comm@CS.UTK.EDU Tue Nov 24 15:37:43 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02830; Tue, 24 Nov 92 15:37:43 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA19627; Tue, 24 Nov 92 15:12:24 -0500 Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19622; Tue, 24 Nov 92 15:12:21 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA25860; Tue, 24 Nov 92 14:12:19 CST From: gropp@antares.mcs.anl.gov (William Gropp) Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA11600; Tue, 24 Nov 92 14:12:17 CST Date: Tue, 24 Nov 92 14:12:17 CST Message-Id: <9211242012.AA11600@godzilla.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: Test Implementation of the Draft Standard We have completed a first and rough draft of a test implementation. We have included (and run!) three sample programs written in MPI on a sun network and on an Intel ipsc/i860. It is available by anonymous ftp from info.mcs.anl.gov in pub/mpi . The README file there gives more information, including how to get, install, and run the examples. This is a rough draft; the file mpi.man.ps.Z contains a description of the implementations architecture and man pages for all of the routines. We will be updating this implementation with additional examples and fewer restrictions soon. Enjoy! Bill Gropp and Rusty Lusk From owner-mpi-comm@CS.UTK.EDU Tue Nov 24 17:07:49 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06915; Tue, 24 Nov 92 17:07:49 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA21596; Tue, 24 Nov 92 16:45:52 -0500 Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21572; Tue, 24 Nov 92 16:45:38 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA21258; Tue, 24 Nov 92 16:45:35 -0500 Date: Tue, 24 Nov 92 16:45:35 -0500 Message-Id: <9211242145.AA21258@thud.cs.utk.edu> To: mpi-comm@cs.utk.edu Subject: working procedures Chuck Koelbel has passed on some notes from the HPF experence. I think we will find them useful in carring out our meetings. Jack HPFF Working Procedures HPFF never had an explicit vote on adopting any operating procedures. We did, however, informally adopt the following rules. General Organization Most of the technical details were hammered out in subgroups, which made the base proposals. The main group would discuss these proposals and (usually) adopt them. In practice, lots of the technical discussions were done on email lists. Keeping those lists public was a big part of the openness of the forum. For a while, each subgroup brought its own proposal as handouts. Eventually, we ended up with a unified draft thanks to Dave Loveman. The intent (which we never lived up to) was that all proposals discussed at a meeting would be in that draft. Despite always having a loose proposal or two, the draft was extremely useful - among other things, it made editing a final report possible. Main Group / Plenary Session Matters For running discussions, we used loosely-enforced Robert's Rules of Order. Basically, we didn't stand on formalities unless the discussion was becoming unruly. Some of the more often-invoked rules: 1. Moving and voting on amendments before the main proposal. Basically, this kept the discussions focussed. 2. Motions coming out of committee (subgroup) are automatically seconded; others need a second from the floor. 3. Triply-nested amendments are not allowed. This keeps the confusion down. Generally, whoever was making the presentation ran the discussion, recognized comments from the floor, etc. Ken usually came in after the meat of the discussion to run the votes. Peer pressure was fairly effective at keeping people from filibustering (I think once there was a call to limit discussion). More common was lots of people wanting to make comments - this was handled by the presenters, usually by saying "OK, let's go clockwise around the table". Organizations were limited to 2 attendees each (not enforced, but no major problems). We passed around an attendence sheet at each meeting to keep track of who was there; because of various planning problems, I recommend using a pre-registration scheme ("send email to xxx@yyy if you plan to attend"). Organizations were asked to commit to having the same attendee at every meeting, and this was generally followed. It's very important to have this kind of continuity in attendees, else we would have spent too much time in remedial education. Each organization (school, company, lab) got one vote - note this was on an organization basis, not a person. We didn't have trouble with cheating on this rule - just make sure the representatives from an organization have agreed on who is voting. An organization was eligible to vote if it had attended 2 of the last 3 meetings, counting the current meeting (i.e. you could attend every other meeting and still vote; you could not vote at your first meeting). Obviously, we couldn't enforce this rule at the first two meetings. Accepting a section of the HPF language spec was a multi-step process. 1. Someone wrote a draft specification; often there were a couple drafts. Details of these were hashed out in the subgroup. 2. First Reading: The subgroup leader (or occasionally the draft author) presented the subgroup-approved draft to the main group. When there was controversy, it was usually pointed out. The main group discussed, suggested changes (sometimes), and held a series of "straw votes" on the proposal. All attendees were eligible to vote in straw votes, which were not binding on the subgroups. 3. More subgroup discussion, both electronic and in person at the next meeting, producing a revised proposal. 4. Second Reading: The subgroup leader presented the revised proposal to the main group. Sections that were substantially the same as the original (or an alternative presented at the first reading) were amended by motion and eventually voted on. Eligibility for votes was as explained above. Major additions (for example, when PURE functions were added to FORALL between meetings) were treated as first readings at this point. Often most of the proposal was accepted, and a few sections were sent back to subgroup for more work; these came back as second readings at the next meeting. 5. Once a section was accepted at second reading, it was "frozen" until the end of the HPFF process. Revisions were only allowed for clarity or when new information surfaced (like discovering that the draft was self-contradictory). This limitation was enforced as strictly as we could, to avoid backtracking. 6. At the end of the process (December), we promised to allow reconsideration of any feature. See also the discussion of the Journal of Development below. Subgroup Matters Subgroups met independently of the main body, usually the afternoon and evening of the day before. The leader of the subgroup ran these meetings using whatever style he or she was comfortable with. Most of us took votes from whoever showed up, and ran a pseud-democratic style. Also, there was a mailing list for each subgroup where most of the discussions went on. Each subgroup was devoted to one topic from the following list: Data distribution FORALL and other parallel statements Fortran 90, storage & sequence association, and the HPF subset Intrinsic functions Parallel I/O EXTRINSIC (nee LOCAL, nee FOREIGN) functions The groups met in parallel, which caused a little friction but was logstically unavoidable. When a subject straddled two groups (like intrinsic functions for enquiring about distributions), the subgroup leaders would talk to each other and decide who would handle it - this often led to minor anti-turf battles (also known as "after you" deadlock). Both groups would act as sanity checks on the results. When it came time to write the draft, each subgroup became a chapter. The subgroup leader was the editor (and usually major author) of the chapter, and was responsible for making sure the chapter reflected the decisions made in the subgroup and in committee. Logistical Matters Meetings were 2.5 days, starting Wednesday afternoon, in Dallas (there were exceptions to this for logistical reasons, but we would have prefered to keep them all on this schedule). A typical schedule was Wednesday 1:30-6:00 Subgroup meetings, about 3 going on in parallel (reserve 4 "breakout" rooms with the hotel, fo 10-20 people each) 6:00-7:30 Unofficial dinner break (usually the subgroup leaders ate together & planned the meeting agenda) 7:30-10:30 More subgroup meetings (subgroup leaders usually ended up staying up late to revise drafts) Thursday 9:00-12:00 Full group meeting (coffee breaks at natural breaks) 12:00-1:30 Lunch (provided) 1:30-6:00 Full group meeting (and coffee breaks) 6:00-8:00 Dinner (attendees pay, but hotel provided transport to area restaurant) 8:00-10:00 (Sometimes) Full group meeting (when no full meeting, subgroups usually met instead) Friday 9:00-12:00 Full group meeting (and coffee breaks) 12:00-1:30 (Sometimes) Lunch (provided if we thought the meeting would last long) 1:30-3:00 (Sometimes) Full group meeting We tried to finish as early as possible on Friday, to allow people to catch flights. Since we scheduled all the meetings early on, we set up a contract with the hotel. I didn't handle the details, but I think this saved us a little money and a lot of aggravation (talk to Theresa Chatman if you need more info). At any rate, this is a great thing for someone on your staff to set up. We financed the meetings primarily from a per-meeting fee of $95 ($75 if we didn't have lunch the second day) per attendee. Ken offered to foot the bill for academics to attend, based on the expectation of NSF funds. We're still waiting on those funds; highly recommend that you don't make such offers without first getting the cash. Documentation The Draft: The earlier you have one the better. David Loveman served as our general editor, collecting the chapters and trying to smooth format details. I contributed the introduction and credits, mainly cribbed from meeting notes and other announcements. Guy Steele contributed a set of macros for BNF syntax and reproducing code segments. We tried to find some formatting from an ANSI standard, but no dice. Toward the end of the process, version control became a problem. We (tried to) set up the following framework: 1. David had the "official" version of the draft. He set deadlines for receiving the chapters, and edited for formatting. 2. David sent me the whole document when all the chapters were done. I LaTeXed it to check for system dependences (never a problem) 3. I sent the document to the "core" mail group (see below), put it out for anonymous FTP, and announced it on the net. 4. Any further changes were supposed to be made to David's edited copy, not the original version (this to make the next round of edits easier). The system worked pretty well, until the final draft when the wrong version of Chapter 6 got distributed... Each subgroup wrote one chapter, generally written and/or edited by the subgroup leader. While the meetings were going on, the draft contained all proposals still under consideration (Guy provided a "\alternative" macro that marked sections as being alternatives to each other). Also, the author of each section was identified in a footnote at the beginning of the section, along with the date (and occasionally other version information). In the current draft, we've moved summaries of some discarded proposals to the "Journal of Development" (primarily the ones dropped for lack of time). All proposals exist in the archives; we may put together a "rationale" document to explain some of the more controversial decisions. We're also planning to move the authorship information from footnotes to the credits chapter in the December draft. It's too early to know how this organization will work for the reader, but the modularity has helped in writing the document. Other lessons from the draft: 1. Make sure the chapter authors realize they are writing a draft chapter, not a stand-alone document. 2. Make sure the person in charge of the credits chapter has plenty of time to work on it (I spent 50% of my time for a week appeasing various corporate egos, wording things right, and checking spelling of names). 3. Proofread the document before sending it out - both spell checking and careful chapter reading. Mailing lists: Every subgroup had its own mailing list. Those lists are where most of the technical action happened; they were very high-volume (final total was something like 1.5Mb of archived messages). On top of that, there was a list for everybody in the world interested in HPF, and another for the "core" group. The "everybody" list was used for the meeting minutes and a few miscellaneous announcements. The core group list was primarily for the meeting attendees, but a few others were also on it for political and/or practical reasons (for example, Gil Weigand and Theresa Chatman were both on the list); it got meeting details, and copies of the various proposals before the meetings. All lists were kept at cs.rice.edu (there were a couple smaller lists other places, including a European list at GMD and several local lists for particular campuses). People could add or delete themselves to/from any of the lists by mailing to another mail alias at Rice. Automating this was a huge win (although I got 2 or 3 requests sent directly to me every week). FTP: We made everything we could available by anonymous FTP at Rice; eventually other sites picked up some of it as well. This included the various drafts of the language spec; any other handouts from the meetings; supporting (sometimes critical) commentary; papers on Fortran D, Vienna Fortran, ADAPT, and other systems; archived email traffic; and the minutes of the meetings. Generally, I put up a couple versions of any document (particularly text formatting stuff) - TeX or LaTeX, Postscript, and compressed versions of large files. This was important, since I've found out some fascinating things about portability (or lack thereof) of different formats. (Hints: keep all lines 80 characters or less; no fancy macros in TeX; at most one "." in a file name (filename.tex.Z doesn't translate well to IBM!); never include ANY pathname.) Minutes: Extremely popular, but lots of work. I'm convinced that making them available was a key to HPFF's success. Make them as detailed as possible, particularly including counts of the votes taken, major topics discussed, and lists of people (humor helps occasionally, too). "Executive Summary" sections with the major issues at the front seemed to be popular. I also used the minutes to announce additions to the FTP archives. Advertizing: Announcements of major news (new drafts, etc.) went everywhere I could think of. This includes the "world" mailing list; newsgroups comp.lang.fortran, comp.lang.misc, and comp.parallel; (indirectly) na-net, scinet, and hpcwire. Meeting minutes went to the world and core mailing lists and the newsgroups. I've been writing short "What is HPF" articles constantly, and I think the same is true of others. Ken, Dave Loveman, Guy, and I have all given large numbers of talks on HPF; again, the same is probably true of others. This includes invited talks at conferences and visits to various companies. From chk@cs.rice.edu Mon Nov 23 15:45:48 1992 Return-Path: Received: from cs.rice.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26184; Mon, 23 Nov 92 15:45:39 -0500 Received: from DialupEudora (charon.rice.edu) by cs.rice.edu (AA22750); Mon, 23 Nov 92 14:44:21 CST Message-Id: <9211232044.AA22750@cs.rice.edu> Date: Mon, 23 Nov 1992 14:47:55 -0600 To: dongarra@cs.utk.edu From: chk@cs.rice.edu Subject: HPFF organization X-Attachments: :Macintosh HD:4227:HPFF rules: Status: RO Here it is. If anything is unclear, just ask me or (for logistical details) Theresa Chatman (tlc@cs.rice.edu). Chuck HPFF Working Procedures HPFF never had an explicit vote on adopting any operating procedures. We did, however, informally adopt the following rules. General Organization Most of the technical details were hammered out in subgroups, which made the base proposals. The main group would discuss these proposals and (usually) adopt them. In practice, lots of the technical discussions were done on email lists. Keeping those lists public was a big part of the openness of the forum. For a while, each subgroup brought its own proposal as handouts. Eventually, we ended up with a unified draft thanks to Dave Loveman. The intent (which we never lived up to) was that all proposals discussed at a meeting would be in that draft. Despite always having a loose proposal or two, the draft was extremely useful - among other things, it made editing a final report possible. Main Group / Plenary Session Matters For running discussions, we used loosely-enforced Robert's Rules of Order. Basically, we didn't stand on formalities unless the discussion was becoming unruly. Some of the more often-invoked rules: 1. Moving and voting on amendments before the main proposal. Basically, this kept the discussions focussed. 2. Motions coming out of committee (subgroup) are automatically seconded; others need a second from the floor. 3. Triply-nested amendments are not allowed. This keeps the confusion down. Generally, whoever was making the presentation ran the discussion, recognized comments from the floor, etc. Ken usually came in after the meat of the discussion to run the votes. Peer pressure was fairly effective at keeping people from filibustering (I think once there was a call to limit discussion). More common was lots of people wanting to make comments - this was handled by the presenters, usually by saying "OK, let's go clockwise around the table". Organizations were limited to 2 attendees each (not enforced, but no major problems). We passed around an attendence sheet at each meeting to keep track of who was there; because of various planning problems, I recommend using a pre-registration scheme ("send email to xxx@yyy if you plan to attend"). Organizations were asked to commit to having the same attendee at every meeting, and this was generally followed. It's very important to have this kind of continuity in attendees, else we would have spent too much time in remedial education. Each organization (school, company, lab) got one vote - note this was on an organization basis, not a person. We didn't have trouble with cheating on this rule - just make sure the representatives from an organization have agreed on who is voting. An organization was eligible to vote if it had attended 2 of the last 3 meetings, counting the current meeting (i.e. you could attend every other meeting and still vote; you could not vote at your first meeting). Obviously, we couldn't enforce this rule at the first two meetings. Accepting a section of the HPF language spec was a multi-step process. 1. Someone wrote a draft specification; often there were a couple drafts. Details of these were hashed out in the subgroup. 2. First Reading: The subgroup leader (or occasionally the draft author) presented the subgroup-approved draft to the main group. When there was controversy, it was usually pointed out. The main group discussed, suggested changes (sometimes), and held a series of "straw votes" on the proposal. All attendees were eligible to vote in straw votes, which were not binding on the subgroups. 3. More subgroup discussion, both electronic and in person at the next meeting, producing a revised proposal. 4. Second Reading: The subgroup leader presented the revised proposal to the main group. Sections that were substantially the same as the original (or an alternative presented at the first reading) were amended by motion and eventually voted on. Eligibility for votes was as explained above. Major additions (for example, when PURE functions were added to FORALL between meetings) were treated as first readings at this point. Often most of the proposal was accepted, and a few sections were sent back to subgroup for more work; these came back as second readings at the next meeting. 5. Once a section was accepted at second reading, it was "frozen" until the end of the HPFF process. Revisions were only allowed for clarity or when new information surfaced (like discovering that the draft was self-contradictory). This limitation was enforced as strictly as we could, to avoid backtracking. 6. At the end of the process (December), we promised to allow reconsideration of any feature. See also the discussion of the Journal of Development below. Subgroup Matters Subgroups met independently of the main body, usually the afternoon and evening of the day before. The leader of the subgroup ran these meetings using whatever style he or she was comfortable with. Most of us took votes from whoever showed up, and ran a pseud-democratic style. Also, there was a mailing list for each subgroup where most of the discussions went on. Each subgroup was devoted to one topic from the following list: Data distribution FORALL and other parallel statements Fortran 90, storage & sequence association, and the HPF subset Intrinsic functions Parallel I/O EXTRINSIC (nee LOCAL, nee FOREIGN) functions The groups met in parallel, which caused a little friction but was logstically unavoidable. When a subject straddled two groups (like intrinsic functions for enquiring about distributions), the subgroup leaders would talk to each other and decide who would handle it - this often led to minor anti-turf battles (also known as "after you" deadlock). Both groups would act as sanity checks on the results. When it came time to write the draft, each subgroup became a chapter. The subgroup leader was the editor (and usually major author) of the chapter, and was responsible for making sure the chapter reflected the decisions made in the subgroup and in committee. Logistical Matters Meetings were 2.5 days, starting Wednesday afternoon, in Dallas (there were exceptions to this for logistical reasons, but we would have prefered to keep them all on this schedule). A typical schedule was Wednesday 1:30-6:00 Subgroup meetings, about 3 going on in parallel (reserve 4 "breakout" rooms with the hotel, fo 10-20 people each) 6:00-7:30 Unofficial dinner break (usually the subgroup leaders ate together & planned the meeting agenda) 7:30-10:30 More subgroup meetings (subgroup leaders usually ended up staying up late to revise drafts) Thursday 9:00-12:00 Full group meeting (coffee breaks at natural breaks) 12:00-1:30 Lunch (provided) 1:30-6:00 Full group meeting (and coffee breaks) 6:00-8:00 Dinner (attendees pay, but hotel provided transport to area restaurant) 8:00-10:00 (Sometimes) Full group meeting (when no full meeting, subgroups usually met instead) Friday 9:00-12:00 Full group meeting (and coffee breaks) 12:00-1:30 (Sometimes) Lunch (provided if we thought the meeting would last long) 1:30-3:00 (Sometimes) Full group meeting We tried to finish as early as possible on Friday, to allow people to catch flights. Since we scheduled all the meetings early on, we set up a contract with the hotel. I didn't handle the details, but I think this saved us a little money and a lot of aggravation (talk to Theresa Chatman if you need more info). At any rate, this is a great thing for someone on your staff to set up. We financed the meetings primarily from a per-meeting fee of $95 ($75 if we didn't have lunch the second day) per attendee. Ken offered to foot the bill for academics to attend, based on the expectation of NSF funds. We're still waiting on those funds; highly recommend that you don't make such offers without first getting the cash. Documentation The Draft: The earlier you have one the better. David Loveman served as our general editor, collecting the chapters and trying to smooth format details. I contributed the introduction and credits, mainly cribbed from meeting notes and other announcements. Guy Steele contributed a set of macros for BNF syntax and reproducing code segments. We tried to find some formatting from an ANSI standard, but no dice. Toward the end of the process, version control became a problem. We (tried to) set up the following framework: 1. David had the "official" version of the draft. He set deadlines for receiving the chapters, and edited for formatting. 2. David sent me the whole document when all the chapters were done. I LaTeXed it to check for system dependences (never a problem) 3. I sent the document to the "core" mail group (see below), put it out for anonymous FTP, and announced it on the net. 4. Any further changes were supposed to be made to David's edited copy, not the original version (this to make the next round of edits easier). The system worked pretty well, until the final draft when the wrong version of Chapter 6 got distributed... Each subgroup wrote one chapter, generally written and/or edited by the subgroup leader. While the meetings were going on, the draft contained all proposals still under consideration (Guy provided a "\alternative" macro that marked sections as being alternatives to each other). Also, the author of each section was identified in a footnote at the beginning of the section, along with the date (and occasionally other version information). In the current draft, we've moved summaries of some discarded proposals to the "Journal of Development" (primarily the ones dropped for lack of time). All proposals exist in the archives; we may put together a "rationale" document to explain some of the more controversial decisions. We're also planning to move the authorship information from footnotes to the credits chapter in the December draft. It's too early to know how this organization will work for the reader, but the modularity has helped in writing the document. Other lessons from the draft: 1. Make sure the chapter authors realize they are writing a draft chapter, not a stand-alone document. 2. Make sure the person in charge of the credits chapter has plenty of time to work on it (I spent 50% of my time for a week appeasing various corporate egos, wording things right, and checking spelling of names). 3. Proofread the document before sending it out - both spell checking and careful chapter reading. Mailing lists: Every subgroup had its own mailing list. Those lists are where most of the technical action happened; they were very high-volume (final total was something like 1.5Mb of archived messages). On top of that, there was a list for everybody in the world interested in HPF, and another for the "core" group. The "everybody" list was used for the meeting minutes and a few miscellaneous announcements. The core group list was primarily for the meeting attendees, but a few others were also on it for political and/or practical reasons (for example, Gil Weigand and Theresa Chatman were both on the list); it got meeting details, and copies of the various proposals before the meetings. All lists were kept at cs.rice.edu (there were a couple smaller lists other places, including a European list at GMD and several local lists for particular campuses). People could add or delete themselves to/from any of the lists by mailing to another mail alias at Rice. Automating this was a huge win (although I got 2 or 3 requests sent directly to me every week). FTP: We made everything we could available by anonymous FTP at Rice; eventually other sites picked up some of it as well. This included the various drafts of the language spec; any other handouts from the meetings; supporting (sometimes critical) commentary; papers on Fortran D, Vienna Fortran, ADAPT, and other systems; archived email traffic; and the minutes of the meetings. Generally, I put up a couple versions of any document (particularly text formatting stuff) - TeX or LaTeX, Postscript, and compressed versions of large files. This was important, since I've found out some fascinating things about portability (or lack thereof) of different formats. (Hints: keep all lines 80 characters or less; no fancy macros in TeX; at most one "." in a file name (filename.tex.Z doesn't translate well to IBM!); never include ANY pathname.) Minutes: Extremely popular, but lots of work. I'm convinced that making them available was a key to HPFF's success. Make them as detailed as possible, particularly including counts of the votes taken, major topics discussed, and lists of people (humor helps occasionally, too). "Executive Summary" sections with the major issues at the front seemed to be popular. I also used the minutes to announce additions to the FTP archives. Advertizing: Announcements of major news (new drafts, etc.) went everywhere I could think of. This includes the "world" mailing list; newsgroups comp.lang.fortran, comp.lang.misc, and comp.parallel; (indirectly) na-net, scinet, and hpcwire. Meeting minutes went to the world and core mailing lists and the newsgroups. I've been writing short "What is HPF" articles constantly, and I think the same is true of others. Ken, Dave Loveman, Guy, and I have all given large numbers of talks on HPF; again, the same is probably true of others. This includes invited talks at conferences and visits to various companies. From owner-mpi-comm@CS.UTK.EDU Tue Nov 24 18:09:18 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08323; Tue, 24 Nov 92 18:09:18 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA22602; Tue, 24 Nov 92 17:43:42 -0500 Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22596; Tue, 24 Nov 92 17:43:35 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA11950; Tue, 24 Nov 92 17:43:40 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 174208.17287; Tue, 24 Nov 1992 17:42:08 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-92031301) id AA07130; Tue, 24 Nov 1992 16:14:37 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA06815; Tue, 24 Nov 92 16:14:36 -0600 Date: Tue, 24 Nov 92 16:14:36 -0600 Message-Id: <9211242214.AA06815@brisk.kai.com> To: dongarra@cs.utk.edu Cc: mpi-comm@cs.utk.edu In-Reply-To: Jack Dongarra's message of Tue, 24 Nov 92 15:04:05 -0500 <9211242004.AA20345@thud.cs.utk.edu> Subject: more on the intro From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 Concerning the introduction: "The paradigm will not be made obsolete by architectures combining the shared- and distributed-memory views, or by increases in network speeds." I think this sentence is unnecessarily defensive. I'd delete the sentence and the following "thus". I earlier raised an objection to \item A simple way to create processes for the SPMD model there seems to be some redundancy here with \item Process groups The former is a subset of the latter. I'll repeat my noncirculated comments: I would rather not see us say too much about the process model at this stage; except to say that there is some process model defined by the language using the standard and to say something about side effects between processes that affect the message passing semantics (i.e., that there should be none). That does, of course, leave us with some interesting problems for identifying our communication; traditionally a difficult subject in message passing (hence my distribution of this message to the whole committee). The easy route would be to have a global name space of shared "channels" each with a communication characteristic [type] (e.g., "point-to-point" or "one-to-many"): a compiler can check correct usage if we identify the right rules (e.g. a synchronized unidirectional point-to-point communication is shared by only two processes). Such logical naming would map well across all the existing systems and avoid the problems of name servers, processor id, topologies and so forth since any topology can be described by the set of names and its users. Processor ids are, in effect, names in a global name space - and are often channels that have a "many-to-one" characteristic. Steven From owner-mpi-comm@CS.UTK.EDU Sat Nov 28 18:38:03 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07921; Sat, 28 Nov 92 18:38:03 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA22407; Sat, 28 Nov 92 18:35:38 -0500 Received: from cunyvm.cuny.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22403; Sat, 28 Nov 92 18:35:36 -0500 Message-Id: <9211282335.AA22403@CS.UTK.EDU> Received: from YKTVMV by CUNYVM.CUNY.EDU (IBM VM SMTP V2R2) with BSMTP id 6064; Sat, 28 Nov 92 18:35:01 EST Date: Sat, 28 Nov 92 18:32:55 EST From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-comm@cs.utk.edu Subject: previously sent postcript file Reply-To: SNIR@watson.ibm.com Contains an outline I want to use for discussing issues in the poin-to-point subcommittee. I send it to the entire committee since it touches many issues related to collective communication and formal definitions. If somebody cannot print postscript and prefers latex, pls let me know. From owner-mpi-comm@CS.UTK.EDU Sat Nov 28 18:40:52 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07938; Sat, 28 Nov 92 18:40:52 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA22399; Sat, 28 Nov 92 18:33:55 -0500 Received: from cunyvm.cuny.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22394; Sat, 28 Nov 92 18:33:40 -0500 Message-Id: <9211282333.AA22394@CS.UTK.EDU> Received: from YKTVMV by CUNYVM.CUNY.EDU (IBM VM SMTP V2R2) with BSMTP id 6053; Sat, 28 Nov 92 18:33:04 EST Date: Sat, 28 Nov 92 18:32:22 EST From: "Marc Snir" To: MPI-COMM@CS.UTK.EDU %!PS-Adobe-2.0 %%Creator: dvips 5.47 Copyright 1986-91 Radical Eye Software %%Title: MPIOUTLN.DVI.* %%Pages: 10 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: texc.pro /TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{ ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]/id ch-image N /rw ch-width 7 add 8 idiv string N /rc 0 N /gp 0 N /cp 0 N{rc 0 ne{rc 1 sub /rc X rw}{G}ifelse}imagemask restore}B /G{{id gp get /gp gp 1 add N dup 18 mod S 18 idiv pl S get exec}loop}B /adv{cp add /cp X}B /chg{rw cp id gp 4 index getinterval putinterval dup gp add /gp X adv}B /nd{/cp 0 N rw exit}B /lsh{rw cp 2 copy get dup 0 eq{pop 1}{dup 255 eq{pop 254}{dup dup add 255 and S 1 and or}ifelse}ifelse put 1 adv}B /rsh{rw cp 2 copy get dup 0 eq{pop 128}{dup 255 eq{pop 127}{dup 2 idiv S 128 and or}ifelse}ifelse put 1 adv}B /clr{rw cp 2 index string putinterval adv}B /set{rw cp fillstr 0 4 index getinterval putinterval adv}B /fillstr 18 string 0 1 17{2 copy 255 put pop}for N /pl[{adv 1 chg}bind{adv 1 chg nd}bind{1 add chg}bind{1 add chg nd}bind{adv lsh}bind{ adv lsh nd}bind{adv rsh}bind{adv rsh nd}bind{1 add adv}bind{/rc X nd}bind{1 add set}bind{1 add clr}bind{adv 2 chg}bind{adv 2 chg nd}bind{pop nd}bind]N /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval (Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail} B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{ /SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 1 16 df15 D E /Fb 51 125 df<127012F8B012701200A5 127012F8A31270051C779B18>33 D<137013F01201EA03C0EA0780EA0F00121E121C123C123812 781270A212F05AA87E1270A212781238123C121C121E7EEA0780EA03C0EA01F0120013700C2479 9F18>40 D<126012F012787E7E7EEA0780120313C0120113E01200A213F01370A813F013E0A212 0113C0120313801207EA0F00121E5A5A5A12600C247C9F18>I<123C127E127FA3123F120F120E 121E127C12F81270080C788518>44 DI48 DII<387FFFC0B512E0A3C8FCA4B512E0A36C13C0130C7E9318>61 D<137013F8A213D8A2EA01DCA3138CEA038EA41306EA0707A4380FFF80A3EA0E03A2381C01C0A2 387F07F038FF8FF8387F07F0151C7F9B18>65 DI<3801FCE0EA03FEEA07FFEA0F07EA1E03EA3C01EA78001270A200F013005AA87E0070 13E0A21278EA3C01001E13C0EA0F073807FF806C1300EA01FC131C7E9B18>IIII<3801F9C0EA07FF5AEA1F0FEA1C03123CEA78011270A200F0C7FC5AA5 EB0FF0131F130F38F001C0127013031278123CEA1C07EA1F0FEA0FFFEA07FDEA01F9141C7E9B18 >I<387F07F038FF8FF8387F07F0381C01C0A9EA1FFFA3EA1C01AA387F07F038FF8FF8387F07F0 151C7F9B18>II76 D<38FC01F8EAFE03A2383B06E0 A4138EA2EA398CA213DCA3EA38D8A213F81370A21300A638FE03F8A3151C7F9B18>I<387E07F0 38FF0FF8387F07F0381D81C0A313C1121CA213E1A313611371A213311339A31319A2131D130DA3 EA7F07EAFF87EA7F03151C7F9B18>III82 D<3807F380EA1FFF5AEA7C1FEA7007EAF00312E0A2 90C7FC7E1278123FEA1FF0EA0FFEEA01FF38001F80EB03C0EB01E01300A2126012E0130100F013 C0EAFC07B512801400EAE7FC131C7E9B18>I<387FFFF8B5FCA238E07038A400001300B2EA07FF A3151C7F9B18>I<38FF83FEA3381C0070B2001E13F0000E13E0EA0F013807C7C03803FF806C13 00EA007C171C809B18>I<38FF07F8A3381C01C0A4380E0380A4EA0F0700071300A4EA038EA4EA 018C13DCA3EA00D813F8A21370151C7F9B18>I<38FE03F8A338700070A36C13E0A513F8A2EA39 DCA2001913C0A3138CEA1D8DA4000D13801305EA0F07A2EA0E03151C7F9B18>I97 D<127E12FE127E120EA5133EEBFF80000F13C0EBE3E0EB80F0EB0070 1478000E1338A5120F14781470EB80F0EBC3E0EBFFC0000E138038067E00151C809B18>IIIII<3803F1F03807FFF85A381E1F30 383C0F00EA3807A5EA3C0FEA1E1EEA1FFC485AEA3BF00038C7FC123CEA1FFF14C04813E0387801 F038F00078481338A36C1378007813F0EA7E03383FFFE0000F13803803FE00151F7F9318>I<12 7E12FE127E120EA5133FEBFF80000F13C0EBE1E013801300A2120EAA387FC3FC38FFE7FE387FC3 FC171C809B18>II<12FEA3120EA5EB3FF0137F133FEB0780EB0F00131E5B5B5BEA0FF87F139C131EEA0E0FEB 0780130314C038FFC7F8A3151C7F9B18>107 DI<387DF1F038FFFBF86CB47E381F1F1CEA1E1EA2EA1C1CAB387F1F1F39FFBF BF80397F1F1F001914819318>IIII<387F87E038FF9FF8EA7FBF3803 FC78EBF030EBE0005BA35BA8EA7FFEB5FC6C5A15147F9318>114 DI<487E1203A4387FFFC0B5FCA238038000A9144014E0A21381EBC3C0EA01 FF6C1380EB7E0013197F9818>I<387E07E0EAFE0FEA7E07EA0E00AC1301EA0F073807FFFC6C13 FE3801FCFC1714809318>I<38FF8FF8A3383800E0A3381C01C0A2137113F9A213D9A2380DDD80 A3138DEA0F8FA23807070015147F9318>119 D<387F8FF000FF13F8007F13F0380E01C0EB0380 A21207EB0700A2EA03871386138EEA01CEA2EA00CCA213DC1378A31370A313F05B1279EA7BC0EA 7F806CC7FC121E151E7F9318>121 D<126012F0B3B012600424769F18>124 D E /Fc 12 119 df97 DI<137E48B4FC38038380EA0F07121E001C1300EA3C0248C7FCA35AA5EA70 021307EA381EEA1FF8EA07E011147C9315>I<1478EB03F814F0EB0070A314E0A4EB01C0A213F1 EA03FD38078F80EA0E07121C123C14001278A3EAF00EA31430EB1C60133CEA707C3878FCC0EA3F CF380F078015207C9F17>I<137C48B4FCEA0783380F0180121E123CEB0300EA780EEA7FFC13E0 00F0C7FCA412701302EA7807EA3C1EEA1FF8EA07E011147C9315>I103 D<136013F0A213E01300A7120FEA1F8012 3113C0EA6380A212C3EA0700A3120EA3EA1C301360A2EA38C01218EA1F80EA0F000C1F7D9E0E> 105 D108 D<381E07C0383F1FE03833B870EA63E0EBC038138000C71370EA0700A3000E 13E0A3EB01C3001C13C6A2EB038C1301003813F8381800F018147D931A>110 D<137C48B4FC38038380380F01C0121E001C13E0123C1278A338F003C0A3EB07801400EA700F13 1EEA3838EA1FF0EA07C013147C9317>I116 D<380F01C0EA1F83003113E0 13C1EA6380A200C313C0EA0700A3380E0180A3EB0300A21306A2EA0F0CEA07F8EA01E013147D93 15>118 D E /Fd 28 121 df<903807F83F017FB512C03A01FC0FE3E03903F01FC7EA07E0D80F C01387ED83C0ED8000A6B612FCA2390FC01F80B2397FF8FFF8A223237FA221>11 D<13181330136013C01201EA0380120713005A121EA2123E123CA2127CA3127812F8AD1278127C A3123CA2123E121EA27E7E13801203EA01C012001360133013180D317BA416>40 D<12C012607E7E121C7E120F7E1380EA03C0A213E01201A213F0A3120013F8AD13F01201A313E0 A2120313C0A2EA078013005A120E5A12185A5A5A0D317DA416>I63 D67 D71 D77 D87 D97 DIII<13FE3807FF80380F87C0381E01E0003E13F0EA7C0014 F812FCA2B5FCA200FCC7FCA3127CA2127E003E13186C1330380FC0703803FFC0C6130015167E95 1A>II<3801FE1F0007B51280380F87E7EA1F03391E01E000003E7FA5001E5BEA1F03 380F87C0EBFF80D819FEC7FC0018C8FC121CA2381FFFE014F86C13FE80123F397C003F8048131F 140FA3007CEB1F00007E5B381F80FC6CB45A000113C019217F951C>II<120E121FEA3F80A3EA1F00 120EC7FCA7EAFF80A2121FB2EAFFF0A20C247FA30F>I108 D<3AFF87F00FE090399FFC3FF83A1FB87E70FC9039E03EC07C9039C03F807EA2018013 00AE3BFFF1FFE3FFC0A22A167E952F>I<38FF87E0EB9FF8381FB8FCEBE07CEBC07EA21380AE39 FFF1FFC0A21A167E951F>I<13FE3807FFC0380F83E0381E00F0003E13F848137CA300FC137EA7 007C137CA26C13F8381F01F0380F83E03807FFC03800FE0017167E951C>I<38FF8FE0EBBFF838 1FF07CEBC03E497E1580A2EC0FC0A8EC1F80A2EC3F00EBC03EEBE0FCEBBFF8EB8FC00180C7FCA8 EAFFF0A21A207E951F>I114 DI<13C0A41201A2120312 07120F121FB5FCA2EA0FC0ABEBC180A51207EBE300EA03FEC65A11207F9F16>I<38FF83FEA238 1F807EAF14FEA2EA0F833907FF7FC0EA01FC1A167E951F>I<39FFF01FE0A2390FC00600A2EBE0 0E0007130CEBF01C0003131813F800015BA26C6C5AA2EB7EC0A2137F6D5AA26DC7FCA2130EA21B 167F951E>I<39FFF07FC0A2390FC01C006C6C5A6D5A6C6C5A00015B3800FD80017FC7FCA27F6D 7E497E80EB67F013E33801C1F8380381FC48C67E000E137E39FF81FFE0A21B167F951E>120 D E /Fe 1 4 df<1207A3EAE738EAFFF8EA7FF0EA1FC0A2EA7FF0EAFFF8EAE738EA0700A30D0E 7E8E12>3 D E /Ff 33 118 df<1318137013E0EA01C0EA0380A2EA0700120EA2121E121C123C A25AA412F85AA97E1278A47EA2121C121E120EA27EEA0380A2EA01C0EA00E0137013180D2D7DA1 14>40 D<12C012707E7E7EA27EEA0380A213C0120113E0A2EA00F0A413F81378A913F813F0A4EA 01E0A213C012031380A2EA0700120EA25A5A5A12C00D2D7DA114>I<14E0B0B712C0A3C700E0C7 FCB022237D9C29>43 D<1238127C12FE12FFA2127F123B1203A21206A2120E120C121812701220 08107C860F>I<137013F0120F12FF12F31203B3A4B51280A2111D7C9C1A>49 DIII<14E0A2497EA3497EA2497EA2 497E130CA2EB187FA201307F143F01707FEB601FA201C07F140F48B57EA2EB800748486C7EA200 06801401000E803AFFE01FFFE0A2231F7E9E28>65 D69 DI73 D78 D82 D<3803FC08380FFF38381E03F8EA3C00481378143812F814187E1400B4FC13F86C B4FC14C06C13E06C13F06C13F8120338001FFC13011300A200C0137CA36C1378A200F813F038FE 01E038E7FFC000811300161F7D9E1D>I<007FB512FCA2397C0FE07C0070141C0060140CA200E0 140E00C01406A400001400B10007B512C0A21F1E7E9D24>I87 D97 D100 DII<3801FC3C3807FFFE380F07DEEA1E03003E13E0A5001E13C0380F 0780EBFF00EA19FC0018C7FCA2121C381FFF8014F06C13F8003F13FC387C007C0070133E00F013 1EA30078133CA2383F01F8380FFFE000011300171E7F931A>II<121C123F5AA37E121CC7FCA6B4FCA2121F B0EAFFE0A20B217EA00E>I108 D<3AFE0FE03F8090393FF0FFC03A1E70F9C3E09039C07F01F0381F807EA2EB007CAC3AFFE3FF8F FEA227147D932C>I<38FE0FC0EB3FE0381E61F0EBC0F8EA1F801300AD38FFE3FFA218147D931D> I<48B4FC000713C0381F83F0383E00F8A248137CA200FC137EA6007C137CA26C13F8A2381F83F0 3807FFC00001130017147F931A>I<38FF1FC0EB7FF0381FE1F8EB80FCEB007EA2143E143FA614 3E147E147CEB80FCEBC1F8EB7FE0EB1F8090C7FCA7EAFFE0A2181D7E931D>I114 DII<38FF07F8A2EA1F00AD1301A2EA0F073807FEFFEA03F818147D931D>I E /Fg 77 124 df<90380FC3E090387FEFF09038E07C783801C0F8D8038013303907007000A7B6 1280A23907007000B0387FE3FFA21D20809F1B>11 DII<90380F80F890387FE7FE9038E06E063901C0 FC0F380380F8380700F00270C7FCA6B7FCA23907007007B03A7FE3FE3FF0A22420809F26>I<90 380FC0FFEB7FE79038E07E0F3801C0FC4848487E38070070A7B7FCA23907007007B03A7FE3FE3F F0A22420809F26>I34 D<127012F812FCA2127C120CA31218A21238123012601240060E7C9F0D >39 D<136013C0EA0180EA03005A12065A121C12181238A212301270A31260A212E0AC1260A212 70A312301238A21218121C120C7E12077EEA0180EA00C013600B2E7DA112>I<12C012607E7E12 1C120C7E12077E1380A2120113C0A31200A213E0AC13C0A21201A313801203A213005A12065A12 1C12185A5A5A0B2E7DA112>I<1306AFB612F0A2D80006C7FCAF1C207D9A23>43 D<127012F812FCA2127C120CA31218A21238123012601240060E7C840D>II<127012F8A3127005057C840D>I<1303A213071306A2130E130CA2131C1318A213381330A2 13701360A213E013C0A212011380A312031300A25A1206A2120E120CA2121C1218A212381230A2 12701260A212E05AA2102D7DA117>IIIII<130EA2131E133EA2136E13EE13CEEA018E120313 0E1206120E120C121812381230126012E0B512F0A238000E00A7EBFFE0A2141E7F9D17>II<137CEA01FEEA07 83380E0380EA0C07121C3838030090C7FC12781270A2EAF3F8EAF7FEEAFC0E487EEB0380A200F0 13C0A51270A214801238EB0700121CEA0E1EEA07FCEA01F0121F7E9D17>I<1260387FFFC0A214 80EA600138C003001306A2C65A5BA25B5BA213E05B1201A3485AA41207A76CC7FC121F7D9D17> II I<127012F8A312701200AA127012F8A3127005147C930D>I<127012F8A312701200AA127012F8 A312781218A41230A21260A21240051D7C930D>I<007FB512E0B612F0C9FCA8B612F06C14E01C 0C7D9023>61 D63 D65 DI<90381FC04090387F F0C03801F8393803C00D38078007380F0003121E003E1301123C127C1400127812F81500A80078 14C0127CA2123C003EEB0180121E6CEB0300EA07803803C00E3801F81C38007FF0EB1FC01A217D 9F21>IIII<90380FC02090387FF8603901F81CE03803E0063807800338 0F0001121E14005A127C1560127812F81500A6EC7FFCA20078EB01E0127CA2123C7EA27E380780 03EA03E03901F80E6039007FFC2090380FE0001E217D9F24>I<39FFF8FFF8A23907800F00AC90 B5FCA2EB800FAD39FFF8FFF8A21D1F7E9E22>II< EAFFFEA2EA0780B11406A4140EA2140C141C143C14FCB5FCA2171F7E9E1C>76 DI<39FF807FF813C0 0007EB07809038E00300A2EA06F0A21378133CA2131EA2130FA2EB078314C31303EB01E3A2EB00 F3A2147BA2143F80A280A2000F7FEAFFF0801D1F7E9E22>III82 D<3807E080EA0FF9EA1C1FEA300FEA7007EA600312E01301A36CC7FCA21278127FEA3FF0EA1FFC 6C7EEA03FF38001F801307EB03C0A2130112C0A400E01380EAF00338F80700EAFE0EEACFFCEA81 F812217D9F19>I<007FB512E0A238780F010070130000601460A200E0147000C01430A4000014 00B23807FFFEA21C1F7E9E21>I<39FFFC7FF8A23907800780EC0300B3A300031302EBC006A200 015B6C6C5AEB7830EB3FE0EB0FC01D207E9E22>I<39FFF007FEA2390F8001F090C712E06C6C13 C0A2EBC00100031480A2EBE00300011400A23800F006A3EB780CA36D5AA36D5AA2EB1F70EB0F60 A214E06D5AA26D5AA31F207F9E22>I<3BFFF07FF83FF0A23B0F0007800F80EE0300A23A07800F C006A3913819E00ED803C0140CA214393A01E030F018A33A00F0607830A3ECE07C903978C03C60 A390393D801EC0A390383F000F6D5CA3010E6DC7FCA32C207F9E2F>I<397FF83FF8A23907C00F 800003EB06003801E00EEBF00C00005BEB7838EB7C30EB3C70EB3E60EB1EC0130F5C1307808013 0DEB1DF0EB18F8EB3878EB307CEB603CEBE01EEBC01F48487E0003EB0780010013C0EA0F8039FF E01FFEA21F1F7F9E22>I92 D97 D<120E12FEA2120EA9133FEB FF80380FC3C0EB00E0000E13F014701478A7147014F0120FEB01E0EBC3C0380CFF80EB3E001520 7F9F19>IIII<133C13FEEA01CFEA038F1306EA0700A7EAFFF0A2EA0700B0EA7FF0A21020 809F0E>II<120E12FEA2120EA9133E13FF380FC380EB01C0 A2120EAD38FFE7FCA216207F9F19>I<121C121E123E121E121CC7FCA6120E127EA2120EAFEAFF C0A20A1F809E0C>I<13E0EA01F0A3EA00E01300A61370EA07F0A212001370B3A21260EAF0E0EA F1C0EA7F80EA3E000C28829E0E>I<120E12FEA2120EA9EB1FF0A2EB0F80EB0E00130C5B5B1370 13F0EA0FF81338EA0E1C131E130E7F1480130314C038FFCFF8A215207F9F18>I<120E12FEA212 0EB3A9EAFFE0A20B20809F0C>I<390E3F03F039FEFF8FF839FFC1DC1C390F80F80EEB00F0000E 13E0AD3AFFE7FE7FE0A223147F9326>IIII<3803E180EA0FF9EA1E1FEA3C0712781303127012F0A6127012781307EA3C0FEA 1E1FEA0FF3EA03E3EA0003A7EB3FF8A2151D7E9318>III<1206A4120EA2121E123E EAFFF8A2EA0E00AA1318A5EA073013E0EA03C00D1C7F9B12>I<380E01C0EAFE1FA2EA0E01AC13 03A2EA070FEBFDFCEA01F116147F9319>I<38FF87F8A2381E01E0000E13C01480A238070300A3 EA0386A2138EEA01CCA213FC6C5AA21370A315147F9318>I<39FF9FF3FCA2391C0780F01560EC C0E0D80E0F13C0130C14E00007EBE180EB186114713903987300EBB033A2143F3801F03EEBE01E A20000131CEBC00C1E147F9321>I<387FC7FCA2380703E0148038038300EA01C7EA00EE13EC13 781338133C137C13EEEA01C7138738030380380701C0000F13E038FF87FEA21714809318>I<38 FF87F8A2381E01E0000E13C01480A238070300A3EA0386A2138EEA01CCA213FC6C5AA21370A313 60A35B12F0EAF18012F3007FC7FC123C151D7F9318>III E /Fh 29 122 df<48B4FC000F13E0383E03F8007813FCEA7E01 00FF13FEA5387E03FC1200EB07F0EB0FE01480EB1F00131E5BA25BA21370A690C7FCA61370EA01 FC487EA56C5AEA0070172A7CA920>63 D 70 D<91387FE003903903FFFC0F011FEBFF1F90397FF00FFF9038FF8001D803FEC7FC48488048 48804980485A003F815B007F81A3484891C7FCA90203B512F8A2EA7FC0DA00011300A2123F7F12 1F6C7E7F6C7E6C6C5B3800FF8090387FF00F011FB5123F0103EBFC0F9039007FE0032D297CA836 >I73 D77 D<90387F80603903FFF0E0000F13FF38 1F807F383F001F003E1307007E1303127C00FC1301A214007E7E6D130013F8EBFF806C13F814FE 6C7F6C14C07E6C14E0000114F0EA003F010113F8EB001F1407A200E013031401A37E15F06C1303 6C14E0B413079038E01FC090B5120000E05B38C01FF01D297CA826>83 D87 D<48B47E000F13F0381F 81FC486C7E147FA2EC3F80A2EA0F00C7FCA2EB0FFF90B5FC3807FC3FEA1FE0EA3F80127F130012 FEA3147F7E6CEBFFC0393F83DFFC380FFF0F3801FC031E1B7E9A21>97 D99 DIII<9038FF81F00003EB E7FC390FC1FE7C391F80FCFC003FEBFE7C9038007E3848EB7F00A66C137EEB80FE001F5B380FC1 F8381FFFE0001813800038C8FC123CA2123E383FFFF814FF6C14C06C14E06C14F0121F397E0007 F8007C13015A1400A36C1301007EEB03F06CEB07E0390FC01F803903FFFE0038007FF01E287E9A 22>II<1207EA1FC013E0123FA3121F13C0EA0700C7FCA7EAFFE0A3 120FB3A3EAFFFEA30F2B7DAA14>I107 DI<3BFFC07F800FF0 903AC1FFE03FFC903AC783F0F07E3B0FCE03F9C07F903ADC01FB803F01F8D9FF00138001F05BA3 01E05BAF3CFFFE1FFFC3FFF8A3351B7D9A3A>I<38FFC07F9038C1FFC09038C787E0390FCE07F0 9038DC03F813F813F0A313E0AF3AFFFE3FFF80A3211B7D9A26>II<38FFE1FE9038E7FF809038FE07E039 0FF803F8496C7E01E07F140081A2ED7F80A9EDFF00A25DEBF0014A5A01F85B9038FE0FE09038EF FF80D9E1FCC7FC01E0C8FCA9EAFFFEA321277E9A26>I<38FFC3F0EBCFFCEBDC7E380FD8FF13F8 5BA3EBE03C1400AFB5FCA3181B7E9A1C>114 D<3803FE30380FFFF0EA3E03EA7800127000F013 70A27E6C1300EAFFE013FE387FFFC06C13E06C13F0000713F8C613FC1303130000E0137C143C7E A26C13787E38FF01F038F7FFC000C11300161B7E9A1B>I<1370A413F0A312011203A21207381F FFF0B5FCA23807F000AD1438A73803F870000113F03800FFE0EB1F8015267FA51B>I<39FFE03F F8A3000F1303B11407A2140F0007131F3A03F03BFF803801FFF338003FC3211B7D9A26>I<3AFF FE03FF80A33A07F0007000A26D13F000035CEBFC0100015CA26C6C485AA2D97F07C7FCA2148FEB 3F8E14DEEB1FDCA2EB0FF8A36D5AA26D5AA26D5A211B7F9A24>I<3BFFFE7FFC0FFEA33B0FE007 E000E03B07F003F001C0A29039F807F80300031680A23B01FC0EFC0700A2D9FE1E5B000090381C 7E0EA29039FF383F1E017F141C0278133C90393FF01FB8A216F86D486C5AA26D486C5AA36D486C 5AA22F1B7F9A32>I<39FFFC0FFFA33907F003C06C6C485AEA01FC6C6C48C7FCEBFF1E6D5AEB3F F86D5A130FA2130780497E497E131EEB3C7F496C7E496C7ED801E07FEBC00F00036D7E3AFFF01F FF80A3211B7F9A24>I<3AFFFE03FF80A33A07F0007000A26D13F000035CEBFC0100015CA26C6C 485AA2D97F07C7FCA2148FEB3F8E14DEEB1FDCA2EB0FF8A36D5AA26D5AA26D5AA2495AA2EA3807 007C90C8FCEAFE0F130E131E5BEA7C78EA3FE0EA0FC021277F9A24>I E /Fi 18 119 df<127012F812FCA2127C120CA41218A21230A212601240060F7C840E>44 D49 DI56 DI77 D<39FF800FFF13C00007EB01F89038E000607F12061378A27F133E13 1E7FA2EB078014C01303EB01E0A2EB00F01478A2143CA2141E140FA2EC07E0A214031401A2381F 8000EAFFF0156020227EA125>I<3803F020380FFC60381C0EE0EA3803EA7001A2EAE000A21460 A36C1300A21278127FEA3FF0EA1FFE6C7E0003138038003FC0EB07E01301EB00F0A2147012C0A4 6C136014E06C13C0EAF80138EF038038C7FF00EA81FC14247DA21B>83 D97 D<120E12FEA2121E120EAAEB1F80EB7FE0380FC0F0EB0078000E1338143C 141C141EA7141C143C000F1338EB8070EBC1F0380C7FC0EB1F0017237FA21B>II101 D<121C121E123E121E121CC7FC A8120E12FEA2121E120EAFEAFFC0A20A227FA10E>105 D<390E1FC07F3AFE7FE1FF809039C0F3 03C03A1F807E01E0390F003C00000E1338AE3AFFE3FF8FFEA227157F942A>109 D<380E1F8038FE7FC038FFC1E0381F80F0380F0070120EAE38FFE7FFA218157F941B>II114 D<38FFC3FEA2381E00F8000E1360A26C13C0A338038180 A213C300011300A2EA00E6A3137CA31338A217157F941A>118 D E /Fj 17 124 df73 D77 D79 DI97 D 99 D101 DI<15F090387F03F83901FFCF1C3803C1FC390780F818390F007800 48137C001E133C003E133EA7001E133C001F137C6C13786C6C5A380FC1E0380DFFC0D81C7FC7FC 0018C8FCA2121CA2121E380FFFF814FF6C14804814C0391E0007E00038EB01F048EB0070157848 1438A500701470007814F06CEB01E06CEB03C03907C01F003801FFFC38003FE01E2F7E9F21>I< 1207EA0F80121FA2120FEA0700C7FCABEA078012FFA2120F1207B3A6EA0FC0EAFFF8A20D307EAF 12>105 D108 D<380781FE39FF87FF8090388E07C0390F9803E03807B0019038E000F05BA35BB3486C487E3AFF FC1FFF80A2211F7E9E25>110 D<380783E038FF8FF8EB9C7CEA0FB0EA07F0EBE038EBC000A35B B3487EEAFFFEA2161F7E9E19>114 D<3801FC10380FFF30381E03F0EA38004813705A1430A37E 6C1300127EEA3FF06CB4FC6C1380000313E038003FF0EB03F8EB007800C0133CA2141C7EA27E14 186C13386C137038EF01E038C3FFC03880FE00161F7E9E1A>I<13C0A51201A31203A21207120F 121FB512E0A23803C000B01430A83801E060A23800F0C0EB7F80EB1F00142C7FAB19>II123 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin %%EndSetup %%Page: 1 1 bop 457 227 a Fj(Message)21 b(P)n(assing)g(In)n(terface)i({)f(Outline)869 354 y Fi(Marc)16 b(Snir)772 456 y(No)o(v)o(em)o(b)q(er)d(28,)k(1992)75 643 y Fh(In)n(tro)r(duction)75 745 y Fg(The)f(purp)q(ose)h(of)f(this)g(do)q (cumen)o(t)h(is)f(to)g(pro)o(vide)g(a)g(framew)o(ork)f(that)g(w)o(e)h(can)g (use)h(to)e(discuss)i(issues)75 801 y(in)i(the)e(de\014nition)j(of)d(MPI,)h (to)f(list)i(questions)f(w)o(e)f(need)i(to)e(answ)o(er,)h(and)g(prop)q(ose)f (some)h(answ)o(ers.)75 857 y(I)g(plan)g(to)e(use)i(this)g(do)q(cumen)o(t)f (as)g(a)g(framew)o(ork)f(for)h(the)g(discussions)i(in)f(the)f(p)q(oin)o (t-to-p)q(oin)o(t)h(sub-)75 914 y(committee;)d(I)g(broadcast)g(it)g(to)g(the) g(en)o(tire)g(committee,)g(b)q(ecause)h(man)o(y)f(issues)h(are)f(relev)m(an)o (t)g(to)g(the)75 970 y(other)g(sub)q(committees.)75 1113 y Fh(Goals)131 1214 y Fg(1.)22 b(Design)17 b(an)h(application)h(programming)d (in)o(terface)i(\(not)e(necessarily)j(a)e(target)f(for)h(compilers)189 1271 y(or)d(a)h(system)g(implemen)o(tation)h(library\))131 1363 y(2.)22 b(Allo)o(w)15 b(e\016cien)o(t)h(comm)o(unication:)21 b(Av)o(oid)16 b(memory)e(to)h(memory)g(cop)o(ying)g(and)h(allo)o(w)f(o)o(v)o (erlap)189 1420 y(of)e(computation)g(and)h(comm)o(unication)g(and)g(o\017oad) f(to)g(comm)o(unication)h(copro)q(cessor,)g(where)189 1476 y(a)o(v)m(ailable)131 1569 y(3.)22 b(Allo)o(w)15 b(\(but)g(no)h(mandate\))e (extensions)i(for)f(use)g(in)h(heterogeneous)f(en)o(vironmen)o(t.)131 1661 y(4.)22 b(Allo)o(w)15 b(con)o(v)o(enien)o(t)h(C,)f(C++,)g(F77)f(and)i (F90)e(bindings)j(for)e(in)o(terface.)131 1754 y(5.)22 b(Pro)o(vide)15 b(a)g(reliable)i(comm)o(unication)f(in)o(terface:)21 b(User)15 b(need)h(not)f(cop)q(e)h(with)f(comm)o(unication)189 1810 y(failures.)21 b(Suc)o(h)15 b(failures)i(are)d(dealt)i(b)o(y)f(the)h(underlying)h(comm)o (unication)f(subsystem.)131 1902 y(6.)22 b(F)l(o)q(cus)15 b(on)g(a)g(prop)q (osal)h(that)e(can)h(b)q(e)h(agreed)f(up)q(on)h(in)g(6)f(mon)o(ths.)131 1995 y(7.)22 b(De\014ne)10 b(an)g(in)o(terface)h(that)e(is)i(not)f(to)q(o)f (di\013eren)o(t)i(from)e(curren)o(t)h(practice)h(\(PVM/Express/P)o(armacs\)) 131 2087 y(8.)22 b(De\014ne)13 b(an)g(in)o(terface)f(that)g(can)h(b)q(e)h (quic)o(kly)g(implemen)o(ted)g(on)f(man)o(y)f(v)o(endors)g(platforms,)h(with) 189 2144 y(no)i(signi\014can)o(t)h(c)o(hanges)f(in)h(the)f(underlying)j(comm) o(unication)d(and)h(system)f(soft)o(w)o(are.)146 2245 y(Do)f(w)o(e)h(agree)g (on)g(this?)75 2388 y Fh(F)-6 b(ramew)n(ork)75 2489 y Fg(I)11 b(prop)q(ose)g(to)g(consider)g(comm)o(unication)h(op)q(erations)f(as)g (consisting)g(of)g(the)g(follo)o(wing)g(sub)q(op)q(erations:)75 2591 y Ff(INIT\(op)q(eration,)19 b(params,)d(handle\))24 b Fg(Pro)q(cess)15 b(pro)o(vides)f(all)i(relev)m(an)o(t)e(parameters)g(for)g (its)g(par-)189 2647 y(ticipation)19 b(in)f(the)g(comm)o(unication)g(op)q (eration)g(\(data)f(bu\013er,)h(t)o(yp)q(e,)g(participan)o(ts,)g(etc.\).)27 b(A)189 2704 y(handle)16 b(is)g(created)f(that)g(iden)o(ti\014es)h(the)g(op)q (eration.)964 2828 y(1)p eop %%Page: 2 2 bop 75 45 a Ff(ST)l(AR)l(T\(handle\))24 b Fg(The)16 b(comm)o(unication)g(op)q (eration)f(is)h(started)75 139 y Ff(A)-6 b(W)g(AIT\(handle\))24 b Fg(The)15 b(comm)o(unication)h(op)q(eration)g(is)g(completed.)75 232 y Ff(FREE\(handle\))25 b Fg(The)16 b(handle,)g(and)f(asso)q(ciated)g (resources)h(are)f(freed.)75 337 y(Correct)g(in)o(v)o(o)q(cation)h(of)g (these)g(sub)q(op)q(erations)g(is)h(a)e(sequence)i(of)e(the)h(form)f(1\(23\)) 1539 321 y Fe(\003)1558 337 y Fg(4.)21 b(I.e.,)15 b(a)h(handle)75 394 y(need)g(b)q(e)g(created)g(b)q(efore)g(comm)o(unication)g(o)q(ccurs;)g (it)f(can)h(b)q(e)g(reused)g(only)g(after)f(the)g(previous)i(use)75 450 y(has)12 b(completed;)h(and)f(it)g(need)h(to)e(b)q(e)h(freed)g(ev)o(en)o (tually)h(\(of)e(course,)h(one)g(can)g(assume)f(that)g(all)i(handles)75 507 y(are)i(freed)g(at)g(program)f(termination,)h(b)o(y)g(default\).)146 563 y(The)f(c)o(hoice)i(of)e(these)h(sub)q(op)q(erations)g(is)h(somewhat)d (arbitrary)l(.)20 b(The)15 b(justi\014cation)g(is)g(that)f(com-)75 619 y(m)o(unication)k(ma)o(y)f(b)q(e)h(a)g(length)o(y)g(pro)q(cess)f(one)h (desires)h(to)d(o)o(v)o(erlap)i(with)g(computation,)f(hence)i(the)75 676 y(separation)g(of)g(2)h(and)f(3;)i(also)e(comm)o(unication)i(setup)e(ma)o (y)g(in)o(v)o(olv)o(e)h(a)f(signi\014can)o(t)i(o)o(v)o(erhead)e(one)75 732 y(desires)i(to)f(amortize)h(o)o(v)o(er)e(man)o(y)h(successiv)o(e)i(comm)o (unications)f(with)g(the)g(same)f(c)o(haracteristics,)75 789 y(hence)c(the)g(separation)f(of)f(1)h(and)h(4)f(from)f(2)h(and)g(3.)75 932 y Fh(Issues)75 1035 y Fd(Whic)n(h)20 b(op)r(erations?)75 1121 y Fg(SEND)15 b(and)g(RECEIVE)g(for)g(p)q(oin)o(t)g(to)f(p)q(oin)o(t)i (comm)o(unication.)k(Longer)15 b(list)h(for)e(collectiv)o(e)j(comm)o(u-)75 1177 y(nication)g(\(note:)k(SEND)16 b(and)g(RECEIVE)g(are)g(particular)g (case)g(of)g(broadcast,)f(for)g(group)g(of)h(size)h(2;)75 1234 y(this)i(observ)m(ation)g(can)g(b)q(e)g(used)h(to)e(c)o(hec)o(k)h(if)g (de\014nition)i(of)d(collectiv)o(e)i(comm)o(unication)g(seman)o(tics)75 1290 y(are)15 b(consisten)o(t)g(with)h(de\014nition)h(of)e(p)q(oin)o(t-to-p)q (oin)o(t)h(comm)o(unication\).)146 1347 y(One)g(can)f(argue)g(for)f(a)h(SEND) p 684 1347 14 2 v 17 w(RECV)g(\(or)f(EX)o(CHANGE\))h(op)q(eration.)146 1403 y(I)g(prop)q(ose)g(to)g(lea)o(v)o(e)g(out,)f(for)g(the)i(time)f(b)q (eing,)h(things)f(lik)o(e)i(remote)d(pro)q(cedure)i(calls,)g(monitors,)75 1460 y(atomic)h(op)q(erations,)i(semaphores,)e(activ)o(e)h(messages,)g (message)f(handlers,)i(etc.)27 b(This,)19 b(in)f(order)g(to)75 1516 y(fo)q(cus)f(initially)j(on)c(a)h(smaller)g(set)g(of)f(issues,)i(with)f (less)g(dep)q(endencie)q(s)i(on)e(the)g(underlying)i(pro)q(cess)75 1572 y(mo)q(del.)146 1629 y(Opinions?)75 1750 y Fd(What)g(calls)h(are)e(made) h(a)n(v)m(ailable)h(to)f(the)f(user?)75 1836 y Fg(Options:)75 1941 y Ff(\(1+2+3+4\))23 b Fg(blo)q(c)o(king)16 b(send/receiv)o(e)75 2035 y Ff(\(1+2\))i(\(3+4\))23 b Fg(non)o(blo)q(c)o(king)17 b(send/receiv)o(e,)f(follo)o(w)o(ed)f(b)o(y)h(w)o(ait)75 2128 y Ff(\(1\))i(\(2+3\))g(\(4\))24 b Fg(c)o(hannel)16 b(creation;)f(blo)q(c)o (king)i(c)o(hannel)f(send/receiv)o(e;)g(c)o(hannel)h(cancelation)75 2222 y Ff(\(1\))h(\(2\))g(\(3\))h(\(4\))k Fg(c)o(hannel)17 b(creation;)e(non)o(blo)q(c)o(king)h(send/receiv)o(e;)g(w)o(ait;)f(c)o (hannel)h(destruction)146 2327 y(An)o(y)f(more)g(needed?)22 b(An)o(y)15 b(less?)75 2448 y Fd(When)j(is)i(a)f(comm)n(unication)h(op)r (eration)e(completed?)75 2534 y Fg(The)12 b(main)f(distinction)j(is)e(b)q(et) o(w)o(een)f Fc(lo)n(c)n(al)g Fg(termination)g(and)h Fc(glob)n(al)f Fg(termination.)19 b(A)11 b(send)h(terminates)75 2591 y(lo)q(cally)23 b(when)f(data)f(has)h(b)q(een)h(copied)g(out)e(of)g(the)h(sender)g(bu\013er)g (\(formally)l(,)h(when)f(the)f(sender)75 2647 y(cannot)e(a\013ect)g(an)o (ymore)g(the)h(outcome)g(of)f(the)h(send)g(op)q(eration\).)33 b(It)20 b(terminates)g(globally)g(when)75 2704 y(data)d(has)h(b)q(een)h (receiv)o(ed)g(b)o(y)f(the)f(receiv)o(er)i(\(formally)l(,)f(when)h(receiv)o (er)f(has)g(terminated)g(executing)964 2828 y(2)p eop %%Page: 3 3 bop 75 45 a Fg(an)o(y)15 b(op)q(eration)h(that)f(logically)j(precedes)f(the)f (execution)g(of)g(the)g(receiv)o(e\).)22 b(The)16 b(distinction)h(has)f(no)75 102 y(meaning)g(for)e(a)h(receiv)o(e)h({)f(since)i(lo)q(cal)f(termination)g (of)e(a)h(receiv)o(e)h(implies)i(global)d(termination.)146 158 y(Choices:)131 252 y(1.)22 b(T)l(ermination)16 b(is)f(alw)o(a)o(ys)g(lo)q (cal.)131 346 y(2.)22 b(T)l(ermination)e(is)g(either)g(lo)q(cal)h(or)e (global,)i(at)e(user)g(c)o(hoice)i(\(e.g.,)e(user)g(can)h(c)o(ho)q(ose)g(b)q (et)o(w)o(een)189 402 y(sync)o(hronous)15 b(and)g(async)o(hronous)g(comm)o (unication\))131 496 y(3.)22 b(T)l(ermination)16 b(is)f(alw)o(a)o(ys)g (global.)146 590 y(The)21 b(curren)o(t)f(MPI)h(prop)q(osal)g(c)o(ho)q(oses)g (2.)37 b(If)21 b(w)o(e)g(stic)o(k)g(to)f(this)h(c)o(hoice)h(then)f(I)g (suggest)g(that)75 646 y(termination)h(mo)q(de)g(\(global)g(or)f(lo)q(cal\))h (is)g(part)f(of)g(the)h(op)q(eration)f(parameters)g(that)g(are)g(set)g(up)75 703 y(in)g(sub)q(op)q(eration)g(1.)34 b(F)l(urthermore,)21 b(I)f(w)o(ould)h(suggest)e(to)h(o\013er)f(the)h(same)g(option)h(for)e (collectiv)o(e)75 759 y(comm)o(unications.)146 816 y(Another)13 b(issue)h(is)g(whether)g(termination)g(mo)q(de)f(need)h(b)q(e)h(consisten)o (t)e(for)g(all)h(participan)o(ts.)20 b(Cur-)75 872 y(ren)o(t)c(MPI)g(prop)q (osal)g(requires)h(that)e(a)g(sync)o(hronous)h(send)h(b)q(e)g(matc)o(hed)e(b) o(y)h(a)g(sync)o(hronous)g(receiv)o(e)75 928 y(\(either)i(all)h(participan)o (ts)f(use)g(lo)q(cal)h(completion)g(or)e(all)i(use)f(global)g(completion\).) 29 b(There)18 b(seem)g(to)75 985 y(b)q(e)d(no)g(comp)q(elling)i(logical)f (reason)f(for)f(this)h(restriction,)g(and)g(I)g(am)g(not)f(sure)h(it)g(has)g (signi\014can)o(t)g(im-)75 1041 y(plemen)o(tation)i(adv)m(an)o(tages.)k(The) 16 b(alternativ)o(e)g(is)g(to)f(allo)o(w)h(eac)o(h)g(participan)o(t)g(in)h(a) e(comm)o(unication)75 1098 y(op)q(eration)h(to)g(c)o(ho)q(ose)g(either)h(lo)q (cal)h(or)e(global)h(termination)f(\(this)h(b)q(ecomes)g(more)e(signi\014can) o(t)j(for)d(a)75 1154 y(collectiv)o(e)i(op)q(eration,)e(where)g(the)h(c)o (hoice)g(is)g(meaningful)g(at)f(more)f(than)i(one)f(pro)q(cess\).)146 1211 y(I)i(assume)h(lo)q(cal)g(termination)g(for)f(sub)q(op)q(eration)h(1)f ({)g(no)h(sync)o(hronization)g(with)g(another)f(pro-)75 1267 y(cessor)e(is)h(required)g(for)f(this)g(phase.)146 1324 y(Opinions?)146 1380 y(Should)h(w)o(e)f(c)o(ho)q(ose)g(1)g(or)g(3,)f(rather)h(than)g(2?)146 1437 y(Should)g(w)o(e)f(use)g(another)g(mec)o(hanism)g(to)g(c)o(ho)q(ose)g (lo)q(cal)h(and)f(global)h(termination)f(\(e.g.,)f(a)g(global)75 1493 y(\015ag\)?)146 1549 y(Should)g(w)o(e)e(require)h(the)g(c)o(hoice)h(of)e (\\sync)o(hronous")g(to)g(b)q(e)h(done)g(consisten)o(tly)h(b)q(e)f(all)h (participan)o(ts)75 1606 y(in)j(a)f(comm)o(unication?)75 1728 y Fd(When)j(do)r(es)g(a)h(call)h(return?)75 1813 y Fg(The)14 b(ob)o(vious)f(c)o(hoice)i(is:)k(\\A)13 b(call)i(return)e(when)h(the)f (corresp)q(onding)i(sub)q(op)q(eration\(s\))e(terminated".)75 1870 y(An)21 b(alternativ)o(e)f(is)h(to)f(allo)o(w)h(for)f(unsuccessful)i (return)f(as)f(w)o(ell.)37 b(E.g.,)20 b(send)h(can)g(return)f(unsuc-)75 1926 y(cessfully)l(,)d(if)f(the)f(system)g(cannot)g(accept)g(the)h(message,)e (for)h(whatev)o(er)g(reasons;)f(or)h(receiv)o(e)h(returns)75 1983 y(unsuccessfully)f(if)e(the)g(receiv)o(e)h(cannot)e(execute)i(within)g (a)e(set)h(time;)g(and)g(so)g(on.)19 b(This)13 b(is)g(the)g(curren)o(t)75 2039 y(MPI)i(de\014nition.)146 2096 y(My)j(p)q(ersonal)i(c)o(hoice)g(is)g (against)e(unsuccessful)j(return.)32 b(This)20 b(is)f(more)g(adequate)g(for)f (system)75 2152 y(programming,)f(rather)h(than)f(application)j(programming;)e (it)g(in)o(tro)q(duces)h(seman)o(tic)f(dep)q(endencies)75 2209 y(on)h(implemen)o(tation;)i(it)e(forces)f(the)h(user)f(to)g(c)o(hec)o(k)h (for)f(success)i(at)e(eac)o(h)g(op)q(eration,)i(rather)e(then)75 2265 y(deal)j(with)g(failure)g(of)f(comm)o(unication)h(using)g(an)f (exception)i(handling)g(mec)o(hanism.)36 b(Of)20 b(course,)75 2322 y(implemen)o(ters)f(are)f(w)o(elcome)g(to)f(pro)o(vide)i(a)e(reasonable) i(exception)f(handling)i(mec)o(hanism)f(for)e(the)75 2378 y(MPI)e(library)h ({)f(I)h(suggest)e(to)h(lea)o(v)o(e)g(op)q(en)h(the)f(de\014nition)i(of)e (suc)o(h)h(mec)o(hanism.)146 2434 y(Opinions?)75 2556 y Fd(Correctness)i (criteria)75 2642 y Fg(I)e(use)f(the)g(follo)o(wing)h(terminology:)964 2828 y(3)p eop %%Page: 4 4 bop 75 45 a Ff(safe)17 b(program)22 b Fg(A)c(program)f(that)h(should)h (execute)f(successfully)i(in)f(an)o(y)f(implemen)o(tation,)i(inde-)189 102 y(p)q(enden)o(tly)c(of)f(the)h(amoun)o(t)e(of)h(a)o(v)m(ailable)i(system) d(resources.)75 195 y Ff(unsafe)j(program)23 b Fg(A)d(program)f(that)g(will)j (succeed)f(or)f(fail,)i(dep)q(ending)g(on)e(implemen)o(tation)h(or)189 252 y(a)o(v)m(ailable)c(system)d(resources.)75 346 y Ff(erroneous)i(program) 22 b Fg(A)16 b(program)e(that)g(will)j(alw)o(a)o(ys)e(fail.)146 452 y(Examples)g(\(in)o(v)o(olving)h(t)o(w)o(o)e(pro)q(cessors)h(with)h(ids)g (1)f(and)g(2,)f(and)i(using)g(MPI)f(lik)o(e)h(syn)o(tax\))146 508 y(The)f(follo)o(wing)h(program)e(is)i(safe,)e(and)i(should)g(alw)o(a)o (ys)e(succeed.)75 659 y Fb(IF)24 b(\(GETID\(\))e(==)i(1\))g(THEN)147 715 y(CALL)f(CSEND\(mode=blocking,)e(buf=sendbuf,)h(dest=2,)h(type=0,)g (len=1000\))147 772 y(CALL)g(CRECV\(mode=blocking,)e(buf=recdbuf,)h (source=2,)h(type=0,)g(len=1000\))75 884 y(ELSE)g(!)h(\(GETID\(\))f(==)g(2\)) 147 941 y(CALL)g(CRECV\(mode=blocking,)e(buf=recbuf,)h(source=1,)h(type=0,)g (len=1000\))147 997 y(CALL)g(CSEND\(mode=blocking,)e(buf=sendbuf,)h(dest=1,)h (type=0,)g(len=1000\))146 1148 y Fg(The)15 b(follo)o(wing)h(program)e(is)i (erroneous,)e(and)i(should)g(alw)o(a)o(ys)e(fail.)75 1310 y Fb(IF)24 b(\(GETID\(\))e(==)i(1\))g(THEN)147 1367 y(CALL)f (CSEND\(mode=synchronous,)e(buf=sendbuf,)h(dest=2,)h(type=0,)g(len=1000\))147 1423 y(CALL)g(CRECV\(mode=synchronous,)e(buf=recdbuf,)h(source=2,)g(type=0,)h (len=1000\))75 1536 y(ELSE)47 b(!)24 b(\(GETID\(\))f(==)g(2\))147 1593 y(CALL)g(CSEND\(mode=synchronous,)e(buf=sendbuf,)h(dest=1,)h(type=0,)g (len=1000\))147 1649 y(CALL)g(CRECV\(mode=synchronous,)e(buf=recbuf,)h (source=1,)h(type=0,)g(len=1000\))146 1755 y Fg(The)12 b(follo)o(wing)h (program)d(is)j(unsafe,)f(and)h(ma)o(y)e(succeed)i(or)f(fail,)h(dep)q(ending) h(on)e(implemen)o(tation.)75 1918 y Fb(IF)24 b(\(GETID\(\))e(==)i(1\))g(THEN) 147 1975 y(CALL)f(CSEND\(mode=blocking,)e(buf=sendbuf,)h(dest=2,)h(type=0,)g (len=1000000\))147 2031 y(CALL)g(CRECV\(mode=blocking,)e(buf=recdbuf,)h (source=2,)h(type=0,)g(len=1000000\))75 2144 y(ELSE)g(!)h(\(GETID\(\))f(==)g (2\))147 2200 y(CALL)g(CSEND\(mode=blocking,)e(buf=sendbuf,)h(dest=1,)h (type=0,)g(len=1000000\))147 2257 y(CALL)g(CRECV\(mode=blocking,)e (buf=recbuf,)h(source=1,)h(type=0,)g(len=1000000\))146 2363 y Fg(This)14 b(program)f(can)h(succeed)h(only)g(if)f(the)g(system)g(has)g (su\016cien)o(t)g(bu\013er)g(space)g(to)g(bu\013er)g(1)f(MgB)75 2420 y(of)i(data.)146 2476 y(The)j(strict)g(de\014nition)i(of)e(\\safe")f (needs)i(to)f(b)q(e)g(somewhat)g(temp)q(ered)h(in)g(practice.)29 b(A)18 b(correct)75 2532 y(sequen)o(tial)13 b(program)d(ma)o(y)h(fail)i(b)q (ecause)f(of)g(time)g(limits)h(or)e(other)g(resource)h(restrictions.)19 b(Lik)o(ewise,)13 b(a)75 2589 y(safe)h(parallel)i(program)d(ma)o(y)l(,)g (exceptionally)l(,)k(exceed)e(system)f(resources.)19 b(Still,)d(common)e (sense)h(can)75 2645 y(di\013eren)o(tiate)i(b)q(et)o(w)o(een)g(resources)f (that)g(are)h(\\practically)g(un)o(b)q(ounded",)h(i.e.)25 b(with)17 b(limits)g(that)f(are)75 2702 y(unlik)o(ely)k(to)d(b)q(e)h(normally)g(encoun) o(tered)g(b)o(y)g(correct)f(programs,)g(and)g(resources)h(that)f (\\practically)964 2828 y(4)p eop %%Page: 5 5 bop 75 45 a Fg(b)q(ounded",)23 b(i.e.)37 b(with)21 b(limits)h(one)f(encoun)o (ters)g(frequen)o(tly)g(with)g(correct)g(programs,)f(and)h(whic)o(h)75 102 y(often)13 b(impinges)i(on)f(program)e(p)q(ortabilit)o(y)l(.)20 b(A)14 b(formal)f(de\014nition)i(will)h(assume)d(that)g(some)g(resources)75 158 y(are)19 b(unlimited,)k(and)c(implemen)o(tations)i(will)g(striv)o(e)f(to) f(pro)o(vide)h(a)f(reasonable)h(appro)o(ximation)f(of)75 214 y(this)f(assumption.)26 b(F)l(urthermore,)17 b(the)g(user)h(should)g(b)q(e)g (able)g(to)f(query)g(and)g(p)q(ossibly)i(con)o(trol)e(an)o(y)75 271 y(implemen)o(tation)f(sp)q(eci\014c)i(resource)d(limitation)h(that)f (a\013ects)f(program)g(b)q(eha)o(vior.)146 327 y(One)k(need)g(to)f(allo)o(w)h (\(indeed,)h(one)f(cannot)f(disallo)o(w\))h(v)o(endors)g(to)e(supp)q(ort)i (some)f(unsafe)h(pro-)75 384 y(grams)c(as)h(w)o(ell.)21 b(The)15 b(requiremen)o(ts)h(there)f(should)h(b)q(e)g(that)143 478 y Fa(\017)23 b Fg(The)15 b(seman)o(tics)g(of)g(successful)i(program)d (execution)i(b)q(e)g(w)o(ell-de\014ned)143 571 y Fa(\017)23 b Fg(The)16 b(system)f(pro)o(vide)i(clear)f(information)h(and)f(p)q(ossibly)h (con)o(trol)f(on)g(those)g(parameters)f(that)189 628 y(determine)h(the)f(b)q (eha)o(vior)h(of)f(unsafe)g(programs)f(\(e.g.,)g(bu\013er)h(sizes\).)146 722 y(A)g(handle)j(is)e Fc(active)g Fg(\(at)f(a)g(pro)q(cess\))h(from)f(the)h (start)f(of)g(sub)q(op)q(eration)i(1)e(to)g(the)h(completion)h(of)75 778 y(sub)q(op)q(eration)f(4)g(for)f(this)h(handle.)22 b(A)15 b(comm)o(unication)i(is)f Fc(active)f Fg(\(at)g(a)g(pro)q(cess\))h(from)e (the)i(start)e(of)75 835 y(sub)q(op)q(eration)i(2)f(to)g(the)g(completion)h (of)f(sub)q(op)q(eration)h(3.)k(I)15 b(suggest)g(the)g(follo)o(wing)h(design) g(p)q(oin)o(t:)131 941 y(1.)22 b(The)13 b(n)o(um)o(b)q(er)g(of)g(activ)o(e)g (comm)o(unication)g(handles)i(p)q(er)e(pro)q(cess)g(is)h(\\practically)g(un)o (b)q(ounded".)131 1035 y(2.)22 b(The)d(n)o(um)o(b)q(er)g(of)f(initiated)j (comm)o(unications)e(p)q(er)h(pro)q(cessor)f(is)g(\\practically)h(un)o(b)q (ounded".)189 1091 y(\(Note)12 b(that)h(there)g(ma)o(y)g(b)q(e)g(only)h(one)g (activ)o(e)f(comm)o(unication)h(op)q(eration)f(p)q(er)h(activ)o(e)f (handle\).)131 1185 y(3.)22 b(The)16 b(amoun)o(t)g(of)g(a)o(v)m(ailable)i (system)d(bu\013er)i(space)f(is)h(limited)h({)e(a)g(safe)g(program)f(cannot)h (rely)189 1241 y(on)f(it.)146 1348 y(This)g(design)h(p)q(oin)o(t)g(has)f(the) h(follo)o(wing)f(implications:)131 1454 y(1.)22 b(A)11 b(t)o(yp)q(e)g(1)f(or) h(t)o(yp)q(e)g(2)g(sub)q(op)q(eration)g(will)i(alw)o(a)o(ys)d(complete,)i (indep)q(enden)o(tl)q(y)h(of)e(other)g(pro)q(cesses)189 1510 y(activities.)21 b(Of)15 b(course,)g(the)g(same)g(is)h(true)f(for)g(a)f(t)o (yp)q(e)i(4)f(sub)q(op)q(eration.)131 1604 y(2.)22 b(The)15 b(successful)h(completion)h(of)d(a)h(t)o(yp)q(e)g(3)g(sub)q(op)q(eration)h (ma)o(y)e(dep)q(end)j(on)e(the)g(participation)189 1661 y(of)h(other)g(pro)q (cesses.)24 b(A)17 b(send)g(ma)o(y)f(b)q(e)h(blo)q(c)o(k)o(ed)g(un)o(til)h(a) e(matc)o(hing)h(receiv)o(e)g(o)q(ccurs.)24 b(And,)17 b(of)189 1717 y(course,)h(a)g(receiv)o(e)h(ma)o(y)e(not)h(complete)g(un)o(til)i(a)d (matc)o(hing)h(send)h(o)q(ccurs.)29 b(\(This)18 b(extends)h(to)189 1774 y(collectiv)o(e)e(comm)o(unication.\))146 1880 y(T)l(o)e(this,)h(w)o(e)f (need)i(add)f(p)q(ositiv)o(e)g(requiremen)o(ts)g(for)f(progress)g(and/or)h (fairness.)21 b(Let's)16 b(sa)o(y)f(that)75 1936 y(a)k(comm)o(unication)h(is) f Fc(enable)n(d)f Fg(\(at)h(a)f(pro)q(cess\))h(if)h(a)f(matc)o(hing)g(comm)o (unication)h(is)f(activ)o(e)h(\(i.e.)31 b(a)75 1993 y(matc)o(hing)16 b(receiv)o(e)h(for)e(a)h(send)g(or)f(a)h(matc)o(hing)g(send)g(for)g(a)f (receiv)o(e\).)23 b(An)16 b(enabled)h(comm)o(unication)75 2049 y(ma)o(y)g(b)q(ecome)i(disabled)h(either)e(b)q(ecause)h(it)g(completes)f (successfully)i(\(the)e(comm)o(unication)h(o)q(ccurs)75 2106 y(and)d(sub)q(op)q(eration)h(3)e(terminates\))g(or)h(b)q(ecause)g(the)g(matc) o(hing)g(comm)o(unication)h(b)q(ecomes)f(inactiv)o(e)75 2162 y(\(e.g.,)e(a)h(receiv)o(e)h(that)e(matc)o(hes)h(one)h(send)g(ma)o(y)e(b)q(e) i(satis\014ed)g(b)o(y)f(another)g(send,)h(th)o(us)f(disabling)i(the)75 2219 y(\014rst)e(send\).)75 2325 y Ff(progress)21 b Fg(If)14 b(some)f(comm)o(unication)h(in)g(the)f(system)g(is)h(enabled)h(then)e(some)g (comm)o(unication)h(ev)o(en-)189 2381 y(tually)i(o)q(ccurs)f(in)h(the)g (system.)75 2475 y Ff(fairness)22 b Fg(An)c(activ)o(e)f(comm)o(unication)h (either)g(completes)g(successfully)h(or)d(b)q(ecomes)i(p)q(ermanen)o(tly)189 2532 y(disabled.)964 2828 y(5)p eop %%Page: 6 6 bop 146 45 a Fg(W)l(e)19 b(assume)g(in)h(these)f(de\014nitions)i(that)d(eac)o (h)h(pro)q(cess)g(in)o(v)o(ok)o(e)g(sub)q(op)q(erations)h(in)g(the)f(correct) 75 102 y(order,)c(i.e.)23 b(1\(23\))393 85 y Fe(\003)411 102 y Fg(4.)f(Of)16 b(course,)f(a)h(program)f(where)h(an)g(activ)o(e)g(comm)o (unication)g(b)q(ecomes)h(p)q(erma-)75 158 y(nen)o(tly)f(disabled)h(\(e.g.,)c (a)i(send)h(is)f(left)h(forev)o(er)e(with)i(no)f(matc)o(hing)g(receiv)o(e\))h (is)f(erroneous,)g(and)g(will)75 214 y(not)g(terminate)g(successfully)l(.)146 271 y(The)20 b(enforcemen)o(t)g(of)g(these)g(t)o(w)o(o)f(conditions)i (requires)g(a)f(\015o)o(w)g(con)o(trol)g(proto)q(col:)29 b(A)21 b(pro)q(cess)75 327 y(should)c(not)f(b)q(e)h(prev)o(en)o(ted)g(from)e (accepting)j(one)e(message)g(that)g(w)o(as)f(sen)o(t)h(b)q(ecause)i(its)e (bu\013ers)g(are)75 384 y(full)21 b(with)f(other)g(messages,)g(p)q(ossibly)h (sen)o(t)f(earlier.)34 b(When)21 b(a)e(receiv)o(e)i(matc)o(hes)e(sev)o(eral)h (p)q(ossible)75 440 y(messages)d(\(e.g.,)f(a)h(receiv)o(e)h(with)g(a)f (\\don't)f(care")h(sender)h(\014eld\))g(then)g(alternativ)o(e)g(options)f (should)75 497 y(b)q(e)f(considered)g(fairly)g(\(e.g.,)e(b)o(y)h(using)h(an)f (LR)o(U)h(p)q(olicy)h(when)e(considering)i(matc)o(hing)e(senders\).)146 553 y(A\014cionados)d(of)f(formalism)h(will)i(note)d(that)g(I)h(used)g(the)g (w)o(eak)o(est)f(de\014nition)j(of)d(fairness)h(\(ev)o(en)o(tual)75 610 y(fairness\),)j(rather)f(than)h(b)q(ounded)i(fairness)f(or)e(other)h (alternativ)o(es.)146 666 y(Examples:)146 723 y(The)g(follo)o(wing)h (programs)e(are)h(safe,)f(and)i(should)g(succeed:)75 823 y Fb(!)24 b(assume)f(only)g(two)h(processes,)e(numbered)h(1)h(and)f(2)75 936 y(IF)h(\(GETID\(\))e(==)i(1\))g(THEN)147 992 y(DO)f(I=0,)g(N)218 1048 y(CALL)g(CSEND\(mode=nonblocking,)e(buf=buf\(I\),)i(dest=2,)f(type=I,)h (len=1000\))147 1105 y(END)g(DO)75 1218 y(ELSE)g(!)h(\(GETID\(\))f(==)g(2\)) 147 1274 y(DO)g(I=0,)g(N)218 1331 y(CALL)g(CRECV\(mode=nonblocking,)e (buf=buf\(I\),)i(dest=2,)f(type=N-I,)h(len=1000\))147 1387 y(END)g(DO)146 1487 y Fg(Pro)q(cess)16 b(1)g(uses)h(non)o(blo)q(c)o(king)h (sends)f(to)f(send)h(a)f(sequence)i(of)e(messages)g(to)f(pro)q(cess)i(2;)g (pro)q(cess)75 1544 y(2)d(receiv)o(es)h(these)f(messages)f(in)i(the)f(rev)o (erse)g(order,)g(using)h(non)o(blo)q(c)o(king)g(receiv)o(es.)20 b(In)15 b(practice,)g(suc)o(h)75 1600 y(program)e(ma)o(y)g(fail)h(if)g(the)g (n)o(um)o(b)q(er)g(of)g(p)q(ending)h(messages)f(implied)i(b)o(y)d(this)i (program)d(is)i(larger)g(than)75 1657 y(the)20 b(system)g(limit.)36 b(But)21 b(the)f(system)g(limit)h(should)h(b)q(e)e(large)h(enough)f(as)g(to)g (mak)o(e)f(suc)o(h)i(failure)75 1713 y(extremely)16 b(unlik)o(ely)l(.)75 1813 y Fb(!)24 b(Assume)f(a)g(large)h(number)f(of)g(processes)75 1926 y(!)h(Consumer)e(code)75 1983 y(IF)i(\(GETID\(\))e(==)i(1\))g(THEN)123 2039 y(DO)f(I=)h(1,)f(INFOG\(\))194 2095 y(CALL)h(CRECV\(mode=blocking,)d (buf=buf\(I\),)h(source=I,)h(type=0,)g(len=1000\))123 2152 y(END)g(DO)75 2265 y(!)h(Producers)75 2321 y(ELSE)123 2378 y(CALL)f(CSEND\(mode=blocking,)e(buf=sendbuf,)h(dest=1,)h(type=0,)g (len=1000\))146 2478 y Fg(All)17 b(pro)q(cesses)f(send)g(a)f(message)h(to)f (pro)q(cess)h(1.)21 b(Pro)q(cess)15 b(1)h(receiv)o(es)g(these)g(messages)f (in)i(a)e(\014xed)75 2534 y(order.)31 b(A)20 b(\015o)o(w)e(con)o(trol)h (proto)q(col)g(is)h(needed)g(to)f(mak)o(e)f(sure)h(that)g(bu\013ers)g(at)f (pro)q(cess)i(1)f(will)h(not)75 2591 y(o)o(v)o(er\015o)o(w.)e(The)c(sending)h (pro)q(cesses)f(ma)o(y)g(b)q(e)g(dela)o(y)o(ed)h(un)o(til)g(the)f(receiv)o(e) g(pro)q(cess)g(is)h(ready)f(to)f(accept)75 2647 y(their)j(message.)146 2704 y(Consider)f(the)h(follo)o(wing)g(implemen)o(tation)g(alternativ)o(es)f (for)g(this)h(co)q(de:)964 2828 y(6)p eop %%Page: 7 7 bop 143 45 a Fa(\017)23 b Fg(Messages)18 b(are)h(sen)o(t)g(immediately)i(and) e(stored)g(in)h(the)f(ph)o(ysical)i(memory)d(of)h(the)g(receiving)189 102 y(no)q(de)g({)g(no)g(pro)o(vision)g(for)g(retry)l(.)31 b(This)19 b(is)h(a)e(bad)i(implemen)o(tation)g(since)g(ph)o(ysical)g(storage) 189 158 y(limits)c(are)f(lik)o(ely)i(to)e(b)q(e)g(encoun)o(tered.)143 252 y Fa(\017)23 b Fg(Messages)c(are)g(sen)o(t)h(immediately)h(and)f(stored)f (in)i(the)f(receiv)o(er)g(ph)o(ysical)i(memory)l(.)33 b(If)20 b(the)189 308 y(receiv)o(er)e(is)g(unable)h(to)f(receiv)o(e)g(the)g(message,) g(the)f(send)i(is)f(retried.)28 b(This)19 b(is)f(a)f(correct,)h(but)189 365 y(p)q(ossibly)e(ine\016cien)o(t)h(implemen)o(tation.)143 459 y Fa(\017)23 b Fg(A)16 b(\\request-to-send")h(proto)q(col)g(is)g(used;)h (the)f(receiv)o(er)g(allo)o(ws)g(the)g(send)g(to)g(pro)q(ceed)g(only)g(if)189 515 y(it)g(has)g(storage)f(for)h(the)g(message.)26 b(The)18 b(receiv)o(ers)g(need)g(to)e(store)h(information)g(on)h(p)q(ending,)189 571 y(unsatis\014ed)13 b(requests)g(to)f(send.)19 b(The)13 b(maxim)o(um)f(n)o(um)o(b)q(er)h(of)f(p)q(ending)j(requests)d(at)g(a)g(no)q (de)h(is)g(a)189 628 y(system)f(limit)i(that)e(should)i(b)q(e)f(set)f(large)h (enough)g(as)f(to)g(b)q(e)i(considered)g(\\practically)g(in\014nite".)146 722 y(Opinions?)146 778 y(Should)k(w)o(e)e(b)q(e)h(ev)o(en)h(more)e (restrictiv)o(e)h(and)g(require)g(that)g(safe)f(programs)f(run)i(correctly)l (,)h(ev)o(en)75 835 y(when)c(exceeding)h(\\v)o(ery)d(large")h(resource)h (limits?)21 b(\(Or)13 b(should)i(w)o(e)e(a)o(v)o(oid)g(to)f(burden)j(the)e (implemen-)75 891 y(tation)i(with)g(requiremen)o(ts)h(to)f(handle)h(what)f (most)f(lik)o(ely)j(are)e(pathologic)h(programs?\))146 948 y(Should)h(w)o(e)f(try)g(to)g(b)q(e)h(more)f(formal)g(in)h(de\014ning)h(when) f(some)f(of)g(the)h(\\unsafe")f(programs)f(will)75 1004 y(run)h(to)g (completions?)24 b(\(Can)16 b(w)o(e)g(do)g(it)g(without)g(b)q(eing)i(more)d (sp)q(eci\014c)j(ab)q(out)e(the)h(implemen)o(tation)75 1060 y(mo)q(del?\))146 1117 y(Should)f(w)o(e)f(require)h(fairness?)21 b(Should)c(w)o(e)d(ha)o(v)o(e)h(a)g(stronger)f(requiremen)o(t)i(of)f (fairness?)75 1260 y Fh(What)23 b(are)g(the)g(comm)n(unication)d(parameters?) 75 1363 y Fd(Message)e(data)i(\(bu\013er\))131 1449 y Fg(1.)i(scalar)15 b(\(b)o(yte/halfw)o(ord/w)o(ord/doublew)o(ord\),)e(presumably)j(coming)f (from)g(a)g(register)131 1543 y(2.)22 b(Con)o(tiguous)15 b(space)g(in)h (memory)f(\(initial)i(address/length\))131 1637 y(3.)22 b(Bu\013er)15 b(with)g(constan)o(t)g(stride)g(\(initial)i(address/size)f(of)f(eac)o(h)g (blo)q(c)o(k/stride/n)o(um)o(b)q(er)i(blo)q(c)o(ks\))131 1731 y(4.)22 b(A)15 b(t)o(yp)q(ed)g(message)g(\(of)g(a)f(t)o(yp)q(e)i(supp)q (orted)g(in)g(the)f(domain)g(language\).)131 1824 y(5.)22 b(A)d(union)i(of)d (sev)o(eral)i(messages)f(\(p)q(ossibly)i(sp)q(eci\014ed)g(b)o(y)e(an)h(arra)o (y)e(of)h(p)q(oin)o(ters)h(to)e(message)189 1881 y(descriptors\).)g(The)13 b(seman)o(tics)f(of)f(a)h(union)h(is)f(probably)h(conjunctiv)o(e)f(at)g(the)g (sender)g(end)h(\(send)189 1937 y(all\);)g(at)f(the)h(receiv)o(e)g(end,)h(it) f(ma)o(y)e(b)q(e)j(conjunctiv)o(e)f(\(receiv)o(e)g(all\))h(or)e(disjunctiv)o (e)i(\(receiv)o(e)f(an)o(y\).)146 2044 y(The)j(curren)o(t)g(MPI)f(prop)q (osal)i(has)e(2)h(and)g(3)g(\(2)f(is)h(really)h(a)f(particular)g(case)g(of)g (3\).)21 b(Do)15 b(w)o(e)h(w)o(an)o(t)75 2100 y(more?)k(In)c(particular,)g (do)f(w)o(e)g(w)o(an)o(t)f(t)o(yp)q(ed)h(messages?)20 b(Ho)o(w)15 b(general?)75 2222 y Fd(Groups)k(and)g(con)n(text)75 2307 y Fg(Messages)c(are)g(tagged)g(b)o(y)g(pro)q(cess)h(names)f(\(sender)h(and)f (receiv)o(er\))h(and)g(t)o(yp)q(e)f(names.)21 b(Both)15 b(names)75 2364 y(spaces)g(can)f(b)q(e)i(\015at)e(or)g(structured.)20 b(If)14 b(a)h(\015at)f(pro)q(cess)h(name)f(space)h(is)g(used,)g(then)g(a)f (pro)q(cess)h(names)75 2420 y(is)e(an)f(in)o(teger)g(\(from)f(0)h(to)f(n)o (um)o(b)q(er)p 687 2420 14 2 v 17 w(of)p 741 2420 V 16 w(pro)q(cesses-1,)i (for)e(C)h(think)o(ers\).)19 b(If)13 b(a)e(structured)i(pro)q(cess)f(name)75 2477 y(space)18 b(is)g(used)f(then)h(a)f(pro)q(cess)h(name)f(is)h(a)f(pair)h (of)f(the)g(form)g(\(group,)g(rank\).)26 b(Similarly)19 b(in)f(a)f(\015at)75 2533 y(t)o(yp)q(e)g(name)h(space,)g(a)f(t)o(yp)q(e)g(is)h(a)f(n)o(um)o(b)q (er)h(in)g(a)f(range;)h(in)g(a)f(structured)g(t)o(yp)q(e)h(name)f(space,)h (it)f(is)h(a)75 2590 y(pair)e(of)e(the)i(form)e(\(group,)g(lo)q(cal)p 659 2590 V 18 w(t)o(yp)q(e\).)146 2646 y(The)21 b(usefulness)h(of)f(groups)f (seem)h(clear)h(in)f(the)g(con)o(text)g(of)f(collectiv)o(e)j(op)q(erations,)f (where)f(a)75 2703 y(groupid)c(is)g(a)f(handle)h(to)f(sp)q(ecify)h(the)g(set) f(of)f(participan)o(ts)i(in)g(a)f(collectiv)o(e)i(comm)o(unication)f(\(Note:) 964 2828 y(7)p eop %%Page: 8 8 bop 75 45 a Fg(some)15 b(of)h(the)f(discussions)j(on)d(groups)g(seem)h(to)f (assume)h(that)f(a)g(group)g(is)i(alw)o(a)o(ys)e(represen)o(ted)h(b)o(y)f(a) 75 102 y(mem)o(b)q(ership)f(list)g(that)f(is)h(presen)o(t)f(at)f(eac)o(h)i (group)f(mem)o(b)q(er.)19 b(While)14 b(is)g(the)f(represen)o(tation)h(lik)o (ely)h(to)75 158 y(b)q(e)g(used)h(on)e(small)i(systems,)e(it)h(is)g(b)o(y)g (no)g(means)f(the)h(unique)h(p)q(ossible)h(one.)j(An)o(y)15 b(connected)g(graph)75 214 y(structure)h(can)h(b)q(e)g(used)g(to)f(k)o(eep)h (trac)o(k)f(of)g(group)g(mem)o(b)q(ership,)i(with)f(ob)o(vious)f(tradeo\013s) g(b)q(et)o(w)o(een)75 271 y(storage)h(and)h(p)q(erformance.)28 b(A)18 b(complete)g(list)h(at)e(eac)o(h)h(no)q(de)h(is)f(one)g(extreme)g(of)g (this,)g(namely)l(,)h(a)75 327 y(complete)g(graph.)28 b(Therefore,)19 b(a)e(group)h(handle)i(is)e(a)g(more)g(abstract)f(and)h(more)g(general)g (concept)75 384 y(than)d(a)g(list)h(of)f(group)g(mem)o(b)q(ers.\))146 440 y(The)h(collectiv)o(e)j(comm)o(unication)e(sub)q(committee)g(has)g(to)f (deal)h(with)g(mec)o(hanisms)g(for)f(creating)75 497 y(and)j(destro)o(ying)f (groups)g(\(for)g(the)g(record,)h(I)g(fa)o(v)o(or)e(dynamic)i(group)g (creation)f(b)q(oth)h(b)o(y)f(explicitly)75 553 y(listing)j(group)f(mem)o(b)q (ers)f(and)h(b)o(y)g(partitioning)h(an)e(existing)i(group.)33 b(I)20 b(do)g(not)f(fa)o(v)o(or)g(a)g(complex)75 610 y(stac)o(k)e(or)h(tree)g (mec)o(hanism)h(to)e(manage)h(groups)f(and)i(na)o(vigate)e(from)h(one)g (group)g(to)f(another)h({)g(at)75 666 y(least)d(not)g(in)h(the)f(basic)h(MPI) f(prop)q(osal\).)146 723 y(In)i(the)f(con)o(text)g(of)g(p)q(oin)o(t-to-p)q (oin)o(t)h(comm)o(unication,)g(the)f(issue)h(is)g(whether)g(w)o(e)f(w)o(an)o (t)f(to)h(use)g(a)75 779 y(structured)f(name)g(space)h(for)e(t)o(yp)q(e)i (and)f(pro)q(cess)h(argumen)o(ts)e(and,)h(if)h(so,)e(ho)o(w.)146 835 y(The)21 b(adv)m(an)o(tage)h(of)f(ha)o(ving)h(a)f(structured)h(name)g (space)f(for)h(pro)q(cesses)g(is)g(that)f(it)h(simpli\014es)75 892 y(com)o(bining)14 b(of)e(distinct)i(parallel)g(co)q(des)f(in)h(a)e(lo)q (osely)i(coupled)g(application:)20 b(Eac)o(h)13 b(functional)h(subset)75 948 y(can)j(use)h(message)f(passing)h(within)g(its)g(con)o(text,)f(with)g(no) h(c)o(hanges)f(in)h(the)f(lo)q(cal)i(co)q(de.)27 b(The)17 b(same)75 1005 y(concern)k(for)f(mo)q(dularit)o(y)h(militates)h(for)e(a)g(structured)g (t)o(yp)q(e)h(space.)36 b(This)21 b(do)q(es)g(not)g(necessarily)75 1061 y(implies)j(or)d(requires)h(a)f(stac)o(k)g(mec)o(hanism)h(for)f (managing)g(groups,)i(or)e(a)g(tree)g(structure)h(among)75 1118 y(groups.)146 1174 y(First,)14 b(t)o(w)o(o)g(remarks:)143 1268 y Fa(\017)23 b Fg(The)c(preexisting)h(group)f Fb(ALL)f Fg(that)g(includes)j(all)f(pro)q(cesses)g(pro)o(vides)f(a)g(global)g(con)o (text.)31 b(If)189 1324 y(all)17 b(comm)o(unications)f(use)h(this)f(con)o (text,)f(then)i(one)f(defaults)g(to)g(\015at)f(name)h(space.)23 b(With)16 b(the)189 1381 y(righ)o(t)g(syn)o(tax,)h(the)g(existence)h(of)f(a)g (structured)g(name)g(space)g(can)g(b)q(e)h(ignored)g(b)o(y)f(users)g(that)189 1437 y(prefer)e(a)g(\015at,)f(global)i(name)f(space.)143 1531 y Fa(\017)23 b Fg(A)c(structured)f(name)h(space)g(can)g(alw)o(a)o(ys)g(fak)o (ed,)g(with)g(no)g(c)o(hanges)g(in)g(the)g(comm)o(unication)189 1588 y(op)q(erations:)g(Replace)d(eac)o(h)e Fb(process)p 868 1588 15 2 v 16 w(id)g Fg(actual)h(parameter)e(b)o(y)h(a)g(a)g(reference)h(to) e(a)h(function)189 1644 y Fb(process\(rank,)22 b(group)p 646 1644 V 16 w(id\))p Fg(;)c(replace)h(eac)o(h)e Fb(type)h Fg(actual)f (parameter)g(b)o(y)h(a)f(reference)h(to)f(a)189 1701 y(function)f Fb(type\(local)p 610 1701 V 15 w(type,)24 b(group)p 889 1701 V 16 w(id\))p Fg(.)146 1794 y(F)l(or)16 b(simplicit)o(y)l(,)j(I)e(shall)h (assume)f(the)f(same)h(group)f(con)o(text)h(is)g(used)g(b)q(oth)g(for)f (relativ)o(e)i(pro)q(cess)75 1851 y(n)o(um)o(b)q(ering)h(and)g(for)f(relativ) o(e)h(t)o(yp)q(e)g(naming.)30 b(I)19 b(b)q(eliev)o(e)h(this)f(simpli\014es)i (the)d(discussion,)j(without)75 1907 y(in)o(tro)q(ducing)13 b(undue)g(restrictions.)20 b(A)12 b(con)o(text)f(for)g(a)h(comm)o(unication)g (op)q(eration)h(can)f(b)q(e)g(established)75 1964 y(in)k(t)o(w)o(o)e(w)o(a)o (ys:)131 2070 y(1.)22 b(The)f(curren)o(t)h(scop)q(e)g(\(curren)o(t)f(group\)) g(of)g(a)g(comm)o(unication)h(op)q(eration)g(is)g(implicit;)27 b(it)21 b(is)189 2126 y(established)16 b(b)o(y)g(the)f(last)g(executed)h(con) o(text)f(con)o(trolling)h(call)g(\()p Fb(MPI)p 1401 2126 V 17 w(PUSHC)e Fg(and)i Fb(MPI)p 1713 2126 V 16 w(POPC)f Fg(in)189 2183 y(the)e(curren)o(t)h(draft\).)k(Th)o(us,)c(an)o(y)f(pro)q(cess)h(n)o(um) o(b)q(er)h(is)f(understo)q(o)q(d)g(to)f(b)q(e)i(relativ)o(e)f(rank)f(in)i (the)189 2239 y(curren)o(t)g(group,)f(and)h(an)o(y)g(t)o(yp)q(e)g(name)g(is)h (understo)q(o)q(d)f(to)g(b)q(e)g(relativ)o(e)h(to)e(curren)o(t)h(group,)g (and)189 2296 y(do)q(es)j(not)f(con\015icts)i(with)f(t)o(yp)q(e)h(names)e (used)i(within)g(other)f(groups.)27 b(Of)18 b(course,)h(the)f(initial)189 2352 y(scop)q(e)d(is)g Fb(ALL)p Fg(,)e(and)i(the)g(system)f(defaults)h(to)f (a)g(\015at)g(name)h(space)g(if)g(scop)q(e)g(setting)f(commands)189 2409 y(are)h(nev)o(er)g(used.)131 2503 y(2.)22 b(The)g(con)o(text)g(of)f(a)h (comm)o(unication)h(op)q(erations)f(need)h(to)f(b)q(e)h(explicitly)i(sp)q (eci\014ed)f(in)f(the)189 2559 y(comm)o(unication)15 b(op)q(eration)f (itself.)21 b(A)14 b(pro)q(cess)h(ma)o(y)e(b)q(elong)j(to)d(sev)o(eral)i (groups)f(at)f(an)o(y)h(p)q(oin)o(t)189 2615 y(in)19 b(time,)g(and)g(ma)o(y)f (use)h(an)o(y)f(of)g(these)h(groups)f(as)g(scop)q(e)h(for)f(a)g(comm)o (unication)h(op)q(eration.)189 2672 y(Again,)14 b(the)f(initial)j(scop)q(e)e Fb(ALL)f Fg(is)h(prede\014ned,)h(and)f(can)g(b)q(e)g(used)g(as)g(a)f(\015at)g (name)h(space.)19 b(\(In)14 b(a)964 2828 y(8)p eop %%Page: 9 9 bop 189 45 a Fg(language)13 b(where)g(argumen)o(ts)f(are)h(optional,)h(con)o (text)e(w)o(ould)i(b)q(e)g(an)f(optional)g(parameter)g(with)189 102 y(default)i(v)m(alue)i Fb(ALL)p Fg(.\))146 208 y(The)h(adv)m(an)o(tage)f (of)g(an)h(explicit)i(con)o(text)d(parameter)g(is)h(that)g(it)g(prev)o(en)o (ts)f(the)h(confusion)h(that)75 264 y(o)q(ccurs)13 b(when)g(an)g(implicit)i (con)o(text)d(is)h(c)o(hanged)g(mid)g(comm)o(unication.)20 b(More)12 b(generally)l(,)i(it)f(prev)o(en)o(ts)75 321 y(the)e(confusion)h (arising)g(when)g(the)f(seman)o(tics)h(of)e(an)i(op)q(eration)f(is)h (quali\014ed)h(b)o(y)e(a)g(scop)q(e)h(that)e(dep)q(ends)75 377 y(on)18 b(the)g(con)o(trol)g(\015o)o(w)g(of)g(the)g(program,)g(rather)f (than)h(its)h(static)f(structure.)29 b(The)18 b(disadv)m(an)o(tage)g(is)75 434 y(the)e(need)g(for)f(an)g(additional)i(parameter,)d(in)j(languages)e (that)g(don't)g(ha)o(v)o(e)g(optional)h(parameters)e(or)75 490 y(o)o(v)o(erloading)h(of)g(pro)q(cedures.)146 547 y(Opinions?)75 690 y Fh(Matc)n(hing)23 b(of)g(send)g(and)h(receiv)n(e)75 793 y Fd(receiv)n(e)18 b(criteria)75 879 y Fg(Messages)c(are)h(matc)o(hed)g(b)o (y)h(sender)f(and)h(t)o(yp)q(e,)f(where)g(eac)o(h)g(can)h(b)q(e)g(a)e (\\don't)h(care".)146 935 y(Our)f(exp)q(erience)j(is)e(that)f(matc)o(hing)g (b)o(y)g(t)o(yp)q(e)h(only)l(,)g(or)f(disallo)o(wing)i(\\don't)e(care")g(in)h (the)f(sender)75 992 y(\014eld)20 b(signi\014can)o(tly)h(simpli\014es)g(the)e (implemen)o(tation)i({)d(but)h(some)g(users)g(complained)i(ab)q(out)e(suc)o (h)75 1048 y(restriction.)146 1105 y(Also,)c(do)g(w)o(e)g(w)o(an)o(t)f(to)g (b)q(e)i(explicit)i(ab)q(out)d(the)g(don't)g(care)g(v)m(alue,)h(or)e(use)i (named)f(constan)o(ts?)146 1161 y(Opinions?)75 1283 y Fd(Matc)n(hing)20 b(of)f(data)75 1369 y Fg(F)l(or)c(un)o(t)o(yp)q(ed)g(messages)g(the)g (options)h(are)131 1462 y(1.)22 b(Send)16 b(and)f(receiv)o(e)h(bu\013er)f(ha) o(v)o(e)g(same)g(size)131 1556 y(2.)22 b(Receiv)o(e)16 b(bu\013er)g(is)f(no)g (shorter)g(than)g(send)h(bu\013er)131 1650 y(3.)22 b(No)15 b(constrain)o(ts)f(\(messages)h(ma)o(y)f(b)q(e)i(truncated\).)131 1744 y(4.)22 b(User)15 b(c)o(ho)q(ose)g(one)g(of)g(the)g(ab)q(o)o(v)o(e)g (options)h(\(one)f(more)f(call)j(parameter,)d(or)g(a)h(global)h(\015ag\).)146 1838 y(Choices)f(should)g(b)q(e)g(the)f(same)g(for)f(all)j(t)o(yp)q(es)e(of)g (send)h(op)q(eration)f(\(the)g(curren)o(t)g(MPI)g(draft)g(seem)75 1894 y(to)h(allo)o(w)g(truncation)g(for)g(con)o(tiguous)g(data,)f(but)i (disallo)o(w)g(it)f(for)g(receiv)o(e)h(with)g(stride\))146 1951 y(The)d(EUI)h(prop)q(osal)f(uses)g(option)h(4.)19 b(I)13 b(w)o(ould)h(b)q(e)g(happier)g(with)f(2,)g(but)h(am)e(uncomfortable)i(with)75 2007 y(3)i(\(truncation)h(seems)g(to)f(fall)i(in)f(the)g(category)f(of)g (comm)o(unication)i(errors,)e(for)g(whic)o(h)h(I)h(assume)e(a)75 2063 y(global)g(exception)g(handling)h(mec)o(hanism\).)146 2120 y(Opinions?)146 2176 y(If)12 b(t)o(yp)q(ed)g(messages)g(are)g(supp)q (orted)g(then)h(one)f(should)h(sp)q(ell)h(out)e(requiremen)o(ts)g(on)g(t)o (yp)q(e)g(compat-)75 2233 y(ibilit)o(y)18 b(\(e.g.,)d(can)h(t)o(w)o(o)e(real) i(n)o(um)o(b)q(ers)g(b)q(e)h(receiv)o(ed)g(in)o(to)f(a)g(complex)g(n)o(um)o (b)q(er?\))23 b({)16 b(this)g(is)g(probably)75 2289 y(sp)q(eci\014c)h(to)e (eac)o(h)g(language)g(binding.)146 2346 y(Opinions?)75 2489 y Fh(Syn)n(tax)75 2590 y Fg(This)h(ma)o(y)e(b)q(e,)i(of)f(course,)f(a)h(sub)s (ject)g(of)g(endless)i(dispute.)k(A)15 b(few)g(ob)o(vious)g(guidelines:)143 2684 y Fa(\017)23 b Fg(Short,)14 b(mnemonic)i(names)964 2828 y(9)p eop %%Page: 10 10 bop 143 45 a Fa(\017)23 b Fg(Systematic)15 b(naming)h(con)o(v)o(en)o(tion)143 137 y Fa(\017)23 b Fg(Systematic)15 b(ordering)g(of)g(parameters)143 230 y Fa(\017)23 b Fg(Use,)15 b(in)h(languages)g(where)g(this)f(is)h(p)q (ossible,)h(of)e(optional)h(parameters)f(and)h(k)o(eyw)o(ord)e(param-)189 286 y(eters.)143 379 y Fa(\017)23 b Fg(Use,)15 b(in)h(languages)f(where)g (this)h(p)q(ossible,)h(of)d(o)o(v)o(erloading.)146 469 y(I)d(heard)g(in)h (our)f(last)g(meeting)g(a)g(preference)h(for)f(m)o(ultiple)h(function)g (names,)g(rather)e(than)h(m)o(ultiple)75 525 y(mo)q(des.)146 582 y(Th)o(us,)f(in)h(C++)g(or)f(F90,)f(one)i(could)g(ha)o(v)o(e)f(the)g(c)o (hoice)h(of)f(the)g(t)o(yp)q(e)g(of)g(datum)g(sen)o(t)g(\(scalar/con)o (tiguous)75 638 y(v)o(ector/noncon)o(tiguous\))15 b(b)q(e)i(dictated)g(b)o(y) f(the)h(t)o(yp)q(e)f(and)h(n)o(um)o(b)q(er)f(of)g(parameters,)f(rather)h (than)g(b)q(e)75 695 y(enco)q(ded)f(in)f(the)g(function)g(name.)19 b(Then)14 b(a)f(p)q(ossible)i(naming)f(c)o(hoice)h(is)f Fb(mode)23 b(|)h(operation)p Fg(,)12 b(where)75 751 y Fb(mode)18 b Fg(=)g Fb(init)g Fg(\(1\),)f Fb(nonblocking)g Fg(\(1+2\),)h Fb(blocking)f Fg(\(1+2+3+4,)h(lo)q(cal)h(termination\),)g(or)f Fb(sync)75 808 y Fg(\(1+2+3+4)d(global)h(termination\),)f(and)g Fb(operation)f Fg(=)h(send,)h(or)e(recv.)146 864 y(In)h(addition,)h(one)f(has)g Fb(DO\(handle\))p Fg(,)f Fb(START\(handle\))p Fg(,)e Fb(AWAIT\(handle\))i Fg(and)h Fb(FREE\(handle\))p Fg(.)146 921 y(Another)21 b(issue)i(is)f (function)g(vs)f(pro)q(cedure)i(call,)h(in)e(F)l(ortran.)38 b(Pro)q(cedure)22 b(calls)h(seem)e(more)75 977 y(natural)15 b(in)h(F)l(ortran,)e(where)h(function)h(in)o(v)o(o)q(cations)g(cannot)f(b)q (e)h(used)g(as)e(statemen)o(ts.)146 1033 y(Opinions?)22 b(Preferences)16 b(for)f(mnemonics?)21 b(Suggestions)16 b(for)e(F77)h(and)g(C?)75 1176 y Fh(Miscelania)75 1277 y Fg(Do)h(w)o(e)g(w)o(an)o(t)g(a)g Fb(TEST)p Fg(,)f(a)i(non)o(blo)q(c)o(king)h Fb(AWAIT)d Fg(op)q(eration?)25 b(\(Alw)o(a)o(ys)16 b(returns;)h(a)f(successful)i(return)75 1334 y(is)e(equiv)m(alen)o(t)h(to)d(a)h(successful)i(return)e(of)g(a)f(blo)q (c)o(king)j Fb(AWAIT)p Fg(\).)146 1390 y(Do)d(w)o(e)h(w)o(an)o(t)f(to)h Fb(AWAIT)f Fg(sev)o(eral)i(messages?)146 1447 y(Do)h(w)o(e)h(w)o(an)o(t)f(a)g Fb(PROBE)h Fg(function)h(\(tells)f(me)g(whic)o(h)h(messages)f(are)g(w)o (aiting)g(to)f(b)q(e)i(receiv)o(ed)g(b)o(y)75 1503 y(me\)?)146 1560 y(Where)12 b(do)h(w)o(e)f(return)h(information)g(on)g(parameters)f(of)g (receiv)o(ed)i(messages)e(suc)o(h)h(as)f(length,)i(and)75 1616 y(t)o(yp)q(e?)24 b(The)16 b(curren)o(t)g(MPI)h(prop)q(osal)f(has)g(length)h (b)q(e)g(the)f(v)m(alue)i(returned)e(b)o(y)h(the)f(receiv)o(e)h(function)75 1673 y(and)d(t)o(yp)q(e)g(returned)h(b)o(y)f(a)f(subsequen)o(t)i(call)g(to)e Fb(MP)p 964 1673 15 2 v 17 w(INFOT)p Fg(.)g(Another)h(natural)g(c)o(hoice)h (w)o(ould)f(seem)h(to)75 1729 y(b)q(e)f(that)f(that)f(length)i(and)g(t)o(yp)q (e)f(are)g(returned)h(b)o(y)f(the)h(len)g(and)g(t)o(yp)q(e)f(parameters)f(of) h(the)h(initial)h(call,)75 1786 y(but)g(this)h(is)g(error)e(prone:)143 1876 y Fa(\017)23 b Fg(The)d(v)m(alues)h(are)e(returned)i(async)o(hronously)f (for)f(non)o(blo)q(c)o(king)j(receiv)o(es.)35 b(The)20 b(user)g(has)g(to)189 1932 y(remem)o(b)q(er)11 b(not)h(to)f(touc)o(h)g(the)h(actual)g(len)g(and)g (t)o(yp)q(e)g(parameters,)f(in)h(addition)h(of)e(not)h(touc)o(hing)189 1989 y(the)j(receiv)o(e)h(bu\013er,)f(un)o(til)h(the)f(op)q(eration)h (completes.)143 2081 y Fa(\017)23 b Fg(A)f(natural)h(implemen)o(tation)h(w)o (ould)f(b)q(e)g(to)f(alw)o(a)o(ys)g(return)g(the)h(length)g(and)g(t)o(yp)q(e) g(of)f(the)189 2138 y(receiv)o(ed)f(messages.)36 b(Ho)o(w)o(ev)o(er,)20 b(most)g(users)g(will)i(assume)f(that)e(the)i(actual)g(len)g(and)g(t)o(yp)q (e)189 2194 y(parameters)14 b(are)h(not)g(up)q(dated)h(when)f(their)h(v)m (alue)h(is)e(not)g(\\don't)f(care".)143 2286 y Fa(\017)23 b Fg(The)13 b(user)h(will)h(not)e(b)q(e)h(able)g(to)f(use)h(constan)o(ts)e(as)h (actual)h(\\don't)f(care")g(parameter)f(v)m(alues)j(\(or)189 2343 y(an)o(ytime,)g(if)g(the)g(\\natural")g(implemen)o(tation)i(is)e(allo)o (w)o(ed\))143 2435 y Fa(\017)23 b Fg(This)17 b(c)o(hoice)h(b)q(ecomes)g(ev)o (en)f(less)h(app)q(etizing)h(when)e(a)g(handle)h(is)g(reused;)g(then)g(the)f (len)h(and)189 2492 y(t)o(yp)q(e)d(v)m(ariables)h(w)o(ould)g(b)q(e)g(reused,) f(to)q(o.)75 2582 y(It)g(seems)g(safer)f(to)g(return)h(these)g(v)m(alues)h (with)f(the)g(call)g(that)g(concludes)h(the)f(execution)h(of)e(sub)q(op)q (er-)75 2638 y(ation)h(3)g({)g(but)g(w)o(e)g(need)h(to)f(think)h(syn)o(tax.) 146 2695 y(Opinions?)952 2828 y(10)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-comm@CS.UTK.EDU Mon Nov 30 18:08:53 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22099; Mon, 30 Nov 92 18:08:53 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA03436; Mon, 30 Nov 92 17:54:53 -0500 Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03434; Mon, 30 Nov 92 17:54:51 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA26470; Mon, 30 Nov 92 17:54:50 -0500 Date: Mon, 30 Nov 92 17:54:50 -0500 Message-Id: <9211302254.AA26470@thud.cs.utk.edu> To: mpi-comm@cs.utk.edu Subject: draft of introduction Enclosed is a draft the introduction subcommittee has put together. We would like to get your comments and input. We feel that it is in a partial state and will be expanded. Hopefully this draft will help keep the discussion of goals, target, audience, etc. from becoming fragmented in each subcommittee. Jack \section{Introduction} \label{sec:intro} \subsection{Overview and Goals} Message passing is a paradigm used widely on certain classes of parallel machines; especially those with distributed memory. Although there are many variations, the basic concept of processes communicating through messages is well understood. Over the last ten years, substantial progress has been made in casting significant applications in this paradigm. Each vendor has implemented their own variant. More recently, several systems \cite{need references} have demonstrated that a message passing system can be efficiently and be portably implemented. It is thus an appropriate time to try to define both the syntax and semantics of a core of library routines that will be useful to a wide range of users and efficiently implementable on a wide range of computers. In designing MPI we have sought to make use of the most attractive features of a number of existing message passing systems, rather than selecting one of them and adopting it as the standard. Thus, MPI has been strongly influenced by work at the IBM T. J. Watson Research Center by Bala, Kipnis, Snir and colleagues \cite{}, Intel's NX/2 \cite{}, Express \cite{}, nCUBE's Vertex \cite{}, and PARMACS \cite{}. Other important contributions have come from Zipcode \cite{}, Chimp \cite{}, PVM \cite{}, and PICL \cite{}. One of the objectives of this paper is to promote a discussion within the concurrent computing research community of the issues that must be addressed in establishing a practical, portable, and flexible standard for message passing. This cooperative process began with a workshop on standards for message passing held in April 1992 \cite{}. The main advantages of establishing a message passing standard are portability and ease-of-use. In a distributed memory communication environment in which the higher level routines and/or abstractions are build upon lower level message passing routines the benefits of standardization are particularly apparent. Furthermore, the definition of a message passing standard, such as that proposed here, provides vendors with a clearly defined base set of routines that they can implement efficiently, or in some cases provide hardware support for, thereby enhancing scalability. The goal of the Message Passing Interface simply stated is to develop a \it{de facto} standard for writing message-passing programs. As such the interface should establishing a practical, portable, efficient, and flexible standard for message passing. \subsection{Who Should Use This Standard?} This standard is intended for use by all those who want to write portable message-passing programs in Fortran 77 and/or C. This includes individual application programmers, developers software designed to run on parallel machines, and creators of higher-level programming languages, environments, and tools. In order to be attractive to this wide audience, the standard must provide a simple, easy-to-use interface for the basic user while not semantically precluding the high-performance message-passing operations available on advanced machines. \subsection{What Platforms Are Targets For Implementation?} The attractiveness of the message-passing paradigm at least partially stems from its wide portability. Programs expressed this way can run efficiently on distributed-memory multiprocessors, networks of workstations, and combinations of all of these. In addition, shared-memory implementations are possible. The paradigm will not be made obsolete by architectures combining the shared- and distributed-memory views, or by increases in network speeds. It thus should be both possible and useful to implement this standard on a great variety of machines, including those "machines" consisting of collections of other machines, parallel or not, connected by a communication network. \subsection{What Is Included In The Standard?} The standard includes (this is temporarily as inclusive as possible): \begin{itemize} \item Point-to-point communication in a variety of modes, including modes that allow fast communication and heterogeneous communication \item Collective operations \item Process groups \item Communication contexts \item A simple way to create processes for the SPMD model \item Bindings for both Fortran and C \item A model implementation \item A formal specification. \end{itemize} \subsection{What Is Not Included In The Standard?} The standard does not specify: \begin{itemize} \item Explicit shared-memory operations \item Operations that require more operating system support than is currently standard; for example, interrupt-driven receives, remote execution, or active messages \item Program construction tools \item Debugging facilities \item Tracing facilities \end{itemize} Features that are not included can always be offered as extensions by specific implementations. From owner-mpi-comm@CS.UTK.EDU Tue Dec 1 16:39:05 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17944; Tue, 1 Dec 92 16:39:05 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA27367; Tue, 1 Dec 92 16:14:27 -0500 Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27363; Tue, 1 Dec 92 16:14:24 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA00550; Tue, 1 Dec 92 16:14:28 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 161350.23711; Tue, 1 Dec 1992 16:13:50 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-92031301 for ) id AA22501; Tue, 1 Dec 1992 15:07:53 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA16316; Tue, 1 Dec 92 15:07:51 -0600 Date: Tue, 1 Dec 92 15:07:51 -0600 Message-Id: <9212012107.AA16316@brisk.kai.com> To: mpi-comm@cs.utk.edu Subject: A proposal. From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 I've seen several proposals and made a few suggestions myself. I have now had a chance to look at Gropp and Lusk's Manual for the test implementation and I like it as a starting point. I propose to all subcommittees that we use this document as the focus for discussion along with the introduction Jack sent around yesterday. This shouldn't be taken as an endorsement by me of the Gropp and Lusk manual - I have several remarks I would like to make at a later stage. However, it is clear that we need a well considered and simple starting point and I see that in this document plus the introduction. Do I have any seconders? In the HPFF committe structure description we received a few days ago it was noted that much success came from someone taking charge of the central document. I implore someone to do this (Jack?). Steven P.S. I am not ignoring the other comments made in the committees - I think they too can - after debate - be folded into the suggested starting point. From owner-mpi-comm@CS.UTK.EDU Tue Dec 1 19:08:43 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21233; Tue, 1 Dec 92 19:08:43 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA01862; Tue, 1 Dec 92 18:43:11 -0500 Received: from cunyvm.cuny.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01858; Tue, 1 Dec 92 18:43:08 -0500 Message-Id: <9212012343.AA01858@CS.UTK.EDU> Received: from YKTVMV by CUNYVM.CUNY.EDU (IBM VM SMTP V2R2) with BSMTP id 7311; Tue, 01 Dec 92 18:42:33 EST Date: Tue, 1 Dec 92 18:43:02 EST From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-comm@cs.utk.edu Subject: HPF process Reply-To: SNIR@watson.ibm.com The HPF process did not start with a central document. Various peoples wrote proposals for various parts of the language design and, after several iterations, when the number of proposals starting being reasonable and these proposals started being coherent, they were collated into one document that evolved (mostly by successive pruning) into a draft. I suppose we can follow the same approach -- with the initial MPI Draft being our first frame of reference. From owner-mpi-comm@CS.UTK.EDU Fri Dec 4 16:10:12 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06597; Fri, 4 Dec 92 16:10:12 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA27740; Fri, 4 Dec 92 15:56:38 -0500 Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27738; Fri, 4 Dec 92 15:56:36 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA07974; Fri, 4 Dec 92 15:56:35 -0500 Date: Fri, 4 Dec 92 15:56:35 -0500 Message-Id: <9212042056.AA07974@thud.cs.utk.edu> To: mpi-comm@cs.utk.edu Subject: addresses for mpi I'm trying to put together a complete mailing list for MPI. Can you fill in the following information and send it to me. Thanks, Jack Name: Address: Email: Fax: Office phone number: Home phone number: (I'll use this only in an emergency and won't circulate.) From owner-mpi-comm@CS.UTK.EDU Mon Dec 7 21:39:33 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10542; Mon, 7 Dec 92 21:39:33 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA29754; Mon, 7 Dec 92 21:28:26 -0500 Received: from vnet.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29750; Mon, 7 Dec 92 21:28:21 -0500 Message-Id: <9212080228.AA29750@CS.UTK.EDU> Received: from AUSVM6 by vnet.ibm.com (IBM VM SMTP V2R2) with BSMTP id 3637; Mon, 07 Dec 92 21:26:25 EST Date: Mon, 7 Dec 92 14:02:48 CST From: panda@vnet.ibm.com To: mpi-comm@cs.utk.edu To: mpi-comm@cs.utk.edu Subject: addresses for mpi I'm trying to put together a complete mailing list for MPI. Can you fill in the following information and send it to me. Thanks, Jack NAME: RAJ PANDA ADDRESS: E39/4305 IBM, 11400 BURNET RD., AUSTIN, TX 78758 EMAIL: PANDA@VNET.IBM.COM FAX: 1-512-823-6144 OFFICE PHONE NUMBER: 1-512-838-6632 HOME PHONE NUMBER: 1-512-835-1959 e.) From owner-mpi-comm@CS.UTK.EDU Mon Dec 7 22:10:17 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10659; Mon, 7 Dec 92 22:10:17 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA29855; Mon, 7 Dec 92 21:41:36 -0500 Received: from vnet.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29851; Mon, 7 Dec 92 21:41:26 -0500 Message-Id: <9212080241.AA29851@CS.UTK.EDU> Received: from AUSVM6 by vnet.ibm.com (IBM VM SMTP V2R2) with BSMTP id 4638; Mon, 07 Dec 92 21:39:36 EST Date: Mon, 7 Dec 92 14:51:31 CST From: LBWARD@AUSVM6.VNET.IBM.COM To: mpi-comm@cs.utk.edu Subject: address for mpi mailing list Please add my name to the MPI forum. Linton Ward Name: Linton Ward Address: 11400 Burnet Rd, Austin TX 78758 Email: lbward@ausvm6.vmnet.ibm.com Fax: (512) 823-6144 Office phone number: (512) 838-5116 Home phone number: (512) 244-1949 From owner-mpi-comm@CS.UTK.EDU Thu Dec 10 13:10:06 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08984; Thu, 10 Dec 92 13:10:06 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA18058; Thu, 10 Dec 92 12:47:20 -0500 Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18051; Thu, 10 Dec 92 12:47:17 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA27178; Thu, 10 Dec 92 12:47:15 -0500 Date: Thu, 10 Dec 92 12:47:15 -0500 Message-Id: <9212101747.AA27178@thud.cs.utk.edu> To: mpi-comm@cs.utk.edu Subject: Information on the MPI meeting Here are some details on the MPI meeting which is set for January 6th-8th 1993 in Dallas. The meeting site will be the: Bristol Suites 7800 Alpha Road Dallas, TX 214-233-7600 The room rate is $89.00. When making a reservation tell them you are with the MPI meeting. TBS Shuttle Service will be providing complimentary shuttle service to and from the airports. If you fly into DFW, use their courtesy telephone and dial 03. If you fly into Love Field, you'll have to use a pay phone. They can be reached at 817-267-5150. Upon boarding the shuttle, refer to the MPI meeting. The registration fee for the meeting will be $75. Please make checks and POs payable to University of Tennessee. We will collect this at the meeting. The registration fee will go for coffee breaks, meeting rooms, AV and printer rentals. We should plan to start at 1:00 pm January 6th and finishing about noon on January 8th. The format of the meeting is: Wednesday, January 6 1:00 pm to 8:00 pm 2 Breakouts for 10 people 1 Breakout for 20 people 5:00 pm to 6:00 pm--On our own for dinner. Thursday, January 7 8:00 am to 10:00 pm 1 U-shape room for 40 people 6:00 pm to 8:00 pm-- The group dinner somewhere in the area. The hotel will provide round trip van transportation. Friday, January 8 8:00 am to 12:00 pm 1 U-shape room for 40 people This is an open meeting and all are welcome. For future planning here is a tentative list of dates, roughly 6 weeks apart, for the series of meetings: January 6-8 February 24-26 April 7-9 May 19-21 June 30-July2 If you have any questions, please feel free to contact me. Jack From owner-mpi-comm@CS.UTK.EDU Wed Dec 23 16:46:29 1992 Received: from convex.convex.com by mozart.convex.com (5.64/1.28) id AA20114; Wed, 23 Dec 92 16:46:25 -0600 Received: from CS.UTK.EDU by convex.convex.com (5.64/1.35) id AA05394; Wed, 23 Dec 92 16:46:20 -0600 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14697; Wed, 23 Dec 92 17:31:12 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 23 Dec 1992 22:31:10 GMT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14687; Wed, 23 Dec 92 17:31:08 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA19206; Wed, 23 Dec 1992 17:29:47 -0500 Date: Wed, 23 Dec 1992 17:29:47 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9212232229.AA19206@rios2.epm.ornl.gov> To: abmacca@cs.sandia.gov, benson@rdvax.enet.dec.com, mpi-comm@cs.utk.edu, peter@sun.math.usfca.edu, srwheat@cs.sandia.gov Subject: MPI Information Status: RO Dear MPI committee member or observer, I need to know how many people to expect at the Dallas so please let me know if you intend to show up by sending email to walker@msr.epm.ornl.gov Please find below some information about the use of the MPI mailing lists, and who's involved in the various subcommittees. Please check that your entry is correct. If there are any errors please sent email to walker@msr.epm.ornl.gov and to dongarra@cs.utk.edu. Also there's some information about the January 6-8, 1993, meeting in Dallas. 1. Use of Mailing Lists ======================= The following mailing lists have been set up. mpi-comm@cs.utk.edu Whole committee (subcommitttes + observers) mpi-intro@cs.utk.edu Introduction subcommittee mpi-pt2pt@cs.utk.edu Point-to-point communication subcommittee mpi-collcomm@cs.utk.edu Collective communication subcommittee mpi-ptop@cs.utk.edu Process topology subcommittee mpi-lang@cs.utk.edu Language binding subcommittee mpi-formal@cs.utk.edu Formal language description subcommittee mpi-envir@cs.utk.edu Environment inquiry subcommittee All mail is being collected and can be retrieved by sending email to netlib@ornl.gov and in the mail message typing: send mpi-comm from mpi send mpi-intro from mpi ...etc. Also try: send index from mpi 2. Membership of Subcommittees (as of Dec. 22, 1993) ==================================================== 2.1. Introduction Subcommittee: Jack Dongarra (Chair) dongarra@cs.utk.edu Univ. of TN & ORNL Jim Cownie jim@meiko.co.uk Meiko Daniel Frye danielf@kgnvma.vnet.ibm.com IBM Kingston Ian Glendinning igl@ecs.soton.ac.uk Southampton Univ. Tony Hey ajgh@ecs.soton.ac.uk Southampton Univ. Robert L. Knighten knighten@ssd.intel.com Intel SSD Rusty Lusk lusk@mcs.anl.gob Argonne National Lab. Peter A. Rigsbee par@cray.com Cray Research, Inc. Steve Zenith zenith@kai.com Kuck & Associates, Inc. 2.2. Point-To-Point Communication Subcommittee Marc Snir (Chair) snir@watson.ibm.com IBM T. J. Watson Scott Berryman berryman@cs.yale.edu Yale Jaeyoung Choi choi@msr.epm.ornl.gov ORNL Jim Cownie jim@meiko.co.uk Meiko Jack Dongarra dongarra@cs.utk.edu Univ. of TN & ORNL Anne Elster elster@cs.cornell.edu Cornell Univ. Al Geist geist@msr.epm.ornl.gov ORNL Adam Greenberg moose@think.com Thinking Machines Corp. Bill Gropp gropp@mcs.anl.gov Argonne National Lab. Leslie Hart hart@fsl.noaa.gov NOAA/FSL Rolf Hempel hempel@gmd.de GMD Tom Henderson hender@fsl.noaa.gov NOAA/FSL Steven Huss-Lederman lederman@super.org Supercomput. Res. Ctr. John Kapenga john@cs.wmich.edu Western Michigan Univ. Robert L. Knighten knighten@ssd.intel.com Intel SSD Arthur B. Maccabe abmacca@cs.sandia.gov Sandia National Labs Oliver McBryan mcbryan@piper.cs.colorado.edu Univ. of Colorado Charles Mosher ccm@arco.com ARCO Paul R. Pierce prp@ssd.intel.com Intel SSD Peter Rigsbee par@cray.com Cray Research, Inc. Anthony Skjellum tony@cs.msstate.edu Mississippi State Univ. Vaidy Sunderam vss@mathcs.emory.edu Emory University Robert van de Geijn rvdg@cs.utexas.edu Univ. of Texas, Austin David Walker walker@msr.epm.ornl.gov ORNL Stephen R. Wheat srwheat@cs.sandia.gov Sandia National Labs Steve Zenith zenith@kai.com Kuck & Associates, Inc. 2.3. Collective Communication Subcommittee Al Geist (Chair) geist@msr.epm.ornl.gov ORNL Jaeyoung Choi choi@msr.epm.ornl.gov ORNL Jim Cownie jim@meiko.co.uk Meiko Mark Debbage md@ecs.soton.ac.uk Southampton Univ. Anne Elster elster@cs.cornell.edu Cornell Univ. Jon Flower jwf@parasoft.com Parasoft Corp. Adam Greenberg moose@think.com Thinking Machines Corp. Leslie Hart hart@fsl.noaa.gov NOAA/FSL C. T. Howard Ho ho@almaden.ibm.com IBM Almaden John Kapenga john@cs.wmich.edu Western Michigan Univ. Robert L. Knighten knighten@ssd.intel.com Intel SSD Arthur B. Maccabe abmacca@cs.sandia.gov Sandia National Labs Charles Mosher ccm@arco.com ARCO Steve Otto otto@cse.ogi.edu Oregon Graduate Inst. Paul R. Pierce prp@ssd.intel.com Intel SSD Peter Rigsbee par@cray.com Cray Research, Inc. Anthony Skjellum tony@cs.msstate.edu Mississippi State Univ. Marc Snir snir@watson.ibm.com IBM T. J. Watson Vaidy Sunderam vss@mathcs.emory.edu Emory University Robert van de Geijn rvdg@cs.utexas.edu Univ. of Texas, Austin David Walker walker@msr.epm.ornl.gov ORNL Stephen R. Wheat srwheat@cs.sandia.gov Sandia National Labs Steve Zenith zenith@kai.com Kuck & Associates, Inc. 2.4. Process Topology Subcommittee Rolf Hempel (Chair) hempel@gmd.de GMD Jim Cownie jim@meiko.co.uk Meiko Jon Flower jwf@parasoft.com Parasoft Corp. Tom Henderson hender@fsl.noaa.gov NOAA/FSL Steven Huss-Lederman lederman@super.org Supercomput. Res. Ctr. Robert L. Knighten knighten@ssd.intel.com Intel SSD Oliver McBryan mcbryan@piper.cs.colorado.edu Univ. Of Colorado Steve Otto otto@cse.ogi.edu Oregon Graduate Inst. Lew Tucker tucker@think.com Thinking Machines Corp. 2.5. Language Binding Subcommittee Scott Berryman (Chair) berryman@cs.yale.edu Yale Bruce Leisure bleisure@kai.com Kuck & Associates, Inc. Robert L. Knighten knighten@ssd.intel.com Intel SSD Cherri M. Pancake pancake@cs.orst.edu Oregon State Univ. Peter Rigsbee par@cray.com Cray Research, Inc. 2.6. Formal Language Description Subcommittee Steve Zenith (Chair) zenith@kai.com Kuck & Associates, Inc. Ian Glendinning igl@ecs.soton.ac.uk Southampton Univ. Tony Hey ajgh@cs.ston.ac.uk Southampton Univ. Robert L. Knighten knighten@ssd.intel.com Intel SSD Rusty Lusk lusk@mcs.anl.gov Argonne National Lab. 2.7. Environment Inquiry Subcommittee Bill Gropp (Chair) gropp@mcs.anl.gov Argonne National Lab. Daniel Frye danielf@kgnvma.vnet.ibm.com IBM Kingston Robert L. Knighten knighten@ssd.intel.com Intel SSD Steve Zenith zenith@kai.com Kuck & Associates, Inc. 2.8. Observers Ed Benson benson@rdvax.enet.dec.com DEC Clemens H. Cap cap@ifi.unizh.ch University of Zurich Michel J. Dayde dayde@enseeiht.fr ENSEEIHT-IRIT Jim Feeney feeneyj@gdlvm6.vnet.ibm.com IBM Kyle Gallivan gallivan@csrd.uiuc.edu CSRD, Univ. of Illinois Michael Heath heath@ncsa.uiuc.edu NCSA, Univ. of Illinois Mark Bowen Hill mbh@ecs.soton.ac.uk Southampton Univ. Shlomo Kipnis kipnis@watson.ibm.com IBM T. J. Watson ??? langley@dirac.scri.fsu.edu Florida State Univ. Bob Leary leary@sdsc.edu San Diego Super. Ctr. Lorie Liebrock lorie@cs.rice.edu Rice University Miron Livny miron@cs.wisc.edu Univ. of Wisconsin Ken Kennedy ken@rice.edu Rice University Neil MacDonald nbm@epcc.ed.ac.uk Univ. of Edinburgh Dan Cristian Marinescu dcm@cs.purdue.edu Purdue University Harish Nag harish@ssd.intel.com Intel SSD Dan Nessett nessett@llnl.gov LLNL Angela Ovearly fsang@kira.lerc.nasa.gov NASA Lewis Res. Ctr. Peter S. Pacheco peter@sun.math.usfca.edu Univ. of CA, San Franc. Raj Panda panda@vnet.ibm.com IBM Arnulfo Perez arperez@mtecv2.mty.itesm.mx Center for AI, Mexico Matthew Peters mpeters@vnet.ibm.com IBM UK Sci. Ctr. Jean-Laurent Philippe philippe@archipel.fr Archipel Robert S. Schreiber schreibr@riacs.edu RIACS, NASA Ames Mike Surridge ms@pac.soton.ac.uk Southampton Univ. Tricot tricot@archipel.fr Archipel Robert Voigt rvoigt@note.nsf.gov NSF Linton Ward lbward@ausvm6.vmnet.ibm.com IBM Dennis Weeks weeks@convex.com Convex Francis Wray falk@parsytec.de Parsytec 3. Institutions Involved in MPI =============================== 3.1. Universities Univ. of Edinburgh Emory University N.L. Mexico Centro de Intelligencia Artifical, Mexico Mississippi State University Oregon Graduate Institute Oregon State University Purdue University Rice University Western Michigan University Yale University University of Illinois, CSRD University of Illinois, NCSA University of Southampton University of Tennessee University of Texas at Austin University of Wisconsin University of Zurich University of Colorado 3.2. Laboratories ARCO Exploration and Production Technology Argonne National Laboratory Cerfacs ENSEEIHT-IRIT, France GMD Lawrence Livermore National Laboratory NASA RIACS NOAA Oak Ridge National Laboratory San Diego Supercomputer Center Sandia National Laboratories Supercomputer Research Center 3.3. Companies ARCHIPEL S.A. Cray Research, Inc. Digital Equipment Corporation IBM Austin IBM Almaden Research Center IBM Kingston IBM T.J. Watson Research Center IBM UK Scientific Centre Intel Supercomputer Systems Division Kuck and Associates, Inc. ParaSoft Corporation Parsytec Computer GmbH Thinking Machines Corporation 4. MPI Committee Meeting, Jan 6-8, 1993 ======================================= The next meeting of the MPI subcommittees will take place at Bristol Suites Hotel 7800 Alpha Road Dallas, Texas The meeting will start at 1pm on Wednesday January 6, 1993, and finish at noon on Friday, January 8, 1993. Rooms are $89 per night and reservations may be made by calling (214) 233-7600 (mention MPI meeting). The meeting registration fee will be $75. Please make checks and POs payable to University of Tennessee. Jack Dongarra will collect this at the meeting. The registration fee will go for coffee breaks, meeting rooms, AV and printer rentals. TBS Shuttle Service will be providing complimentary shuttle service to and from the airports. If you fly into DFW, use their courtesy telephone and dial 03. If you fly into Love Field, you'll have to use a pay phone. They can be reached at 817-267-5150. Upon boarding the shuttle refer to the MPI meeting. We have made arrangements with Delta Airlines to get a deal on airfares. When you book your flights please call Delta or have your travel agent call 1-800-241-6760 for reservations and refer to File Number S2145. The file goes by the name Message Passing Interface. A certain restrictions apply and seats are limited. An agenda will be sent out soon to all subcommittee members. We intend to rent a laser printer for use by attendees with protable computers. From owner-mpi-comm@CS.UTK.EDU Mon Jan 4 11:02:02 1993 Received: from convex.convex.com by mozart.convex.com (5.64/1.28) id AA06142; Mon, 4 Jan 93 11:02:00 -0600 Received: from CS.UTK.EDU by convex.convex.com (5.64/1.35) id AA08877; Mon, 4 Jan 93 11:01:56 -0600 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15255; Mon, 4 Jan 93 11:54:20 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 04 Jan 1993 16:54:19 GMT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from timbuk.cray.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15247; Mon, 4 Jan 93 11:54:16 -0500 Received: from teak18.cray.com by cray.com (4.1/CRI-MX 2.4) id AA21055; Mon, 4 Jan 93 10:54:14 CST Received: by teak18.cray.com id AA05581; 4.1/CRI-5.6; Mon, 4 Jan 93 10:54:13 CST From: par@teak.cray.com (Peter Rigsbee) Message-Id: <9301041654.AA05581@teak18.cray.com> Subject: Profilers etc. (fwd) To: mpi-comm@cs.utk.edu Date: Mon, 4 Jan 93 10:54:10 CST X-Mailer: ELM [version 2.3 PL11b-CRI] Status: R James Cownie writes: > > I have an implementation issue which I would like to raise in the MPI > forum, since it is unclear where it fits into the sub-committee > structure I have mailed to both mpi-pt2pt and mpi-lang. Apologies to > those of you who receive this message twice. > > Issue > ===== > The major objective of MPI1 is to achieve portability of applications. > This has major benefits for us all (not least in legitimising and > therefore growing the MPP marketplace). > > One of the benefits which it would also be nice to achieve would be > the wide availability of different tools which support programming in > the MPI1 model. The most immediately obvious such tools (to me at > least !) are > > 1) HPF to Fortran + MPI1 translators > 2) Performance monitoring/tuning tools > 3) Debuggers > > Support for the first is easy (since it just requires what we're > already doing). > > Portable support for the second is not so trivial, since the > collection of useful performance information is much more intrusive. > > Portable support for the third is harder still, and I won't discuss it > further. First, I think this is an important issue. I'd argue that the need isn't necessarily to support portable tools, but it would be nice to define a mechanism by which performance and debugging tools could obtain at least the most basic information about the communications. Second, I don't think it really fits into either group that Jim sent the message to. The same issue, for example, also applies for global communications. This is why I copied the entire committee, and I think there should be some discussion in Dallas was to whether this is a topic that should be covered by the standard. Third, I fear this will be a very difficult subject to address. Clearly Jim had some specific use in mind when he made his suggestions, but didn't go into much detail (and properly so -- this isn't intended as a criticism). But there are some very basic differences between the ways that different tools are designed that need to be considered. What information should be recorded? Is data needed for each transmission (a "trace") or should it be accumulated by the library? Should the library pass information to the tool or should the tool query the library? I don't think there are clear cut answers here. But I think this is an important issue that should receive some airtime in Dallas. - Peter Rigsbee par@cray.com I've included the rest of Jim's proposal for those who didn't see the original mailing... > > Options > ======= > We have various possible options which we can take. > > 1) Ignore the problem > Provide no support for portable performance monitoring tools, and > leave each tool provider with a large porting problem. > > I don't like this solution, it loses some of the benefit of the > standard, which should be attracting people to build tools. > > 2) Document specific implementation hooks as part of MPI1. > In effect these would be callbacks from the library to profiling > code which could then do whatever it liked. > > 3) As 2, but without REQUIRING that a conforming implementation > provide the functions. They're there as a recommendation, rather > than being mandatory. > > > I think we should be concerned about this, and I'd like us at least to > make some recommendation. Personally I'd probably implement two > separate interface to the library, one of which provided the hooks, > and the other of which didn't so that you don't pay the cost of > checking the profiling hooks unless you asked to. > > Of course even if we do nothing it's not too hard to escape (using > horrible macros) in C, but Fortran doesn't always have macros, so a > properly specified internal solution is definitely preferable. > > Thoughts ??? > Flame me at Dallas. I'm travelling tomorrow. When are we having a > meeting in Europe ??? > > -- Jim > James Cownie > Meiko Limited > 650 Aztec West > Bristol BS12 4SD > England > > Phone : +44 454 616171 > FAX : +44 454 618188 > E-Mail: jim@meiko.co.uk or jim@meiko.com > > From owner-mpi-comm@CS.UTK.EDU Tue Jan 5 17:33:28 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07561; Tue, 5 Jan 93 17:33:28 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27130; Tue, 5 Jan 93 17:32:46 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 05 Jan 1993 22:32:45 GMT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27122; Tue, 5 Jan 93 17:32:42 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA23201; Tue, 5 Jan 93 22:32:36 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA01387; Tue, 5 Jan 93 15:31:37 MST Date: Tue, 5 Jan 93 15:31:37 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9301052231.AA01387@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: A few general comments Here are a few comments/questions regarding the MPI1 draft proposal from folks here (Forecast Systems Lab). Apologies if you have seen them before! 1) Currently, communication between groups operating in different communication contexts appears to be very difficult. One way to alleviate this problem might be to provide a way to "name" contexts. Communicating groups could then refer to a context by name to ensure that the same message types were used. Message registration would also accomplish this and might be easier to implement (see our "Proposal 1" below). 2) We believe that MPI1 should include a specification of low level protocols that vendors should implement so that heterogeneous computing can be made available. Each media type (standard ethernet, FDDI, HIPPI, etc.) should have a specification. This would greatly aid in this standard being useful in a workstation environment. The protocol document would be a companion document to the syntactical standard for MPI1. 3) On p. 9 "No reference may be made to any process or group outside the current group context." How do different groups that are simultaneously working on different stages of a problem communicate? MPI_PARTG()?? 4) The use of stacks to manage contexts and groups may be a bad idea if MPI ever intends to support MIMD programs (how do groups synchronize stacks of message contexts?). Message registration would do the job without stacks. Registration of blocks of messages at the beginning of a program (or large module) would not affect performance very much. For groups, we like Al Geist's approach of identifying a process by one or more (group ID, process ID) pairs. We feel that communication between groups is very useful, even for SPMD programs. 5) It is clear that some way of responding to asynchronous events must be provided. We believe that "interrupt" type routines (such as Intel's hrecv() and hsend()) may not be the best approach. We have a brief alternative proposal for "Remote Services" below ("Proposal 2"). 6) The consensus seems to be that "parallel I/O" should not be a part of MPI1. Should it be a part of MPI2 or MPI3? If our intent is to create a standard that supports the development of portable code then we believe that I/O must be considered at some point. We should make sure that the specification of MPI1 includes the hooks necessary to include I/O at a later date. If our intent is to create only a low level library, then we can leave all this up to us users. We have some ideas about I/O and Data Decomposition below ("Proposal 3"). Successful implementation of such a proposal would require process topology information (yes, here's a plug for the Process Topology subcommittee). The proposals follow. Regards, Leslie Hart (hart@fsl.noaa.gov) Tom Henderson (hender@fsl.noaa.gov) Bernardo Rodriguez (bernardo@fsl.noaa.gov) PROPOSALS Proposal 1. Message Type Registration. Message types are managed using a global registry. Routines are provided that allow the user to "reserve" blocks of message types for use by the application. Message type registration is intended to prevent message type "collisions" when user code and third-party library routines are used together (historically, a serious problem with typed message passing systems). Initial concepts for message type registration are due to Tony Skjellum of Lawrence Livermore National Laboratory. Two proposed routines to register and monitor message types are described below. Arguments have been omitted for brevity. Message type registration routine: MPI_ReserveMsgType() Reserve one or more message types for use by the calling process. This routine is used to prevent message type "collisions" between user applications and libraries. Inputs include an ID string and the number of message types to be reserved. The routine first checks the locally maintained message type registry and returns an error condition if the ID string matches an existing entry. The routine then goes to the global message type registry to get the message types. If the ID string matches an existing entry in the global message type registry then the message types are copied from that entry to the calling process. Otherwise, new message types are registered with the ID string, entered into the global message type registry, and copied to the calling process. The new ID string and message types are entered into the locally maintained message type registry. Message type registration inquiry routine: MPI_InquireMsgTypeByName() Ask the message type registry to obtain the message types reserved under an ID string. An error is returned if no message types have been reserved under the ID string. Proposal 2. Remote Services (brief concept only). A remote service is similar in concept to a RPC (remote procedure call). Remote services are most often used to respond to specified asynchronous events. The actions taken in response to asynchronous events are performed by agent processes known as protocol handlers. A remote service is requested of one process by the same or another process. The remote service request is accompanied by a small amount of data to guide the action of the service. Routines are provided to allow user-created routine to be attached to specified requests. An attached user-created routine is called a protocol handler. A protocol handler does not have access to the address space of the its associated process. The protocol handler has the status of a full process (ie. it may perform I/O and message-passing communication). This is an advantage over some current interrupt-driven approaches. Remote service requests are handled one at a time on a first-come first-served basis (ie. a new remote service request will not be handled until all pending remote service requests are handled). This may be a drawback to this approach. Proposal 3. Some ideas about Cartesian Data Decomposition and Parallel I/O. Due to length, we have just included a few ideas. If there is any interest, we can provide (lots) more detail. 1) We propose Cartesian data decomposition routines to support the scalable distribution of multidimensional arrays among the processes in a group. These routines isolate the user from the details of implementing scalable data parallelism in a message passing environment. The routines also enhance performance by distributing data in a way that is most advantageous for a particular architecture. 2) A Cartesian data decomposition defines the distribution of a multidimensional data array onto a "logical" array of processes with the same dimension. The processes are members of a specified group. A Cartesian data decomposition is defined by information provided by the user and by run-time parameters determined by the system. The user specifies the group, array dimension, and number of elements in each dimension of the data array. The user may also optionally specify re-ordering of processes in the group and boundary behavior. The Cartesian data decomposition routines then combine this information with the number of processes in the group (provided by the system at run time) to determine which portion of the data array will reside on each process in the group. All information about the decomposition is then stored in a structure (or FORTRAN77 array). The "logical" array of processors is created using the Processor Topology routines (this idea has already been discussed in some detail in that subcommittee). 3) Cartesian data decomposition and I/O should be intimately related. Once a decomposition structure has been created, I/O should be possible using routines with syntax similar to this: cget(file, array, decomp) cput(file, array, decomp) where decomp is the decomposition structure. 4) Data decomposition tools have been available for several years, most notably from Parasoft's Express software package. These tools are extremely useful for almost all data parallel applications. Use of these tools in conjunction with I/O greatly simplifies program development on message passing systems. 5) Few areas in parallel processing have been dealt with in less detail by standardization efforts than parallel I/O. We would like to see behavior defined for such operations as seek/read and seek/write when several processes are accessing the same file. The concept of "loose synchronization" (Express and elsewhere) has proven to be very useful and should be included in a standard. Encapsulation of I/O in standardized routines would also permit performance optimization by the people who (should) know best how to do it (ie. the hardware vendors). From owner-mpi-comm@CS.UTK.EDU Wed Jan 6 11:42:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA24005; Wed, 6 Jan 93 11:42:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12383; Wed, 6 Jan 93 11:41:35 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 06 Jan 1993 16:41:34 GMT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12375; Wed, 6 Jan 93 11:41:32 -0500 Received: from carbon.pnl.gov (130.20.65.121) by pnlg.pnl.gov; Wed, 6 Jan 93 08:39 PST Received: from fermi.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA25662; Wed, 6 Jan 93 08:38:48 PST Received: by fermi.pnl.gov (4.1/SMI-4.1) id AA18870; Wed, 6 Jan 93 08:38:46 PST Date: Wed, 06 Jan 93 08:38:45 -0800 From: Robert J Harrison Subject: Re: Comments on Bill Gropp's comments To: mpi-comm@cs.utk.edu Message-Id: <9301061638.AA18870@fermi.pnl.gov> X-Envelope-To: mpi-comm@cs.utk.edu At the end of message <9301051955.AA01014@SSD.intel.com> Paul Pierce wrote > > I strongly recommend that we avoid separated info calls. Info can be returned > through output parameters, although that does annoyingly complicate receive > calls. Alternatively, info can be associated with handles, but that seriously > complicates handle management. (Handle management is a good topic for a > separate discussion.) I also strongly favour not having separate info calls. The necessary information is brief and does not excessively clutter argument lists, and can always be hidden in a wrapper if it is routinely not required. Also, by removing sources of ambiguity we give more freedom for implementation specific optimizations. > > If we decide we must have separated info calls, we should use dynamic context > semantics. > Yes to this too. Robert Harrison. From owner-mpi-comm@CS.UTK.EDU Wed Jan 6 17:17:30 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10712; Wed, 6 Jan 93 17:17:30 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29683; Wed, 6 Jan 93 17:16:48 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 06 Jan 1993 22:16:46 GMT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from p.lanl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29672; Wed, 6 Jan 93 17:16:45 -0500 Received: from beta.lanl.gov by p.lanl.gov (5.65/1.14) id AA11282; Wed, 6 Jan 93 15:16:41 -0700 Received: by beta.lanl.gov (5.57/Ultrix2.4-C) id AA29872; Wed, 6 Jan 93 15:15:46 -0700 Date: Wed, 6 Jan 93 15:15:46 -0700 From: sp@beta.lanl.gov (Stephen W Poole) Message-Id: <9301062215.AA29872@beta.lanl.gov> To: mpi-comm@cs.utk.edu I would like to have my name added to the mailing list. Steve... From owner-mpi-comm@CS.UTK.EDU Fri Jan 15 15:31:08 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17910; Fri, 15 Jan 93 15:31:08 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13121; Fri, 15 Jan 93 15:29:52 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 15:29:51 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13112; Fri, 15 Jan 93 15:29:50 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA13694; Fri, 15 Jan 1993 15:29:49 -0500 Date: Fri, 15 Jan 1993 15:29:49 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9301152029.AA13694@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Communication contexts I thought the discussion of communication contexts at the last Dallas meeting was lacking in focus, and I'm afraid the existing subcommittees may not deal with this area. If there are enough people interested perhaps there should be a separate communication context subcommittee. If you have any strong opinions on this please let me know. Also let me know if you would like to be on the communication context subcommittee if one is established. On the topic of communication contexts I think (at least) 4 approaches have been proposed. 1) Implicit communication contexts as in MPI1 controlled by push/pop. 2) Explicit registration as in Zipcode 3) Explicit communication contexts should be merged with explicit group contexts. Since groups cannot communicate with each other a new communication context can be created by creating a new group with the same members as the current group. 4) Chuck Simmons of Oracle has suggested that communication contexts are really a particular type of thread, and so can be handled using existing threads packages. If you have any ideas, or are strongly for or against any of the above approaches let mpi-comm know so we can initiate a discussion of this topic. David Walker From owner-mpi-comm@CS.UTK.EDU Fri Jan 15 15:48:16 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18263; Fri, 15 Jan 93 15:48:16 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14054; Fri, 15 Jan 93 15:47:41 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 15:47:39 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14046; Fri, 15 Jan 93 15:47:38 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA03636; Fri, 15 Jan 93 14:47:35 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA00709; Fri, 15 Jan 93 14:47:32 CST Message-Id: <9301152047.AA00709@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: Communication contexts Cc: walker@rios2.epm.ornl.gov Date: Fri, 15 Jan 93 14:47:31 CST From: Rusty Lusk | | On the topic of communication contexts I think (at least) 4 approaches have | been proposed. | 1) Implicit communication contexts as in MPI1 controlled by push/pop. | 2) Explicit registration as in Zipcode | 3) Explicit communication contexts should be merged with explicit | group contexts. Since groups cannot communicate with each other | a new communication context can be created by creating a new group | with the same members as the current group. | 4) Chuck Simmons of Oracle has suggested that communication contexts | are really a particular type of thread, and so can be handled | using existing threads packages. | If you have any ideas, or are strongly for or against any of the above | approaches let mpi-comm know so we can initiate a discussion of this topic. I think that 1) contradicts our genral agreement to avoid state as much as possible. 4) is a problem on systems that don't support threads, and we have more-or-less agreed to be consistent with thread packages but not depend upon them. 3) should be discussed along with the converse: that contexts rather than groups should be the primary concept and groups defined in terms of contexts. David, perhaps you could create a mailing list on which this discussion can take place, parallel to the subcommittee mailing lists, so it doesn't have to go to the whole committee. Thanks. Rusty From owner-mpi-comm@CS.UTK.EDU Fri Jan 15 16:13:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18561; Fri, 15 Jan 93 16:13:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16784; Fri, 15 Jan 93 16:12:04 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 16:12:03 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from convex.convex.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16768; Fri, 15 Jan 93 16:12:01 -0500 Received: from mozart.convex.com by convex.convex.com (5.64/1.35) id AA21882; Fri, 15 Jan 93 15:11:54 -0600 Received: by mozart.convex.com (5.64/1.28) id AA01950; Fri, 15 Jan 93 15:11:17 -0600 Date: Fri, 15 Jan 93 15:11:17 -0600 From: weeks@mozart.convex.com (Dennis Weeks) Message-Id: <9301152111.AA01950@mozart.convex.com> To: mpi-comm@cs.utk.edu Subject: Communication contexts Cc: lusk@antares.mcs.anl.gov, walker@rios2.epm.ornl.gov Rusty Lusk writes: > David, perhaps you could create a mailing list on which this discussion can > take place, parallel to the subcommittee mailing lists, so it doesn't have to > go to the whole committee. Thanks. I believe the default inclusion of everyone is a good idea, although creating a separate mailing list might be appropriate if some mpi-comm members wish to "unsubscribe". In the meantime, I will add one brief opinion: I felt that the discussions of contexts in the topology subgroup were quite sufficient, and it would probably be better to have the single subgroup which covers both groups and contexts, so that both will appear in the standard in a style which does not make them conflict with each other, and/or to avoid a proposal where groups and contexts would be totally redundant. I definitely agree that it is an important topic, and that the discussions last week in Dallas were far away from getting a proposal for groups and contexts that would be unanimously acceptable, so much work remains to be done. From owner-mpi-comm@CS.UTK.EDU Fri Jan 15 17:01:57 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19329; Fri, 15 Jan 93 17:01:57 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21190; Fri, 15 Jan 93 17:00:57 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 17:00:56 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21182; Fri, 15 Jan 93 17:00:54 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA20492; Fri, 15 Jan 93 22:00:50 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA14823; Fri, 15 Jan 93 14:59:49 MST Date: Fri, 15 Jan 93 14:59:49 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9301152159.AA14823@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: Communication contexts > | > | On the topic of communication contexts I think (at least) 4 approaches have > | been proposed. > | 1) Implicit communication contexts as in MPI1 controlled by push/pop. > | 2) Explicit registration as in Zipcode > | 3) Explicit communication contexts should be merged with explicit > | group contexts. Since groups cannot communicate with each other > | a new communication context can be created by creating a new group > | with the same members as the current group. > | 4) Chuck Simmons of Oracle has suggested that communication contexts > | are really a particular type of thread, and so can be handled > | using existing threads packages. > | If you have any ideas, or are strongly for or against any of the above > | approaches let mpi-comm know so we can initiate a discussion of this topic. > > I think that 1) contradicts our genral agreement to avoid state as much as > possible. 4) is a problem on systems that don't support threads, and we have > more-or-less agreed to be consistent with thread packages but not depend upon > them. 3) should be discussed along with the converse: that contexts rather > than groups should be the primary concept and groups defined in terms of > contexts. > > David, perhaps you could create a mailing list on which this discussion can > take place, parallel to the subcommittee mailing lists, so it doesn't have to > go to the whole committee. Thanks. > > Rusty I'm not sure, but I believe that Zipcode uses push/pop to control contexts (Tony, please correct me if I'm wrong). I'd like to add a 5th approach, basically the one proposed by Paul Pierce: 5) Contexts are used exclusively to insure that message collisions will not occur if independently developed sub-programs are combined. Contexts and groups are orthogonal. Contexts and threads are orthogonal. Each message has an associated context and tag. Message context is managed by library routines and is completely out of a user's control. Message tag is selected by the user. The method of managing message contexts is a separate issue (assuming we want contexts). Existing proposals are: 5a) Stack-based management (objected to due to hidden states). 5b) Explicit registration with user-defined "names" (probably requires some communication). 5c) Explicit registration by a central authority ("dollar bill" registration mentioned by Jim Cownie.) There is enough confusion about contexts and groups that we may want to keep this discussion open to everyone. If the concensus is to make a separate mailing list, please both of us to it. Thanks, Tom Henderson hender@fsl.noaa.gov Leslie Hart hart@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Fri Jan 15 19:06:47 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21669; Fri, 15 Jan 93 19:06:47 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27214; Fri, 15 Jan 93 19:05:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 19:05:41 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27206; Fri, 15 Jan 93 19:05:40 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA00450; Fri, 15 Jan 93 18:05:36 CST Date: Fri, 15 Jan 93 18:05:36 CST From: Tony Skjellum Message-Id: <9301160005.AA00450@Aurora.CS.MsState.Edu> To: hender@macaw.fsl.noaa.gov, mpi-comm@cs.utk.edu Subject: Re: Communication contexts In Zipcode, contexts have a group-wide scope. Groups of processes can involve more than one context. For instance, if 10 processes are involved in several stages of a calculation, each stage might have a different notion of how to do messaging, but each would use all the processes. Groups are important, because they allow control over the scope of global operations. Contexts are important because they provide guarantees to sub-programs about restricted scope of messages. Contexts are added typing-like information, but which is controlled by the system, not ad hoc by each user program. When they determine to start messaging, sub-programs request a context from a "postmaster general" in Zipcode. Currently, this is a group-oriented request (loose synchronization) resulting in each group member getting the context, and other information. In a lower-level system, simpler tactics might be possible. The way we did it was supposed to avoid possibility of races or other problems like that. To build large-scale software, without globalization of the local use of messaging resources, and to provide scalable algorithms with efficient global operations, both groups and contexts are needed. The push/pop state is an implementation detail of little importance. -Tony From owner-mpi-comm@CS.UTK.EDU Sat Jan 16 02:07:09 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26026; Sat, 16 Jan 93 02:07:09 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11696; Sat, 16 Jan 93 02:06:16 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sat, 16 Jan 1993 02:06:15 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11688; Sat, 16 Jan 93 02:06:13 -0500 Received: from id.wmich.edu (id.cs.wmich.edu) by cs.wmich.edu (4.1/SMI-4.1) id AA06036; Sat, 16 Jan 93 02:01:08 EST Date: Sat, 16 Jan 93 02:01:08 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9301160701.AA06036@cs.wmich.edu> To: mpi-comm@cs.utk.edu hi; It was my feeling that the concepts of group and context, that were clearly understood differently by different people at the start of the discussions, became somewhat better agreed on as the pt2pt discussion took place. Groups - used to define groups of processors for collective communication primitives and to allow mapping topologies onto processors. This allows a group of processors to be used in isolation to compute together. Contexts - As Tom (and Tony) express below, > From: hender@macaw.fsl.noaa.gov (Tom Henderson) > Message-Id: <9301152159.AA14823@macaw.fsl.noaa.gov> > To: mpi-comm@cs.utk.edu > Subject: Re: Communication contexts > Status: R > > I'd like to add a 5th approach, basically the one proposed by Paul Pierce: > > 5) Contexts are used exclusively to insure that message collisions will not > occur if independently developed sub-programs are combined. Contexts and > groups are orthogonal. Contexts and threads are orthogonal. Each message > has an associated context and tag. Message context is managed by library > routines and is completely out of a user's control. Message tag is > selected by the user. The only expression stated so far about the difference between tags and contexts I had gleaned was that context should now be wildcardable (IE no -1 for All contexts) while a receive ALL or some other MASK variant would be allowed on tags. > > The method of managing message contexts is a separate issue (assuming we > want contexts). Existing proposals are: > > 5a) Stack-based management (objected to due to hidden states). > 5b) Explicit registration with user-defined "names" (probably requires > some communication). > 5c) Explicit registration by a central authority ("dollar bill" > registration mentioned by Jim Cownie.) Add: 5d) A context is not strictly managed, but use should follow suggested guidelines. This could allow static allocation of contexts when reasonable, but also support a context server (as the Postmaster General in Zipcode). Think of the static contexts or non-registrar provided contexts as reserved with respect to any registration system. This could be part of the MPI initialization call, if a registrar is included. The drawback here of course is the user could violate whatever guidelines were given. It seems to me most types of name registration or stack allocation schemes, that are not too restrictive and allows user code to interact, are likely to allow errant management of contexts as well. I expect that allowing 5d) would let strictly controlled context systems to be built over MPI. > From: Tony Skjellum > Message-Id: <9301160005.AA00450@Aurora.CS.MsState.Edu> > To: hender@macaw.fsl.noaa.gov, mpi-comm@cs.utk.edu > Subject: Re: Communication contexts > Status: R > > In Zipcode, contexts have a group-wide scope. Groups of processes can Of course if we eliminate groups from the pt2pt call then contexts may not be explicitly limited to groups in MPI. (though a context server could be asked to provide a context relative to a group). > involve more than one context. For instance, if 10 processes are involved > in several stages of a calculation, each stage might have a different notion > of how to do messaging, but each would use all the processes. Groups > are important, because they allow control over the scope of global operations. > Contexts are important because they provide guarantees to sub-programs > about restricted scope of messages. Contexts are added typing-like > information, but which is controlled by the system, not ad hoc by each > user program. When they determine to start messaging, sub-programs > request a context from a "postmaster general" in Zipcode. Currently, this ... > To build large-scale software, without globalization of the local use of > messaging resources, and to provide scalable algorithms with efficient > global operations, both groups and contexts are needed. The push/pop > state is an implementation detail of little importance. I expect the use push/pop for context state would make implementation in some environments hard and I have some concern about requiring registration on very large systems. I do think a strictly controlled stack context system could be built over 5d). The question about context management relates to how high a level MPI should be at. I see it at a low level, that higher level systems can be built on. That is one reason for keeping context management "advisory". > > -Tony > I liked the zipcode documentation. I think it would be a good idea to make it available on netlib. john From owner-mpi-comm@CS.UTK.EDU Sun Jan 17 11:46:39 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04093; Sun, 17 Jan 93 11:46:39 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09441; Sun, 17 Jan 93 11:43:57 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sun, 17 Jan 1993 11:43:56 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09433; Sun, 17 Jan 93 11:43:55 -0500 Received: from id.wmich.edu (id.cs.wmich.edu) by cs.wmich.edu (4.1/SMI-4.1) id AA08023; Sun, 17 Jan 93 11:38:43 EST Date: Sun, 17 Jan 93 11:38:43 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9301171638.AA08023@cs.wmich.edu> To: mpi-comm@cs.utk.edu Although the question of IO specification in the MPI-1 has been raised before, I think it still needs to be addressed. Omission of any IO requirements means no program can be portable. If IO specifications are not included the introduction should make a strong case for the reasons. Any attempt to specify high level parallel IO at this point in time seems premature. Perhaps there can be a middle ground. We could specify low level IO, compatible with ANSI/ISO/POSIX standards and perhaps allow an inquire function to test for its presence. I will suggest every process should have access to Level-1 IO, given in the following list, which works for c and c++. -------------------------------------------------------------------- Level-1 Input and Output A process should have access to: 1) open/close/flush read/write lseek 2) readv/writev 3) fcntl dup dup2 ioctl 4) fopen/fclose/fflush scanf/printf fscanf/fprintf sscanf/sprintf 5) varargs vscanf/vprintf vfscanf/vfprintf vsscanf/vsprintf Behavior of these functions is to match that in current standards (specific standard to be chosen). -------------------------------------------------------------------- This could be loosened to only require defined behavior when access is by a single process. A Like Level-1 IO can be specified for Fortran-77 as well. This type of facility is provided in many vendor environments and several portable environments as well. A prototype version of the Level-1 IO could be built using gnu sprintf/sscanf, MPI-1 message passing and the assumption that some process can do IO. -------------------------------------------------------------------- Level-2 Input and Output A case can be made for Level-2 Input and Output as well. This would be group IO, no claim would be made for efficiency, only that the behavior was defined and would not overload the communication network. All processes in a group would execute the same IO calls as in level-1 and the joint behavior would be defined. To avoid state a modified name for each function could be used. A generic level-2 IO could be built on level-1 and the rest of MPI-1. -------------------------------------------------------------------- I propose we include a Level-1 specification and I think we should consider Level-2 as well. I am willing to draft a Level-1 suggestion and a Level-2 suggestion. I'm not sure which mailing list to put this on, but I think the issue needs to be addressed quickly (even if the decision remains to ignore IO). john From owner-mpi-comm@CS.UTK.EDU Mon Jan 18 09:30:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14975; Mon, 18 Jan 93 09:30:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16251; Mon, 18 Jan 93 09:29:26 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 18 Jan 1993 09:29:25 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16243; Mon, 18 Jan 93 09:29:24 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA12246; Mon, 18 Jan 1993 09:29:23 -0500 Date: Mon, 18 Jan 1993 09:29:23 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9301181429.AA12246@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: MPI news 1) NEW SUBCOMMITTEE A new subcommittee on profiling has been set up. It is chaired by Jim Cownie of Meiko, and you can send email to it at mpi-profile@cs.utk.edu. Please let me know if you want to join 2) NEWCOMERS FILE If you are a newcomer to the MPI forum or want general information send the message send mpi-newcomers from mpi to netlib@ornl.gov 3) REVISED VERSION OF MPI1 TECH REPORT A revised version of the MPI1 paper by Dongarra, Hempel, Hey , and Walker is available from netlib. This is basically the original document with some unnecessary and inappropriate routines removed. The PostScript file can be obtained by sending the message send mpi1.ps from mpi to netlib@ornl.gov 4) NEXT MPI MEETING Here are some details on the MPI meeting which is set for February 17th-19th 1993 in Dallas. The meeting site will be the: Bristol Suites 7800 Alpha Road Dallas, TX 214-233-7600 The room rate is $89.00. When making a reservation tell them you are with the MPI meeting. TBS Shuttle Service will be providing complimentary shuttle service to and from the airports. If you fly into DFW, use their courtesy telephone and dial 03. If you fly into Love Field, you'll have to use a pay phone. They can be reached at 817-267-5150. Upon boarding the shuttle, refer to the MPI meeting. The registration fee for the meeting will be $75. Please make checks and POs payable to University of Tennessee. We will collect this at the meeting. The registration fee will go for coffee breaks, meeting rooms, AV and printer rentals. We should plan to start at 1:00 pm February 17th and finishing about noon on February 19th. The format of the meeting is: Wednesday, February 17 1:00 pm to 8:00 pm point to point subcommittee meeting 5:00 pm to 6:00 pm--On our own for dinner. after dinner: other subcommittees meet Thursday, Febrauary 18 8:00 am to 12:00 pm Collective communication subcommittee meeting 1:00 pm to 6:00 pm Subcommittee reports presented to the main group 6:00 pm to 8:00 pm-- The group dinner somewhere in the area. The hotel will provide round trip van transportation. Friday, Febraury 19 8:00 am to 12:00 pm Subcommittee reports presented to the main group For future planning here is a tentative list of dates, roughly 6 weeks apart, for the series of meetings: March 31-April 2 May 19-21 June 30-July2 From owner-mpi-comm@CS.UTK.EDU Mon Jan 18 14:20:28 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21493; Mon, 18 Jan 93 14:20:28 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27089; Mon, 18 Jan 93 14:19:01 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 18 Jan 1993 14:19:00 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27075; Mon, 18 Jan 93 14:18:59 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA15953; Mon, 18 Jan 1993 14:18:58 -0500 Date: Mon, 18 Jan 1993 14:18:58 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9301181918.AA15953@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Contexts group A new subcommittee and associated email group has been set up to responsible for MPI communication contexts. The current group membership is as follows: tony@cs.msstate.edu dongarra@cs.utk.edu lusk@mcs.anl.gov weeks@convex.com hender@fsl.noaa.gov john@cs.wmich.edu csimmons@us.oracle.com walker@msr.epm.ornl.gov If you would to join (or leave) this subcommittee please send email to me. You can send email to the group at mpi-context@cs.utk.edu. You can get the archived email for this group by sending the following message: send mpi-context from mpi to netlib@ornl.gov Regards, David From owner-mpi-comm@CS.UTK.EDU Tue Jan 19 12:29:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19476; Tue, 19 Jan 93 12:29:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24138; Tue, 19 Jan 93 12:28:28 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 19 Jan 1993 12:28:27 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24130; Tue, 19 Jan 93 12:28:25 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA26202; Tue, 19 Jan 93 17:28:13 GMT Received: by nipmuc.fsl.noaa.gov (4.1/SMI-4.1) id AA02004; Tue, 19 Jan 93 10:30:14 MST Date: Tue, 19 Jan 93 10:30:14 MST From: hart@nipmuc.fsl.noaa.gov (Leslie Hart) Message-Id: <9301191730.AA02004@nipmuc.fsl.noaa.gov> To: mpi-comm@cs.utk.edu, john@cs.wmich.edu > From: john@cs.wmich.edu (John Kapenga) > To: mpi-comm@cs.utk.edu > > Although the question of IO specification in the MPI-1 has been > raised before, I think it still needs to be addressed. Omission > of any IO requirements means no program can be portable. If IO > specifications are not included the introduction should make a > strong case for the reasons. We (Tom Henderson and myself) agree and have also suggested such a proposal. Ours had a little more high level stuff, but not much. We do think you need some specification of how basic I/O performs in a parallel environment to provide a high level of portability. We think opportunities for parallelism can be expressed as well by routines that are loosely snychronous. > ... > Any attempt to specify high level parallel IO at this point in time > seems premature. Perhaps there can be a middle ground. We could > specify low level IO, compatible with ANSI/ISO/POSIX standards > and perhaps allow an inquire function to test for its presence. > I am willing to draft a Level-1 suggestion and a Level-2 suggestion. > I think that is a good idea and would be interested in cooperating on such an endeavor. > I'm not sure which mailing list to put this on, but I think the > issue needs to be addressed quickly (even if the decision remains > to ignore IO). Neither am I, let's hope it's not another subcommittee ;-) > > john > Leslie Hart (hart@fsl.noaa.gov) From owner-mpi-comm@CS.UTK.EDU Tue Jan 19 12:32:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19727; Tue, 19 Jan 93 12:32:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24283; Tue, 19 Jan 93 12:31:27 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 19 Jan 1993 12:31:25 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24275; Tue, 19 Jan 93 12:31:22 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA26204; Tue, 19 Jan 93 17:28:14 GMT Received: by nipmuc.fsl.noaa.gov (4.1/SMI-4.1) id AA02007; Tue, 19 Jan 93 10:30:15 MST Date: Tue, 19 Jan 93 10:30:15 MST From: hart@nipmuc.fsl.noaa.gov (Leslie Hart) Message-Id: <9301191730.AA02007@nipmuc.fsl.noaa.gov> To: mpi-comm@cs.utk.edu, bernardo@fsl.noaa.gov, hart@fsl.noaa.gov, hender@fsl.noaa.gov, csimmons@us.oracle.com Subject: Re: "Message Capsule" Proposal (long) > From: Charles Simmons > To: bernardo@fsl.noaa.gov, hart@fsl.noaa.gov, hender@fsl.noaa.gov > Subject: Re: "Message Capsule" Proposal (long) > > As feedback... > > The message capsule proposal might be a bit high level. E.g. it > seems to implement the topmost layer of the ISO 7-layer communications > model which deals with transmitting data between processes running > on heterogeneous machines. I'm not familiar with with the ISO 7-layer approach, but what we were proposing was a place for each local process (or thread) to store information about one or more messages. Other words for capsule might be "handle" or "envelop". > > " > 6.1 Advantages > > - The number of separate send and receive routines is greatly reduced > without sacrificing functionality. > - A user who is used to "common practice" can use the simplified > versions of the routines. > - A user who wants more flexibility only needs to learn about the features > required for his/her specific application. (For example, if I only need > contiguous messages, then I don't need to know anything about strided data > items. If the receiving process always knows the data description of > received messages, then I don't need to know about the probe and "info" > routines.) > - "Hidden states" are removed so multi-threaded applications won't get > confused. > - Encapsulation of features in message capsules allows new features to be > added later without modifying syntax of existing routines. A new feature > would require addition of one or more new routines to modify and examine > message capsules. > " > > While the number of separate send and receive routines is reduced, > the number of other routines are correspondingly increased (e.g. attach > and info routines). The increase in routines is not combinatorial, but rather additive. For three new modes each of which have two possible settings, there is a maximum of six (6) new routines (ok, maybe 12 if there are other inquiry functions). In another approach the number of routines would be *mulitplied* by two for each new mode (i.e. multiplied by 8). > > Even in the MPI approach, users can limit themselves to simple versions > of various rotuines. Agreed, but I don't think the document has been taken to heart as a syntactical or semantical starting point. > > I don't see how capsules keep multi-threaded applications from > getting confused. If you have two threads in a receiver that are > each waiting for messages that look pretty much the same to the OS > or compiler or library code, it would appear to be fairly easy to > give the messages to the wrong threads. The sender really needs > to attach a thread identifier to a message under certain circumstances. > This responds to some issues that hidden state cause with multithreading. Such as what does the last message mean? We don't claim that it deals with which thread should be a target in a send. It just removes confusion if you probe and then wish to receive the message that you probed about. > > [There was a brief thought I had that this proposal reminded me of... > For certain types of applications, it would be possible for the sender > to know the layout of data in a receiver's address space. In an RPC > application, the receiver could transmit a description of the layout > to the sender. In an SPMD application, the sender might just know without > being told. This would allow the sender to prepend the layout description > to her message, and would allow the receiving OS to splatter received data > directly into user address space.] > > G'luck, Chuck Thanks for your input, we wanted to start a discussion about these issues. (BTW, your response was directly to us, so we included it in its entirety to mpi-comm). Leslie Hart Tom Henderson Bernardo Rodriguez hart@fsl.noaa.gov hender@fsl.noaa.gov bernardo@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Tue Jan 26 16:04:52 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21467; Tue, 26 Jan 93 16:04:52 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20893; Tue, 26 Jan 93 16:03:52 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 26 Jan 1993 16:03:51 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from enet-gw.pa.dec.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20885; Tue, 26 Jan 93 16:03:47 -0500 Received: by enet-gw.pa.dec.com; id AA27890; Tue, 26 Jan 93 13:03:06 -0800 Message-Id: <9301262103.AA27890@enet-gw.pa.dec.com> Received: from rdvax.enet; by decwrl.enet; Tue, 26 Jan 93 13:03:45 PST Date: Tue, 26 Jan 93 13:03:45 PST From: MPS ENGINEERING 223-4656 To: mpi-comm@cs.utk.edu Cc: benson@rdvax.enet.dec.com Apparently-To: mpi-comm@cs.utk.edu Subject: POSIX 1003.4 message queues considered ? Has anyone involved in the MPI effort looked into POSIX 1003.4 (Realtime) message queues ? -Ed From owner-mpi-comm@CS.UTK.EDU Wed Jan 27 12:03:40 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08441; Wed, 27 Jan 93 12:03:40 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09490; Wed, 27 Jan 93 12:02:33 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 27 Jan 1993 12:02:31 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09480; Wed, 27 Jan 93 12:02:24 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA02128; Wed, 27 Jan 93 11:02:21 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA16675; Wed, 27 Jan 93 11:02:16 CST Message-Id: <9301271702.AA16675@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: minutes of January MPI meeting Date: Wed, 27 Jan 93 11:02:15 CST From: Rusty Lusk Minutes of the Message Passing Interface Standard Meeting Dallas, January 6-8, 1993 The MPI Standards Committee met in Dallas on January 6-8, 1993, at the Bristol Suites Hotel in North Dallas. This was the third meeting of the MPI committee, but the first following the format used by the High Performance Fortran Forum. There were both general meetings of the committee as a whole and meetings of several of the subcommittees. Because interest in the Point-to-Point communications and the Collective communications was so general, these met as committees of the whole. No formal decisions were taken at this meeting, but a number of straw votes were taken in the subcommittees. These are included as part of the reports on the work of the subcommittees. These minutes were taken by Rusty Lusk (lusk@mcs.anl.gov) with some additions by Bob Knighten. Marc Snir's notes on the point-to-point subcommittee meetings are included here as well. These minutes are quite long. If you want to see the important topics you can search for --- and this will quickly the lead to each topic (and a few other things.) January 6 --------- ------------------------------------------------------------------------------- General Meeting ------------------------------------------------------------------------------- The meeting was called to order by Jack Dongarra at 1:30. Jack Dongarra presented the rules and procedures that had been circulated in the mailing list. In general, they say that we intend to operate in very open fashion, following the example set by the High-Performance Fortran Committee. He also described the subcommittee structure. For details, see the mailing list, A tentative schedule for future meetings was presented, which was amended on the last day (see there). All meetings will be in Dallas at the Bristol Suites. Steve Otto will coordinate the production of the document. He will obtain a set of LaTeX macros from the HPF Committee and distribute them to the subcommittee heads. It was suggested by Bob Knighten that the Executive Director arrange for copies of all pertinent documents be provided at the meetings. Dennis Weeks, who is somewhat local (Convex), volunteered to help with the relevant copying. The attendees were: Ed Anderson Cray Research eca@cray.com James Cownie Meiko jim@meiko.co.uk Jack Dongarra UT/ORNL dongarra@cs.utk.edu Jim Feeney IBM-Endicott feeneyj@gdlvm6.vnet.ibm.com Jon Flower ParaSoft jwf@parasoft.com Daniel Frye IBM-Kingston danielf@kgnvma.vnet.ibm.com Al Geist ORNL gst@ornl.gov Ian Glendinning Univ. of Southampton igl@ecs.soton.ac.uk Adam Greenberg TMC moose@think.com Bill Gropp ANL gropp@mcs.anl.gov Robert Harrison PNL rj_harrison@pnl.gov Leslie Hart NOAA/FSL hart@fsl.noaa.gov Tom Haupt Syracuse U. haupt@npac.syr.edu Rolf Hempel GMD hempel@gmd.de Tom Henderson NOAA/FSL hender@fsl.noaa.gov C. T. Howard Ho IBM Almaden ho@almaden.ibm.com Steven Huss-Lederman SRC lederman@super.org John Kapenga Western Michigan Univ. john@cs.wmich.edu Bob Knighten Intel SSD knighten@ssd.intel.com Bob Leary SDSC leary@sdsc.edu Rik Littlefield PNL rj_littlefield@pnl.gov Rusty Lusk ANL lusk@mcs.anl.gov Barney Maccabe Sandia abmacca@cs.sandia.gov Phil McKinley Michigan State mckinlehy@cps.msu.edu Chuck Mosher ARCO ccm@arco.com Dan Nessett LLNL nessett@llnl.gov Steve Otto Oregon Graduate Institute otto@cse.ogi.edu Paul Pierce Intel prp@ssd.intel.com Peter Rigsbee Cray Research par@cray.com Ambuj Singh UC Santa Barbara ambuj@cs.ucsb.edu Marc Snir IBM snir@watson.ibm.com Robert G. Voigt NSF rvoigt@nsf.gov David Walker ORNL walker@msr.epm.ornl.gov Dennis Weeks Convex weeks@convex.com Stephen Wheat Sandia NL srwheat@cs.sandia.gov ------------------------------------------------------------------------------- Point-to-point subcommittee ------------------------------------------------------------------------------- Mark Snir called the meeting to order at 1:40 p.m. It adjourned at 4:10 p.m. It resumed the following morning at 9:10 a.m. and adjourned at 4:15 p.m. Marc Snir began by summarizing the decisions that we have to make: * which operations? send receive channels? sendreceive? info arguments operations on queues probe? * operation modes sync async local and/or global termination interrupt-driven? * message types (data types) structure of data in core buffer packing * send-receive matching type (We later decided to call this "tag".) sender? * correctness criteria (See Marc Snir's paper in handouts) * heterogeneous operations * name space how processes are addressed flat? structured? implicit/explicit * error handling * interaction with threads, interrupt handlers, remote signalling * special operations for high performance ready receiver? * process startup * syntax/style (The plan is to postpone this for this meeting.) We will prioritize this list and then go through them one by one. (The priorities assigned were more or less in the order listed above.) Two preliminary questions were then discussed: A. Must we worry about multithreaded environments? James Cownie pointed out that threads were coming, in almost all new systems. Most systems have threads now. It was proposed that a process, which could send and receive messages, should be an address space, so that individual threads would not be (MPI-) addressable. B. What about signals? Paul Pierce suggested that we discuss signals first: do we want to support send/receive from interrupt handlers? These two questions were then discussed at length. Dealing with threads argues against the notion of "last message", since that implies state is maintained by the system. There was general agreement that "state" was a ` bad thing, but arguments in favor of state are: Sometimes one doesn't want all of the information available after an operations, so it shouldn't be returned. Having lots of arguments to calls is bad, especially inout arguments. Ways to avoid state are: Structures could be returned Return individual arguments Return tag to do queries on (but they one needs to free it.) Additional out arguments (OK in Fortran 90, but not in C or f77) User passes in storage to be used (so he knows the address), and MPI provides inquiry functions [For more details, see Jim Cownie's mail message of January 4, 1993 entitled: Multifarious] There was a general agreement that system state decreases portability and manageability, and we should decrease it when we can. James Cownie said that We need a reentrant style, and Mark Snir suggested that we try to make all function calls reentrant. When queried no one in the group objected to trying to make all the functions that are introduced in MPI reentrant. Now we began going through the above-mentioned major topics. Which Operations? ----- ---------- We have send and receive. How about send-receive (also called shift)? It can be efficiently implemented, and buffer can be reused. There was a discussion of the "two-body" send-receive (exchange) and the "three-body" version (ring-shift). Variations include those in which the send-buffer is required to be the same as the received- buffer and those in which is is required to be disjoint from the receive-buffer. Al Geist: We should focus on *required* operations. Steve Otto replied that send-receive *is* a required operation. Using "exchange" can help avoid deadlock. It was agreed that there was no consensus on these issues and it was decided to defer this to the collective communication subcommittee. Operation Modes --------- ----- The next topic that Marc Snir raised for discussion was when do send and receive return. Marc described several options: For send: 1) return as soon as possible 2) return when send-buffer is clear 3) return when the corresponding receive has completed For receive: 1) return as soon as possible 2) return when the receive-buffer is full "Receive has completed" means "when the user knows". In other words, when the sender has returned from send, the receiver has returned from receive. There was a general discussion about whether 3) was necessary? dangerous? Robert Harrison said he believed that 3) was the minimal version that was truly portable. Steve Otto pointed out that 3) is CSP-like. Rusty Lusk said that 3) would be easier to prove things about than the others. Adam Greenberg and Paul Pierce pointed out that neither TMC nor Intel have implemented an operation depending on the behavior of the receiver. A straw vote was taken and the vote was 17-3 in favor of having 3) as an option. Marc Snir pointed out that in his original proposal send returns a handle and the status of the handle is then tested for completion of the send operation, and asked if this is this desirable. There was general agreement that something of this sort was desirable, but a variety of alternatives were mentioned It was pointed out that sometimes one wants to wait on multiple outstanding operations. Al Geist prefers separating "wait" into "sendwait" and receivewait" for code readability. Bill Gropp suggested that instead of using handles, one could supply a routine to be called when an operation completes. James Cownie: "This gets really hairy in Fortran". There was a discussion of probing multiple outstanding receives: If the receives return handles, h1 = recv( ... ) h2 = recv( ... ) wait ( h1 or h2) ? wait ( h1 and h2 ) is not needed. Jame Cownie proposed that we supply an operations to *wait* on a vector of handles, which would return one of those that have succeeded. It would return the handle, not the status. A straw vote as taken on this proposal, which passed 17 - 0. So we have: status (handle) wait (array of handles) The send specifies what completion of send means. Handles need to be freed. It was pointed out the only the existence of such an operation has been decided, the semantics are yet unspecified - e.g. issues such as fairness or what wait returns when several complete are not yet specified. There was a long discussion of cancellation of send and receive. It was observed that there are serious implementation problems because of race conditions, freeing resources, etc. A straw poll was taken on including cancel in the initial MPI. It failed 7-19. This was the end of the Wednesday afternoon point-to-point meeting. January 7 --------- The point-to-point subcommittee (now a Committee of the Whole) resumed at 9:15 a.m. on Thursday morning. Marc Snir opened the meeting and summarized the progress so far: 3 ways in which send can terminate sendreceive postponed no cancel of incomplete send operation status and wait (successful status accomplishes same as wait) We did not get to: channels (the idea of trying to bind as soon as possible as many parameters as possible, so that they can be reused.) probe readyreceive Marc noted that channels and readyrecv address similar issues. Probably want only one of these. Do we want either? Rolf Hempel observed that we don't need channels - can depend on operating system to cache the connection information when doing synchronous communication. Adam Greenberg replied: NO! Want to be able to do this all at user level without "smart" OS. Channel creation and use might look like: handle = send_init( ... ) start(handle) wait(handle) free(handle) This is an intermediate point between bundled send/receive and full named channels. Indeed there are many intermediate points based on various early bindings. Is there enough experience to justify a standard? Bob Knighten observed that there has been substantial experience with channels on the iWarp system. There was next a discussion of the ready-receiver semantics proposed by Lusk and Gropp in the handouts. Steve Huss-Lederman said that such operations could make a difference of as much as 25% for matrix multiplication on the Delta. Some doubt was expressed about the universality of this optimization. Question of use of readyrecv by naive users again. Cownie mentions experience again. Greenberg: facilities for efficiency should not make it difficult to write correct programs. Wheat: Don't penalize users who do understand and can take advantage of efficient procedures. General back and forth discussion. Two straw votes were taken: Ready-receiver operations passed 13 - 10 Channels passed 19 - 2 (Marc Snir will write up a detailed proposal) The next topic discussed was the probe operation. Do we want such an operation, and if so, what should be its semantics? Probing must "lock" the message that it finds, else the information returned by the probe may be unreliable. (Consider the multithreaded environment.) Bill Gropp pointed out that probe is often used to find out the length of a message so that a buffer of the appropriate size can be allocated. Marc Snir pointed out that this is a problem with the November document, that we need to know the length of a message ahead of time. Jon Flowers suggested the need for a blocking probe. What is needed is to probe and then to receive the message found via the probe: handle = probe(params) . . . recv(handle) release(handle) Marc Snir pointed out that the handle serves as a lock on the message. James Cownie pointed out that while we agreed to not have a cancel for a send, we do need to be able to cancel receives, since an outstanding receive is permission for the system to write in the user's address space, which is a permission the user may want to revoke. A straw vote was taken on the existence of some form of probe, and it passed 25 to 0. Send-Receive Matching ------------ -------- The next topic is the matching of send and receive. Currently we have to discuss matching on: tag sender group id context id We will also need to discuss the name space issue for messages. Here are three proposals for the predicate that determine whether a message matches a particular receive: 1) simple matching on fields 2) more general, with mask, etc. 3) user defined function Adam Greenberg said that at TMC: A user defined function is used by the system whenever a message is received by a node to decide if it is to actually be received by the application. The parameters to the user defined receive predicate are tag and group. Issue: If most information is encoded in the tag, then the tag protocol must be understood by all users involved in writing a particular application. True, but not a serious problem. Best to identify small class of specific matching parameters (e.g. group) and use tag for everything else. James Cownie pointed out that the matching function, if not too complicated, can (and is, on many systems) done by special communications processors. There was further discussion of the difficulties of having the system call user code for screening messages. Paul Pierce pointed out that receipt of a message by the hardware is a crucial point for performance. There was general discussion of alternative approaches to getting at least some of this. The question of need for this generality was also raised. TMC has a user who wants and uses his own predicate function. Possibilities: (a) select on mask for fields (including a don't care); (b) simple static logical operations on fields; (c) user defined (b) might be match = AND (( message(i) = pattern(i) ) OR mask(i)) fields A straw vote was taken on whether to pursue allowing user-defines predicates. It was decided 26-1 not to allow user-defined functions for this purpose. (b) deferred until a proposal is available. Marc Snir summarized that matching by tag is generally agreed on and that this is not the only item for selection. After some discussion, matching by sender was also generally agreed on. So now, how do we identify a sender? Rusty Lusk spoke in favor of a flat name space, so that processes could be addressed independently of group, etc. There ensued a general discussion of groups, contexts, and the name space. It was pointed out that the name space expected by send could be flat and groups could be implemented by a function that converted any structured name into a flat integer id. Other proposals were to to have name=(rank,gid) with the restriction that this name be usable only within the given group (gid) and the sender must be a member of this group. By default the group would be ALL. Other alternatives mentioned were name=(rank,ALL)=pid and name=(pid,context). This led to a general discussion of context and the relation to groups. Marc Snir pointed out that we could have pid pid,context in which context did not change the meaning of pid. Paul Pierce said that tags and contexts should be separated since they need to be handled in different ways. Marc Snir pointed out that there should be no "don't care" on context. There was a discussion of servers that can process "any" message. This also led into discussion of flat name space vs. hierarchical name space where we would have pid(group, rank) function. Can use context to define groups, but there are other uses as well. Why groups as well as context? What is the difference between context and groups? Cownie: Context is just another integer used in the same manner as tag. Not quite - it is reserved, but what is the meaning of "reserved"? Greenberg was concerned about connecting send/receive behavior with groups. Snir: Suppose a users wants to have two independently written subroutines that use the usual rank notation. Wheat: Similarly want to use rank notation when partitioning machine. Snir: Both contexts and groups are nice, but do we need both? Gropp: Problem with mixing two applications both of which use 0-based indexing will need a larger common name space when they need to communicate. There was a general discussion of the cost of contexts. Cownie observed that context is cheap if only used to distinguish code - obtain a unique context id for the code by means of the "one-dollar random number generator": each author obtains a one-dollar bill, copies the serial number, and then burns the bill. But in general context is not cheaper than groups. Someone asked about spawning additional processes while program is running? Various people raised the question: If use name=(pid,context), does context change the meaning of the pid (i.e. is pid context {or gid or ???} relative.) There was some discussion of message registration. Paul Pierce observed that tag vs. context is only matter of registration. He wants to divorce tag and context for safety. This implies that one cannot use wild card for selecting on context. Various people noted difficulties with mixing tag and context. Adam Greenberg offered: Proposal - always separate tag and context. Have a context, NONE, so that pid with context NONE is unmodified, but with other contexts the pid may be relative. [NONE, GLOBAL, BASE] tag, context - must match on context Several people noted that there are two very different uses of context - identification of distinct code and identification of a group of processors. There is state, even distributed state, associated with remapping of processors with groups. POSSIBLE FIELDS FOR SEND/RECEIVE: tag context id group - no wild card - set of processes - registration management - receive only from group - managed by system Marc Snir asked whether we could agree on what would be carried with a message: tag context (like tag, except no wild card; management to be determined) Two straw votes were taken: Having contexts passed unanimously. Having the context *not* modify the process id passed unanimously. Groups ------ Three alternatives: no groups (use send(pid(group, rank), ...) instead) group as explicit parameter in operations use contexts to implement groups The basic difference is: do we want to be able to select on group? Straw vote: yes: 10 no:11 on the capability of selecting by group. (Thursday lunch occurred here) Message Data Types ------- ---- ----- WHAT IS A BUFFER? (Language bindings are going to be important here.) There are many options to consider: a) contiguous bytes (non-controversial) General agreement that 0-length messages should be allowed. b) contiguous buffers of other (implementation specific?) units? b) stride? (parameters: base-address, element-length, stride, number-of-elements) c) basic data types? d) arbitrary structures? e) How will we specify data to be communicated in a heterogeneous environment? f) iovec structures (array of pointers and lengths, as in un*x iovec for reads and writes) Marc Snir pointed out that one possibility is to have separate pack/unpack routines and then just send contiguous buffers. Rusty Lusk pointed out that this requires a copy that may be unnecessary on certain machines. Two choices - pack scattered buffer and send OR send scattered buffer. If the second, then may need a pack that produces the descriptor of a scattered buffer to be used by the send scattered buffer. Straw poll: Use IOVEC type send. Passed 18-1 Basic data types were deferred. Marc Snir observed that up to this point, a message is a set of bytes in storage, but now we are about to consider more meanings: message = sequence of *values* Should we use the same calls for homogeneous and heterogeneous calls? Can we have a fast homogeneous implementation of the heterogeneous calls? Bill Gropp pointed out that the current testbed implementation does this. SEND vs. SENDCHAR, SENDREAL, . . . To be compliant with F77 need to have at least SENDCHAR for correctness (and this is a real issue, e.g. on VAX.) Strictly need to have different for each basic data type (but in practice this is not an issue.) But for other than CHARACTER there is also an efficiency issue. 1. F77 conformance 2. Special problem of CHARACTER 3. Performance 4. Heterogeneity (?) Postpone to language binding discussion. This led into the issue of the general problem of converting types between languages and machines! This in turn led to a discussion of XDR (and mention of other systems such as NDR, ...) XDR supports the basic types (INT, REAL, COMPLEX, CHAR, etc.), array constructors, pack/unpack routines, etc. Do we use the same calls for homogeneous and heterogeneous systems? Can we have a fast implementation of heterogeneous procedures for a homogeneous system? What about a "message envelope" that specifies the environmental aspects of messages (e.g. heterogeneity features such as XDR.) When we talk about heterogeneity, do we expect MPI libraries from different vendors on different machines to cooperate? Include general SEND as SENDBYTES? Agreed that do not want SEND in homogeneous to require type information needed for heterogeneous environment. There was a discussion of whether we have to pick an interchange format, for example XDR. There seemed to be some agreement that we do (as MPI implementations from different vendors have to be able to communicate with one another), but no vote was taken. Error Handling ----- -------- The main issue here is whether an error detected by an MPI routine should result in the calling of an error-handler or return of a return code. Other issues are how much of error handling should be standardized as opposed to implementation-dependent, and how much user control there should be over error-handling. There are two types of error environments - soft (recoverable) and hard (unrecoverable). In a soft error environment there is the opportunity for cleanup on the part of both the "application" and the system, while in the hard error environment the system will cleanup and terminate the application. Choices: An mpi routine always returns (though it may return with an error code.) An mpi routine may call an exception handler There may be a default exception handler and there could be a user-installable one as well. Library writers may want to handle errors differently from how a user program wants to handle them (or have them handled by the system). Robert Harrison described the error modes used in TCGMSG and p4: A process has a user-settable state that determines whether an error should result in a (hopefully) graceful takedown of the program or in a error return code. Paul Pierce described the Intel method which uses two syntactically distinct classes of functions. For one class an error results in a message being printed and the process in which the error occurred terminating. For the other class an error code is set. There was some discussion of the problem of maintaining state in a multithreaded environment. Two straw votes were taken: Do we want a version of MPI that calls an exception handler: yes: 23 no: 0 Do we want a version with return codes: yes: 19 no: 1 Specific discussion of modes or "shadow" routines was deferred. Correctness Criteria ----------- -------- This concerns defining what is a correct implementation of MPI An assumption that had to be restated several times during the meeting is that MPI assumes a reliable underlying communication system, i.e. MPI does NOT address what happens if that fails. Two specific topics are order of messages and resource bounds. There was discussion about whether order preservation is required; that is, for messages from one process to another, messages are received in the order they are sent. Maintaining message ordering is troublesome, but seems essential for conveniently writing reliable portable programs. But then comes the question of what exactly this means, particularly with multithreaded processes! What is the effect of probe on the ordering of messages? Straw vote in favor of requiring order preservation: yes: 23 no: 4 On the issue of correctness with regard to resource exhaustion, Marc Snir suggested the following example: Process 1 Process 2 --------- --------- send to 2 send to 1 recv recv What should an implementation be required to do with this program? On the CM-5 this will always deadlock. On Intel and Meiko machines this will "usually" work (but how does one specify exactly when it will work.) Exchange is an even nastier case. ------------------------------------------------------------------------------ Summary of both Wednesday and Thursday point-to-point subgroup meetings by Marc Snir 1. Multithreaded systems and signal handlers. Should these be of concern to us? No vote was taken, but the general feeling was that we should try to define the various communication calls so that they do not rule out the case where the communicating process is multithreaded. The implications seems to be that all calls should be made reentrant, and the communication subsystem is, from the view-point of the application code, stateless. (With one with one obvious exception, namely that due to posted receive or send buffers, and perhaps additional exceptions having to do with global "modes", like error handling mode. 2. Small or large library? No vote taken. The general feeling is that we should provide many options for the advanced programmer that wants to optimize code (otherwise, all "interesting" codes will use non-portable features, but set the syntax so that the use that uses only the "core" functions need not be burdened by the availability of advanced options. 3. What functions? Clearly, SEND and RECEIVE General sentiment that a combined send-receive would be nice ("most used function on CM"), but discussion postponed until we have a proposed definition: Do we we want an exchange (source=dest), or a 3-body function (source != dest), or allow for both? do we want send_buffer identical to receive_buffer, or disjoint from receive_buffer, or allow arbitrary overlap between the two? What attributes are shared by sent message and received message, if at all? WAIT, STATUS and PROBE functions, and persistent handles are discussed later. 4. What modes? We want blocking and nonblocking sends and receives (blocking -- returns when operation terminated; nonblocking -- returns as soon as possible and a different call is needed to terminate the operation). We want synchronous and asynchronous modes (Synchronous -- operation terminated when terminated at all participating nodes. Asynchronous -- operation terminated when terminated at the calling node; e.g. a send terminates asynchronously when the sender buffer can be reused. Please let me know if you dislike this terminology and prefer something like "local" and "global".) The vote went 17-2 toward having a synchronous SEND (completes when RECEIVE has completed, i.e. when the corresponding WAIT has returned, or STATUS has returned successfully.) We did not discuss whether we want all 4 combinations of blocking-nonblocking and synchronous-asynchronous, or just 3 (blocking synchronous, blocking asynchronous and nonblocking asynchronous). We did not discussed explicitly, but "kind of assumed" that any SEND mode can match any RECEIVE mode. 5. How does one complete a nonblocking operation? The SEND and RECEIVE nonblocking operations return a handle that can be used to query for completion. WAIT(handle) blocks until operation completed; STATUS(handle) returns as soon as possible, and returns an indication for successful completion. In addition, these operations return information on completed RECEIVES: tag, message length, etc. for the received message. The information is returned in a structure provided by the caller. After return of a WAIT or successful return of a STATUS the operation handle is freed; the system has no more information on the completed operation, and has freed all associated resources. A more complex WAIT is needed, that waits for the completion of one out of several pending operations. Proposed syntax is WAIT(array_of_handles) that returns information on which operation succeeded and its parameters (voted 17 to 0). No CANCEL operation -- Once a SEND or RECEIVE is posted, it must complete. (Voted 19 to 7. Some peoples asked to reconsider at least canceling posted RECEIVEs, even if posted SENDs must complete). 6. Additional operations "ready-receive" SEND. SEND with a promise that a matching RECEIVE is already posted (A program where such SEND occurs with no preceding matching RECEIVE is erroneous and, hopefully, the implementation detects this error.) The justification is "it exists on some machine" and "it can improve performance by 25% on Delta". Accepted by 13 against 10. Persistent handles. Created by SEND_INIT(params) (resp RECV_INIT(params). can now be repeatedly used to send/receive messages with these parameters, and then explicitly destroyed. Supported by 19 against 2. PROBE. Allows probing for messages available to receive. Justification - "provides a mechanism to allocate memory to a message of unknown length, before it is received". The proposed mechanism is PROBE(params) that returns a lock to a matching message if there is a matching message that can be received. This message is now locked and can only be received using this lock. This was voted 25 to 0. Some level of uncertainty whether we should also allow to unlock without receiving (why should one want to do this?) 7. What is the buffer argument in SENDs and RECEIVEs? A message is a sequence of values, and as a particular case which is of most interest for homogeneous systems, and for which the syntax ought be simpler, a message is a sequence of bytes. There are various ways of specifying this sequence of bytes. a. Contiguous message: Starting address and length b. Regular stride message: Starting address, number blocks, length of blocks, stride. Voted with no opposition. c. IOVEC: a list of entries, each of which describes a type a or type b message. Voted 18 against 1. There was no discussion on a concrete proposal for typed messages, short of agreement that there should be such. The standard is not going to propose a concrete encoding of typed messages, and a concrete mechanism for message exchange in heterogeneous systems. 8. Matching of SENDs and RECEIVEs. A SEND operation associates with a message the following attributes. a. Sender id. b. Tag c. Context The idea of associated a group id, too, was rejected 11 to 10. The RECEIVE criterion is a Boolean predicate on these attributes of the form. (SENDER_ID = param1) and (TAG = param2) and (CONTEXT = param3). Don't cares are allowed for sender_id and tag, but not for context. Sender_id is determined by system, in the obvious manner, and is absolute (not relative to a group or a context). Tag is under sender control. Context is under sender control, but a yet to be determined mechanism is used to allocate valid context values to processes so as to prevent conflicts. All this was approved with no opposition. The idea of allowing the user to provide its own Boolean function as a receive predicate was rejected 26 to 1 (Reason: "hard to do if the matching is done by a communication coprocessor".) 9. Error handling a. We need a version of MPI where errors generate exceptions (user program halts when an error is detected in an MPI call, or a specific exception handling mechanism is invoked). Voted 19 to 1. b. we need to provide a version of MPI where calls return error codes, and do not cause exceptions, whenever possible. Voted 23 to 0. 10. Ordering of messages Messages sent from the same source to the same destination "arrive in the order they where sent". Voted 23 to 0. The exact implications in terms of order in which RECEIVEs can occur has to be worked out. It was pointed out that this condition may be somewhat hard to define in a multithreaded environment. End of Marc Snir's summary --------------------------------------------------------------------------- Collective Communication Subcommittee --------------------------------------------------------------------------- The Collective Communication Subcommittee was called to order by Al Geist at 4:30 p.m. on Wednesday. It continued until 6:40 p.m. when there was a break for dinner. The meeting resumed at 8:25 p.m. and finally adjourned at 10:10 p.m. Al Geist introduced this as the first meeting, since no real discussion on groups and collective communication took place in Minneapolis. One goal of this committee is to maintain consistency with the point-to-point operations. Any discussion of groups necessarily involves this subcommittee. Collective communication operations can be constructed out of the point-to-point primitives, but are desired because they can be implemented efficiently they are convenient for programmers. The committee then went through the set of collective communication primitives that had been proposed by Al Geist during the email discussions. Broadcast: info = MPI_BCAST(buf,bytes,tag,gid,root) On return, contents of buf for root is in buf for all processes. Al Geist pointed out that the group id here is explicit. Root has to be a member of the group. It was at this point that the committee decided that it would use the word "tag" for message type from now on to distinguish it from "type", which will now always mean type of data. Marc Snir pointed out that for consistency with point-to-point operations, there should be both local termination (the operation returns when the local process has done its part) and global termination (the operation returns when all processes have finished their participation) versions. There followed a discussion of the fact that the point-to-point committee seems to be adopting many different versions of send and receive, and that total compatibility will require many different versions of broadcast. There was a discussion of the reason for the tag parameter in the call. It is needed to disentangle multiple broadcasts occurring at approximately the same time. Paul Pierce described how the system can do this by generating sequence numbers. Others argued that the tag was useful for the programmer in any case, particularly for verifying program correctness. Marc Snir argued that there is a problem (because of the intuition that bcast provides barrier) 1 2 3 send(3) bcast rec(don't care) bcast send(3) bcast rec(don't care) Note that 3 may receive from 2 before 1, i.e. no barrier. Al Geist replied that we need barrier, but broadcast is NOT a barrier. James Cownie initiated a general discussion of whether broadcasts could be received by general receives. This would make it simpler to inherit some of the point-to-point semantics. Al Geist said that broadcast should be consistent with the other collective operations, all of which are symmetric. Paul Pierce suggested we specify collective communication routines in terms of model P-P implementation. This has consequences in terms of what options can be supported. Marc Snir pointed out that one can't actually specify collective communication in terms of point-to-point operations because they need dynamically-allocated additional space. It was decided to postpone a straw vote on whether all processes participating in a broadcast should do "broadcast" or only the root should "broadcast" and the others should "receive" because of concern about remaining issues, e.g. different varieties of recieves. The discussion of "error code" was deferred until the issue is settled in the Point-to-point communication subcommittee. MPI_GATHER: (see mail archives for details) It was proposed to have a version in which each participant contributes a different amount of information (a general "concatenate" function). Issues raised: How handle the situation where the number of bytes on each processor is different. How specify the type of data? For example one needs to know the size of the data type for various purposes, e.g. when doing recursive bisection. MPI_GLOBAL_OP: (see archives for definition) This does not include the data types. There was a discussion of how the forwarding processors know where to break buffers if the data type is not specified. Paul Pierce suggested that we should separate the case of user-defined combining operations from the system ones, which could be optimized. Robert Harrison suggested that the buffer be specified as (#items, length) at least for the user-defined operations. (Tag would be retained) Someone noted that "bytes" would be different on each processor in the heterogeneous case. Back to GATHER. Many agreed that the interface should be changed, but no proposal was offered. Straw vote on having separate general concatenation, to go along with the gather operation: yes: 18 no: 0 MPI_SYNCH There was general agreement that "BARRIER" would be a better name. James Cownie suggested that a tag argument would be helpful for debugging. There was also some discussion of failure of such a barrier, e.g. because some node fails. It was agreed that this was not a problem peculiar to this particular function. One individual nonetheless argued strongly for some kind of timeout for the barrier. Groups ------ gid = MPI_MKGRP (list of processes) There was much discussion of the format of the process list. As defined MKGRP defines a group as a subset of a pre-existing group. One alternative would be to allow creating a group consisting of processes from a number of other groups. (NB Identification of processes is unspecified. This is a task for the Point-to-point Communication Subcommittee.) MKGRP provides an implicit barrier among the processes joining the group. There are a number of problems about making sure that gid is uniform and known across the system. This is an efficiency issue. Should it be possible to SEND to a (gid,rank) pair? Marc argued that one should do Point-to-point communication only within a group, not between groups. Note that groups are constant - cannot add or delete members from a group. Also group creation is a barrier for the processes that are part of the group. This raises the question of how the processes joining the group know that they are joining. What is utility of groups? Certainly at present the only commonly used group is ALL. MPI_FREEGRP(gid) MPI_GRPSIZE MPI_MYRANK There was a general discussion of how group id's would be generated. Also a discussion of the mapping information: How to map back from my_rank and gid to rank in ALL? (In order to actually do a SEND.) ----- At this point the group broke for dinner ----- The continuation after dinner was an informal general discussion. There were some general question about experience from Al Geist to Paul Pierce. Adam Greenberg expressed interest in discussing channels. Channels are seen as an early binding, (Curryification) of various of the SEND/RECV functions which offer a number of gains in efficiency. There was a discussion of Fortran language bindings (F77, F90, HPF) of MPI. It was agreed by those knowledgable in the area that there are no special issues in regard to HPF. Steve Wheat discussed the Sandia implementation of channels on the Ncube. Sounds very similar to iWarp channels except that they are dynamic in creation. Jim Cownie noted that global-ops are going to result in non-determinism in numeric routines. Jim also elaborated on Meiko's BAD experience with ready_receive function - lots of user problems. Commonly user's try it on small problems and it works and speeds up. But then on large problems things erratically break and the user bitches. Paul Pierce noted that this is essentially Intel's force type and the Intel experience has not been so bad. In particular it is harder to use and does not generally work easily on small problems. Cownie: In general what to do when a ready_receive fails? No reasonable way to raise error. Response: Use a signal. Cownie: GAACK! This is implementation and not viable on all systems. John Kapenga listed six collective communication issues that he considers particularly important. [Missed the list] Other desirable collective communication features that were mentioned: global-exchange; all-to-all communication. What are criteria for including? Proposal: Difficulty of implementation; frequency of use; efficiency gain John Kapenga asked about 2-D and 3-D mesh operations - e.g. shifts? Adam Greenberg said this should be left to compilers. John: No Way! Adam argued that the compiler can recognize opportunity to avoid memory copies. Unless that same facility is available to user the compiler can do much better. The group adjourned at 10:10 p.m. --------------------------------------------------------------------------- Topologies Subcommittee --------------------------------------------------------------------------- The Topologies Subcommittee was called to order by Rolf Hempel at 4:00 on Wednesday. It lasted until dinner. --------------------------------------------------------------------------- Other Subcommittees --------------------------------------------------------------------------- The other subcommittees (Introduction, Formal Semantics, Environmental Enquiry, Language Binding) met informally after dinner on Wednesday. --------------------------------------------------------------------------- Meeting of the Whole Committee --------------------------------------------------------------------------- Thursday, January 7, 4:30 The Agenda for the rest of the meeting was presented: Introduction subgroup report Collective-communications subgroup report Process Topology subgroup report Environmental Inquiry subgroup report Formal Language subgroup report Language Binding subgroup report Profiling (Jim Cownie) Dates for future meetings Report of the Introduction Subcommittee: ------ -- --- ------------ ------------ Jack Dongarra presented the results of the subcommittee meeting that took place Wednesday night. This is essentially the draft that has been available from netlib for the last six weeks. There was some on-the-fly editing by the group at large. The goal of the Message Passing Interface simply stated is to develop a *de facto* standard for writing message-passing programs. As such the interface should establishing a practical, portable, efficient, and flexible standard for message passing. Goals ----- Design an application programming interface (not necessarily for compilers or a system implementation library). Allow efficient communication: Avoid memory to memory copying and allow overlap of computation and communication and offload to communication coprocessor, where available. Allow (but not mandate) extensions for use in heterogeneous environment. Allow convenient C, Fortran 77, Fortran 90, and C++ bindings for interface. Provide a reliable communication interface. User need not cope with communication failures. Such failures are dealt by the underlying communication subsystem. Define an interface that is not too different from current practice, such as PVM, Express, P4, etc. Define an interface that can be quickly implemented on many vendor's platforms, with no significant changes in the underlying communication and system software. The interface should not contain more functions than are really necessary. (Based on the latest count of send/receive variants, this drew a large laugh from the crowd.) Focus on a proposal that can be agreed upon in 6 months. Added: Semantics of the MPI should be programming language independent. Who Should Use This Standard? --- ------ --- ---- --------- This standard is intended for use by all those who want to write portable message-passing programs in Fortran 77 and/or C. This includes individual application programmers, developers of software designed to run on parallel machines, and creators of higher-level programming languages, environments, and tools. In order to be attractive to this wide audience, the standard must provide a simple, easy-to-use interface for the basic user while not semantically precluding the high-performance message-passing operations available on advanced machines. What Platforms Are Targets For Implementation? ---- --------- --- ------- --- --------------- The attractiveness of the message-passing paradigm at least partially stems from its wide portability. Programs expressed this way can run on distributed-memory multiprocessors, networks of workstations, and combinations of all of these. In addition, shared-memory implementations are possible. The paradigm will not be made obsolete by architectures combining the shared- and distributed-memory views, or by increases in network speeds. It thus should be both possible and useful to implement this standard on a great variety of machines, including those ``machines" consisting of collections of other machines, parallel or not, connected by a communication network. It was agreed that explicit remarks that MPI is intended to be usable with multithreaded processes and with MIMD (not just SPMD) programs should be added somewhere. What Is Included In The Standard? ---- -- -------- -- --- --------- The standard includes: Point-to-point communication in a variety of modes, including modes that allow fast communication and heterogeneous communication Collective operations Process groups Communication contexts A simple way to create processes for the SPMD model Bindings for both Fortran and C Support for Process Topologies In addition A model implementation and A formal specification. will be provided. It was proposed that explanation and rationale for the standard would also be provided as would sample programs and a validation suite. This is getting very ambitious. Jim Cownie also wants wrappers available for use by, for example, profiling. The suggestion is to provide "name shift", e.g. __MPI_SEND, etc. so the profiler can have MPI_SEND call __MPI_SEND after doing whatever is useful for profiling. What Is Not Included In The Standard? ---- -- --- -------- -- --- --------- The standard does not specify: Explicit shared-memory operations Operations that require more operating system support than is currently standard; for example, interrupt-driven receives, remote execution, or active messages Program construction tools Debugging facilities Tracing facilities Features that are not included can always be offered as extensions by specific implementations. Report of the Collective Communication Subcommittee: ------ -- --- ---------- ------------- ------------ Al Geist summarized the meeting that took place Wednesday afternoon (described above). Global functions beyond those discussed by the subcommittee, such as all2all or total_exchange, await written proposals. The (whole) committee added that Fortran 90 and HPF would be a good place to look for more combining functions (other than max, min, sum, etc.) It was agreed that a way to supply user-supplied functions would be useful. Issues mentioned include: What is a group? How are groups formed? Are group elements addressable, if so how? Are groups ordered (e.g. for prefix/suffix operations)? Group always an ordered subset of the ALL group? Partitioning? Connection with virtual topologies? This will be discussed when topology group reports. Friday, January 8 ------ ------- - Jack Dongarra called the meeting to order at 9:00. Report of the Process Topologies Subcommittee: ------ -- --- ------- ---------- ------------ Rolf Hempel reported on the meeting held Wednesday afternoon: Motivation: Applications have structures of processes Most natural way to address processes Processor topology is valuable to user Creation of subgroups is a natural way to implement topologies Draft proposal for MPI functions in support of process topologies (by Rolf Hempel) is in the handout bundle. The subcommittee made some changes to the draft. What functions should MPI contain? specification of logical process structure lookup functions for process id's clean interface to other parts of MPI (process groups) What should it not contain? any reference to particular hardware architectures algorithms for mapping of processes to processors If it does this, the user program will be portable, but will contain full information for processes mapping at the logical level. Claim: The use of process topologies is not an obstruction to quick implementation of MPI, since the first implementation can make random assignments. A process topology is assigned to a process group. Copying groups can be used to overlay different topologies on the same processes. All processes in a group call the topology definition function. Inquiry functions provide the translation of logical process location to process id. Supported Topologies: General graph structure: For each process, define the complete set of neighbors for each node. In principle this is sufficient as it covers all topologies. But it is not scalable as all processes have knowledge of all others. we should investigate a scalable version. However, important special cases should be treated explicitly, because regular structures can be specified in a scalable way easier to implement the mapping they cover a large number of applications. A special case: Cartesian structures grids/tori hypercube is a special case Support for creation of subgroups for regular structures will be useful. Special treatment for trees? deferred User-defined topology definition functions? deferred It will be necessary for the inquiry functions to provide information on the hardware topology, so that a user can provide his own mapping function. Marc Snir: We need to consider consistency of mapping alignments, for example an octtree for image processing with a grid structure. Al Geist: What is connection between group and topology. Recall that a group is a linear *ordered* array which is a kind of topology. General discussion of copying topologies and groups Proposal is to have at most one topology per group so can use group id as name for topology. This is reason that there must be a group copy. David Walker: We need closer coordination between the collective communication subcommittee and the topology subcommittee, since groups are central to both. Report of the Environmental Enquiry Subcommittee: ------ -- --- ------------- ------- ------------ Bill Gropp reported that the Environmental Enquiry subcommittee needs to wait and get a better picture of what MPI will contain. Jon Flower again asked for cpu_time. This was discussed, and we were reminded that these were more-or-less rejected at the Minneapolis meeting as not being part of MPI. Standardization should come from POSIX. Marc Snir: Part of the subcommittee's job should be to decide *what* can be enquired about as well as how it will be done. There was general discussion about inquiring about both MPI parameters and implementation parameters. Also if parameter *setting* as well as enquiry should be supported. (Buffer pool sizes, for example). Jon Flower also asked about system hints. He suggested it should be possible to tell the system about implementation specific tuning in a system independent way. Report of the Formal Specification Subcommittee: ------ -- --- ------ ------------- ------------ Rusty Lusk reported the committee was without its chairman, Steven Zenith, but that it viewed its mission as to try to formalize what the other subcommittees decide on. It will probably use CSP, for lack of experience with any other formal specification language. Bob Knighten suggested that the subcommittee look into LIS (Language Independent Specification) that POSIX defined in order to separate semantics from language bindings. Report on MPI -1 (minus one) ------ -- ------ ----------- James Cownie presented an MPI anti-specification. Ya hadda be there, but in case you weren't or just want to be reminded, here is a transcription of Jim's slides. MPI -1 (Jim Cownie) In the spirit of LPF (Low Performance Fortran) * Bindings ONLY for Mathematica Occam ML * No function take arguments or returns result * Point to Pointless communication * 1024 different sends NO receives * Full support for 0 dimensional topologies * User data in a message limited to 1 byte (of 6 data types) BUT 1 KByte of TAG, CONTEXT * Informal semantics - Formal Syntax * All groups are contexts * All contexts are groups * Non blocking wait * Non blocking barrier * All user programs are unsafe & erroneous, they therefore do all their work in the exception handler. --------------------------------------------------------------------------- A Profile/Instrumentation subgroup was formed with Jim Cownie as chairman. Steve Otto, as general editor, will contact subgroup chairmen to begin discussion of editing concerns. Discussion of meeting format. The following was proposed as a format for subsequent meetings, based on the experience with this meeting. Wed. afternoon: point-to-point Wed. night: all subcommittees other than pt-to-pt and collective comm. Thurs. morning: collective communication Thurs. afternoon: subcommittee reports Fri. afternoon: subcommittee reports Meeting Dates: It was decided to moved the next two meetings up a week from when they were tentatively scheduled. The next meeting will be Feb 17-19. The next one after that will be Mar 31-Apr 2 The currently-scheduled May 19-21 and June 30-July 2 meetings may also be moved up as well. Note that July 2 will be a holiday in the United States. From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 01:42:10 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07543; Wed, 3 Feb 93 01:42:10 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03537; Wed, 3 Feb 93 01:41:08 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 01:41:06 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03528; Wed, 3 Feb 93 01:41:05 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA28816; Wed, 3 Feb 93 00:41:03 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA23933; Wed, 3 Feb 93 00:41:00 CST Message-Id: <9302030641.AA23933@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: A suggestion for a multi-level MPI Date: Wed, 03 Feb 93 00:40:59 CST From: Rusty Lusk At the last meeting, it became apparent that many people were uncomfortable with the large number of routines (1024 sends?) that we seemed to be going towards. There are good reasons for the standard not to be incomprehensibly complex. There are good reasons also for it not to be crippled by lack of functionality. Are we stuck? There are two ways to deal with this problem. This note is a proposal that we adopt both of them. The reason the problem can be solved is that the sets of options that we have been considering (blocking or not, contiguous buffer or not, heterogeneous machines or not, etc.) are orthogonal, and one of a large number of possible send routines can thus be specified simply and without extra parameters. In addition, certain subsets of the set of all operations can be identified and packaged for ease of use. Here is a suggestion for how to organize the routines and end up with a powerful, flexible, yet easy-to-use library of point-to-point functions. (We assume that these will all be renamed to have an MPI prefix, to avoid name-space pollution) Level 4 (very restrictive, very simple): send(dest,tag,buf,len) recv(dest,tag,buf,len) Both of these block until the operation is locally complete. The buffer is a contiguous sequence of bytes. No special protocols or translation for heterogeneous machines. This level is enough to express lots of parallel algorithms. The main reason we need more is for efficiency and control. We now add these in steps. Level 3 (still simple, but allows for overlap of computation and communication and better buffer management, and allows heterogeneous machines to communicate.) bsend(tag,dest,bufadd,buflen) nsend(tag,dest,bufadd,buflen) bsendh(tag,dest,bufadd,datatype,numitems) nsendh(tag,dest,bufadd,datatype,numitems) brecv(tag,dest,bufadd,buflen) nrecv(tag,dest,bufadd,buflen) brecvh(tag,dest,bufadd,datatype,numitems) nrecvh(tag,dest,bufadd,datatype,numitems) Here the n or b selects block or nonblocking operations, h specifies translation for heterogeneous machines. The restrictions (which allow the parameter lists to be short and the number of routines to be small) are: no noncontiguous buffers, data in message all the same type, no globally synchronized, CSP-like send-receive, and no special protocol (like the ready-receiver protocol discussed at the last meeting). In order to emphasize the orthogonality, we might organize these routine names in a diagram like this: [n][send][ ] [b][recv][h] That is, one makes a choice in each column. Level 2 (access to almost all MPI functionality via a large number of orthogonally named routines): [c][n][send][ ][ ] [s][b][recv][h][rr] [g][s] the first column designates the buffer type (contiguous, strided, general gather/scatter), the second is the termination type (nonblocking, blocking, synchronized), then send or receive, "h" or not for heterogeneous communication, "rr" or not for "ready-receiver" protocol. This is a lot of routines (3 x 3 x 2 x 2 x 2 = 72) but because of orthogonality it is easy to understand. The only restriction here is that there are no "channels": the initialization of a send operation is combined with initiating it. Level 1 (full MPI functionality, in a small, flexible set of routines) handle = init_send(tag,dest) mod_send(handle,option,value) do_send(handle) free_send(handle) along with the corresponding receive routines. The idea is that the "init" routines will set up a usable set of defaults for all the options, specifying only minimal parameters, and then the "mod" routine can be repeatedly called to specify all the options with regard to buffer type, termination type, etc. The point is that at this level we can offer not only all possible options, with a small number of routines, and include the "setup" routines proposed by Marc for increased efficiency, but we also allow room for growth in the standard through the addition of more values for "option" in the "mod" calls. So here's how it might look, this time from the bottom up: Level 1: full functionality, small number of routines, channel setup Level 2: sequences of Level 1 calls of the form init... mod... mod... mod... do... are given specific names in an organized way. This promotes ease of use and also efficiency (fewer calls). Level 3: several of the Level 2 routines are renamed and promoted to this level to provide a useful subset of MPI functionality in a small number of routines, each having the minimal number of parameters. Level 4: ultra-simple interface All four levels would be part of the standard and it would be possible to mix levels in the same program. We have not considered carefully yet how this interacts with groups and contexts, but similar ideas might be useful there, where there are similar problems in providing both functionality and simplicity. Comments? Rusty Lusk and Bill Gropp From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 06:16:01 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27255; Wed, 3 Feb 93 06:16:01 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23277; Wed, 3 Feb 93 06:14:47 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 06:14:46 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from gatekeeper.oracle.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23269; Wed, 3 Feb 93 06:14:40 -0500 Received: from wrpyr.us.oracle.com by gatekeeper.oracle.com (5.59.11/37.7) id AA21815; Wed, 3 Feb 93 03:14:37 PST Received: by wrpyr.us.oracle.com (5.59.11/37.7) id AA15597; Wed, 3 Feb 93 03:17:14 PST Message-Id: <9302031117.AA15597@wrpyr.us.oracle.com> Date: Wed, 3 Feb 93 03:17:14 PST From: "Chuck Simmons " To: mpi-comm@cs.utk.edu Subject: Re: A suggestion for a multi-level MPI Reply-To: csimmons@us.oracle.com Original-To: WRPYR:mpi-comm@cs.utk.edu In-Reply-To: WRPYR:owner-mpi-comm@CS.UTK.EDU's message of 02-03-93 00:40 Rusty, Bill -- I do view the interface as being layered, but I would organize the layers differently. And I have a couple of arguments that I haven't tried out yet against some of the send/recv routines. First, it sounds to me like you are suggesting that the 72 routines be implemented as 72 simple routines that each make a small number of subroutine calls to very simple routines. And then the do_send routine would do a lot. I guess the do_send routine on a typical machine might look like: /* convert the data into a vector of buffers */ if (options & MPI_OPT_ONEBUF) { iovlen = 1; iovp = bufp; } else if (options & MPI_OPT_STRIDED) { /* malloc local buffer, gather data, set up iovlen and iovp */ } /* translate the data */ if (options & MPI_OPT_HETER) { /* malloc new buffer, translate data into new buffer */ } /* send the data non-blocking and unreliable */ if (options & MPI_OPT_RR) { sendrr (...); } else send (...); /* block if so requested */ if (options & MPI_OPT_BLOCK) { block (...); } My view of the layering is: basic functionality: non-blocking vector send non-blocking vector receive easy-to-use basic functionality: non-blocking single buffer send non-blocking single buffer receive reliable: synchronous vector send synchronous vector receive easy and reliable: synchronous single buffer send synchronous single buffer receive In the above, each of the later listed routines can be implemented on top of an earlier listed routine. I would then examine each of the remaining aspects of the functionality: heterogeneity, ready-receiver, strided messages, and blocking non-synchronous messages. For heterogeneity, I would argue that it is wrong to put the datatype translation filter in the communications channel. In your model, it is relatively easy to do things like: bsendh (dest, tag, buf, FLOAT, 1); or even: bsendh (dest, tag, buf, ARRAY|FLOAT, nitems); but what about bsendh (dest, tag, buf, rpc_t, 1); where 'rpc_t' is some user defined data structure? Thus, this model lacks generality. Only certain simple messages can be heterogeneously sent efficiently through the interface. A completely different paradigm must be used if you want to send a structure instead of an array. Further, consider how the "bsendh (dest, tag, buf, ARRAY|FLOAT, nitems);" routine is going to be implemented on a typical machine: buflen = sizeof (net_float) * nitems; sendbuf = (net_float *) malloc (buflen); if (!sendbuf) return ENOMEM; for (i = 0; i < nitems; i++) { sendbuf[i] = float2netfloat (buf[i]); } bsend (dest, tag, (void *)sendbuf, buflen); free (sendbuf); (In the above, we assume that a "net_float" holds the network normalized format of a floating point number.) It would be as efficient to implement the translation routines completely separate from the communications routines. But, wait, you say, our machine is not typical and we can implement this as: bsend (dest, tag, &header, sizeof (header)); for (i = 0; i < nitems; i++) { bsend (dest, tag, float2netfloat (buf[i]), sizeof (net_float)); } where 'bsend' is assumed to be a machine instruction. My argument against this is that the gain in efficiency is sufficiently small that it's not clear its worth the added complexity. In both implementations, the cpu time will be dominated by the data conversion. In the "typical" implementation, there is an additional cost for moving data between two buffers, but that data movement occurs at the speed of local memory access which is likely to be much higher than network bandwidth. Thus, it is much easier for the hardware to DMA the first implementation and run some other process while that DMA is in progress. In the second implementation, the sending process is essentially hogging the cpu for the full time it takes to transmit each piece of the message. Obviously, if there exists a standard for network normal formats of datatypes, and if hardware vendors implement the data translation facilities in hardware, then we could DMA out the whole message heterogeneously. But surely there are other things that we would rather have hardware vendors implementing for us. The above is essentially the same argument that I'm going to use for strided messages. Strided messages are only efficient if you have hardware that can DMA the strided vector, and then its not so efficient that it's obviously worth the effort. For ready-receiver, I'm going to argue that the communications subsystem should always assume the receiver is ready and then handle the case where this isn't true. If the application has set things up so that this is true, that application will run efficiently whether or not the sender specifies that the receiver is ready. If the application isn't being careful, it will run slowly in your model, but in my model, it will usually run quickly since memory is usually availabe in the receiver, and my model will run as slowly as your model when congestion is present at the receiver. Implementing the ready-receiver places strong semantics on the sender and receiver. The sender has to know whether or not the receiver was implemented in a fashion that allows the sender to assume that the receiver is ready. Finally, let's consider synchronous messages versus blocking messages. For most applications, you want reliability. Implementing blocking non-synchronous messages doesn't help you implement this. You can't reuse the buffer being sent until you know that it has been received at the other side. So you might as well either send the message synchronously, or send it non-blocking and go off and do something else while you're waiting to see if you'll need to retransmit the buffer. [Hmmm... I did mention, didn't I, that I was still trying out a couple of these arguments...] This reduces the number of interface routines to 8. I could probably be convinced that adding 4 more interface routines to support strided messages doesn't overly complicate the interface. -- Chuck *** We use Oracle*Mail running on Oracle V6.2 and an nCUBE2 supercomputer. *** ---- Included Message ---- Received: 02-02-93 22:48 Sent: 02-03-93 00:40 From: WRPYR:owner-mpi-comm@CS.UTK.EDU To: mpi-comm@cs.utk.edu Subject: A suggestion for a multi-level MPI Reply-To: WRPYR:owner-mpi-comm@CS.UTK.EDU X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 01:41:06 EST Errors-To: owner-mpi-comm@CS.UTK.EDU At the last meeting, it became apparent that many people were uncomfortable with the large number of routines (1024 sends?) that we seemed to be going towards. There are good reasons for the standard not to be incomprehensibly complex. There are good reasons also for it not to be crippled by lack of functionality. Are we stuck? There are two ways to deal with this problem. This note is a proposal that we adopt both of them. The reason the problem can be solved is that the sets of options that we have been considering (blocking or not, contiguous buffer or not, heterogeneous machines or not, etc.) are orthogonal, and one of a large number of possible send routines can thus be specified simply and without extra parameters. In addition, certain subsets of the set of all operations can be identified and packaged for ease of use. Here is a suggestion for how to organize the routines and end up with a powerful, flexible, yet easy-to-use library of point-to-point functions. (We assume that these will all be renamed to have an MPI prefix, to avoid name-space pollution) Level 4 (very restrictive, very simple): send(dest,tag,buf,len) recv(dest,tag,buf,len) Both of these block until the operation is locally complete. The buffer is a contiguous sequence of bytes. No special protocols or translation for heterogeneous machines. This level is enough to express lots of parallel algorithms. The main reason we need more is for efficiency and control. We now add these in steps. Level 3 (still simple, but allows for overlap of computation and communication and better buffer management, and allows heterogeneous machines to communicate.) bsend(tag,dest,bufadd,buflen) nsend(tag,dest,bufadd,buflen) bsendh(tag,dest,bufadd,datatype,numitems) nsendh(tag,dest,bufadd,datatype,numitems) brecv(tag,dest,bufadd,buflen) nrecv(tag,dest,bufadd,buflen) brecvh(tag,dest,bufadd,datatype,numitems) nrecvh(tag,dest,bufadd,datatype,numitems) Here the n or b selects block or nonblocking operations, h specifies translation for heterogeneous machines. The restrictions (which allow the parameter lists to be short and the number of routines to be small) are: no noncontiguous buffers, data in message all the same type, no globally synchronized, CSP-like send-receive, and no special protocol (like the ready-receiver protocol discussed at the last meeting). In order to emphasize the orthogonality, we might organize these routine names in a diagram like this: [n][send][ ] [b][recv][h] That is, one makes a choice in each column. Level 2 (access to almost all MPI functionality via a large number of orthogonally named routines): [c][n][send][ ][ ] [s][b][recv][h][rr] [g][s] the first column designates the buffer type (contiguous, strided, general gather/scatter), the second is the termination type (nonblocking, blocking, synchronized), then send or receive, "h" or not for heterogeneous communication, "rr" or not for "ready-receiver" protocol. This is a lot of routines (3 x 3 x 2 x 2 x 2 = 72) but because of orthogonality it is easy to understand. The only restriction here is that there are no "channels": the initialization of a send operation is combined with initiating it. Level 1 (full MPI functionality, in a small, flexible set of routines) handle = init_send(tag,dest) mod_send(handle,option,value) do_send(handle) free_send(handle) along with the corresponding receive routines. The idea is that the "init" routines will set up a usable set of defaults for all the options, specifying only minimal parameters, and then the "mod" routine can be repeatedly called to specify all the options with regard to buffer type, termination type, etc. The point is that at this level we can offer not only all possible options, with a small number of routines, and include the "setup" routines proposed by Marc for increased efficiency, but we also allow room for growth in the standard through the addition of more values for "option" in the "mod" calls. So here's how it might look, this time from the bottom up: Level 1: full functionality, small number of routines, channel setup Level 2: sequences of Level 1 calls of the form init... mod... mod... mod... do... are given specific names in an organized way. This promotes ease of use and also efficiency (fewer calls). Level 3: several of the Level 2 routines are renamed and promoted to this level to provide a useful subset of MPI functionality in a small number of routines, each having the minimal number of parameters. Level 4: ultra-simple interface All four levels would be part of the standard and it would be possible to mix levels in the same program. We have not considered carefully yet how this interacts with groups and contexts, but similar ideas might be useful there, where there are similar problems in providing both functionality and simplicity. Comments? Rusty Lusk and Bill Gropp From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 06:35:54 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01865; Wed, 3 Feb 93 06:35:54 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23802; Wed, 3 Feb 93 06:35:00 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 06:34:59 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23794; Wed, 3 Feb 93 06:34:56 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA13771 (5.65c/IDA-1.4.4 for ); Wed, 3 Feb 1993 06:34:53 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA26850; Wed, 3 Feb 93 11:34:48 GMT Date: Wed, 3 Feb 93 11:34:48 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302031134.AA26850@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA03425; Wed, 3 Feb 93 11:33:11 GMT To: lusk@antares.mcs.anl.gov Cc: mpi-comm@cs.utk.edu In-Reply-To: Rusty Lusk's message of Wed, 03 Feb 93 00:40:59 CST <9302030641.AA23933@donner.mcs.anl.gov> Subject: A suggestion for a multi-level MPI Content-Length: 581 The fundamental approach seems fine to me (though Chuck makes some good points...) However there's at least one point to note :- 1) The Fortran binding issue suggests that in the heterogeneous cases there should be a routine for each data type, rather than having the data type as an argument. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 07:27:22 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11163; Wed, 3 Feb 93 07:27:22 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25571; Wed, 3 Feb 93 07:26:12 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 07:26:11 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ifi.uio.no by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25557; Wed, 3 Feb 93 07:26:08 -0500 Received: from byflak.ifi.uio.no by ifi.uio.no with SMTP id ; Wed, 3 Feb 1993 13:26:05 +0100 From: "\ystein Gran Larsen" Received: by byflak.ifi.uio.no ; Wed, 3 Feb 93 13:26:04 +0100 Date: Wed, 3 Feb 93 13:26:04 +0100 Message-Id: <9302031226.AA26562@byflak.ifi.uio.no> To: mpi-comm@cs.utk.edu Subject: data types and heterogeneity Hi, I have read the minutes from the MPI meeting, January 6-8 1993. The section on message data types mentiones the use of XDR as a protocol for converting types between languages and machines. The standardization of Scalable Coherent Interface (SCI) has lead to a proposed standard for "Shared-Data Formats Optimized for Scalable Coherent Interface Processors", P1596.5. [Working group P1596.5 is chaired by David V. James (dvj@apple.com)] This proposed standard may be of relevance to the MPI-initiative, when it comes to supporting heterogeneous machines. -{\O}ystein From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 11:56:47 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17274; Wed, 3 Feb 93 11:56:47 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07365; Wed, 3 Feb 93 11:54:56 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 11:54:55 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07357; Wed, 3 Feb 93 11:54:53 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA01169; Wed, 3 Feb 93 10:54:51 CST Date: Wed, 3 Feb 93 10:54:51 CST From: Tony Skjellum Message-Id: <9302031654.AA01169@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, gran@ifi.uio.no Subject: Re: data types and heterogeneity XDR as currently implemented causes one function call per datum. This feature must be changed (loops moved inside fundamental converter routines) to provide for better performance (less bandwidth impact of converting data types). I am eager to see the report mentioned by Oystein. - Tony From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 13:33:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19846; Wed, 3 Feb 93 13:33:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12323; Wed, 3 Feb 93 13:31:59 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 13:31:58 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from iliamna.cse.ogi.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12315; Wed, 3 Feb 93 13:31:56 -0500 Received: by iliamna.cse.ogi.edu (/\==/\ Smail3.1.25.1 #25.15) id ; Wed, 3 Feb 93 10:31 PST Message-Id: Date: Wed, 3 Feb 93 10:31 PST From: otto@iliamna.cse.ogi.edu (Steve Otto) To: mpi-comm@cs.utk.edu Subject: Mix + Match, NOT At first look, I definitely like the proposal of Lusk and Gropp. It seems to be the right way to set up a hierarchy of the 1024+ routines that we are inventing. One thing that bothers me: at the last meeting it was said by several people that one would be allowed to mix+match different kinds of sends and recvs to each other. That is, I could do a "cnsend()" and have that consumed by a "sbrecvhrr()". I think we are asking for lots of trouble if the standard states that this is allowed. If it is allowed, an implementor has to worry about 72 x 72 = 5184 different types of behavior for pt-pt communication. I don't see a strong reason for allowing mix+match, and on the implementation side it seems to me that there are many reasons for wanting the restriction that a send of flavor 53 (out of a possible 72) should be matched by a recv of the same flavor, 53. It is true that it seems like many flavors could be mixed, but consider the synchronization choices. If we allow mixing, then the implementor of the sbrecv() routine has to insure that "sensible" behavior results no matter what send routine was used. As I understand it, this means the implementation cannot be done in an orthogonal manner. I'm not even sure we could define "sensible" for all 5184 combinations! Comments? Am I missing something? --Steve Otto From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 18:43:47 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29539; Wed, 3 Feb 93 18:43:47 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27997; Wed, 3 Feb 93 18:41:34 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 18:41:33 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27989; Wed, 3 Feb 93 18:41:32 -0500 Received: from id.wmich.edu by cs.wmich.edu (4.1/SMI-4.1) id AA14014; Wed, 3 Feb 93 18:35:11 EST Date: Wed, 3 Feb 93 18:35:11 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9302032335.AA14014@cs.wmich.edu> To: mpi-comm@CS.UTK.EDU Subject: Re: A suggestion for a multi-level MPI James Cownie remarks on needing a seperate send routine for each data type in the Fortran Binding. For both the C and Fortran versions there could be seperate calls setting up the Data Addresses (no Data need be moved) and then a single send could take the value returned by the Data Address Setup Call. No system call required on the Data Setup Call. This might require that the array passed to the Data Setup Call not be modified before the send call is made, to avoid recopying it. This also allows later extensions to other Data Addressing modes, without changing the send/recieve syntax. This reduces the number of calls by almost a factor of three, uncouples the data location from the send/recieve syntax and allows arguement types on all calls to be correct and fixed. It does make a bit more processing (trival) and the need for the user to mamage the Data Setup return value and not modify the address array (two of its bad points). john From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 23:45:54 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02591; Wed, 3 Feb 93 23:45:54 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09607; Wed, 3 Feb 93 23:43:48 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 23:43:47 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09599; Wed, 3 Feb 93 23:43:45 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA00850; Wed, 3 Feb 93 22:43:44 CST Date: Wed, 3 Feb 93 22:43:44 CST From: Tony Skjellum Message-Id: <9302040443.AA00850@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu Subject: Anonymous ftp site aurora.cs.msstate.edu A paper of relevance to the MPI project, describing "Zipcode 1.0" is available by anonymous FTP, in the pub/reports subdirectory: zipcode_sep92.ps.Z The site is aurora.cs.msstate.edu. - Tony Skjellum From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 00:02:32 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02797; Thu, 4 Feb 93 00:02:32 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10374; Thu, 4 Feb 93 00:01:41 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 00:01:39 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10364; Thu, 4 Feb 93 00:01:38 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA08517; Wed, 3 Feb 93 23:01:36 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA26443; Wed, 3 Feb 93 23:01:34 CST Message-Id: <9302040501.AA26443@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: Multiple levels Date: Wed, 03 Feb 93 23:01:33 CST From: Rusty Lusk Chuck -- Thanks for your comments. They are of two kinds. The first kind are about the suggested multiple-level organization of the library. There I think I didn't explain some things well enough, and will try to do better here. The other kind are about what should be in or out of the library altogether, and there I can only try to summarize what people talked about in Dallas; the things we incorporated are just the things that the straw votes indicated people wanted in the standard. | | I do view the interface as being layered, but I would organize the layers | differently. And I have a couple of arguments that I haven't tried out | yet against some of the send/recv routines. | | First, it sounds to me like you are suggesting that the 72 routines | be implemented as 72 simple routines that each make a small number of | subroutine calls to very simple routines. This is not at all what we intended to suggest. We used the word "levels" rather than "layers". The upper levels could be *defined* in terms of the lower ones (That's what makes them upper-level) but the intention is that they would *not* be implemented as calls to the lower-level routines. In fact what we called level 2 (large number of routines) exists partly to avoid the multiple subroutine calls necessary in level 1 (where there are a small number of routines.) In fact, level 2 routines don't need to use the data structures implied by the get_handle-mod_handle-use_handle-free_handle at all. They just have to do what they say they do, as efficiently as possible. | | For heterogeneity, I would argue that it is wrong to put the datatype | translation filter in the communications channel. | Our discussions seemed to indicate that people want it there. The popularity of systems like PVM, p4, Express, and others that handle this for the user attests to this. I think the reason is portability; people like being able to move code unchanged from a heterogeneous network to a multicomputer and back. The system can (relatively easily) handle all the translations, including figuring out whether it is necessary at all or not. Many users hardly even know about the non-portability of data formats; they think (in my opinion rightly) that the communication system should handle that if they want it to. | In your model, it | is relatively easy to do things like: | | bsendh (dest, tag, buf, FLOAT, 1); | or even: | bsendh (dest, tag, buf, ARRAY|FLOAT, nitems); | but what about | bsendh (dest, tag, buf, rpc_t, 1); | where 'rpc_t' is some user defined data structure? Thus, this model | lacks generality. Not quite true, although not entirely false. In the "draft implementation" of mpi that you can get from here, we use an iovec-like structure which is a vector of (addr,datatype,nitems) triples to describe a scattered, mixed type buffer. An arbitrary structure can be sent this way, provided the user builds the structure descriptor (It's too bad compilers don't do this for us.) | | It would be as efficient to implement the translation routines completely | separate from the communications routines. | .... | | The above is essentially the same argument that I'm going to use for | strided messages. Strided messages are only efficient if you have | hardware that can DMA the strided vector, and then its not so efficient | that it's obviously worth the effort. Machines do have this kind of hardware, and we can expect to see more of it. One thing that I will continue to argue in favor of is giving the user access to as much of the speed the hardware can deliver as possible. | | For ready-receiver, I'm going to argue that the communications subsystem | should always assume the receiver is ready and then handle the case where | this isn't true. | Some existing machines can perform *significantly* faster if the user can relieve their system of some buffer-management chores. If we don't give the power users access to this efficiency, they simply won't use MPI. And it will come to pass that everyone will agree that to get real efficiency, you have to use the vendor's unique routines. Or you have to use "MPI with the XXX extension package". Portability will be destroyed. This is a way the MPI effort could fail. | | Implementing the ready-receiver places strong semantics on the sender | and receiver. The sender has to know whether or not the receiver was | implemented in a fashion that allows the sender to assume that the receiver | is ready. | Agreed. The naive user can use what we call the Level 3 or Level 4 routines, and never read the chapter in the manual that describes "ready-receiver" semantics and requirements. The power user can use them to write *portable* fast code. | | Finally, let's consider synchronous messages versus blocking messages. For | most applications, you want reliability. Implementing blocking | non-synchronous messages doesn't help you implement this. You can't reuse | the buffer being sent until you know that it has been received at the other | side. So you might as well either send the message synchronously, or send | it non-blocking and go off and do something else while you're waiting to see | if you'll need to retransmit the buffer. | Most people want to compute while the message is in transit, and even while the buffer is being emptied. They don't want this capability taken from them. | | This reduces the number of interface routines to 8. I could probably | be convinced that adding 4 more interface routines to support strided | messages doesn't overly complicate the interface. | The Level 4 routines are only 2. More routines are needed only for more efficiency and functionality. In the last few weeks I have talked to lots of people about the MPI effort, and they are mainly afraid that we will leave something important out. If we don't supply all the functionality and efficiency that we possibly can, MPI will fail. The problem is to do so without making the package too complicated. Our posting about multiple levels was merely an attempt to demonstrate that it could be done. Regards, Rusty Lusk & Bill Gropp From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 01:30:19 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04177; Thu, 4 Feb 93 01:30:19 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13427; Thu, 4 Feb 93 01:29:10 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 01:29:09 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13419; Thu, 4 Feb 93 01:29:08 -0500 Received: by msr.EPM.ORNL.GOV (5.67/1.34) id AA12505; Thu, 4 Feb 93 01:29:05 -0500 Date: Thu, 4 Feb 93 01:29:05 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9302040629.AA12505@msr.EPM.ORNL.GOV> To: mpi-comm@cs.utk.edu Subject: Re: 5184 bottles of sends on the wall... Steve Otto proposes that the 72 (or 648 if each data type is separate) sends be matched to the same type of recv. While I agree with Steve that we will not be able to define much less test the 5184 combinations if we allow otherwise, There are obvious cases where the user would want to mismatch send/recv For example, she may have to do a blocking send, but can get away with a non-blocking recv. Not all combinations make sense, but we can't restrict the user to no mismatches. Al Geist From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 02:10:22 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05537; Thu, 4 Feb 93 02:10:22 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14590; Thu, 4 Feb 93 02:08:44 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 02:08:43 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14582; Thu, 4 Feb 93 02:08:41 -0500 Received: by msr.EPM.ORNL.GOV (5.67/1.34) id AA12583; Thu, 4 Feb 93 02:08:39 -0500 Date: Thu, 4 Feb 93 02:08:39 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9302040708.AA12583@msr.EPM.ORNL.GOV> To: mpi-comm@cs.utk.edu Subject: Re: Rusty and Bill ride again... Whoa there Rusty and Bill. Let's not play to fast and loose with the facts. >Most people want to compute even while the buffer is being emptied... Actually most people don't want to think or know about how messages are sent they just want it reliable and fast. Only high priests are concerned with the pain of non-blocking send. >If we don't supply all the functionality that we possibly can, >MPI will fail. The problem is to do so without making the package >too complicated. The complication I am seeing in MPI will also make it fail. 72 sends?? Appears to violate several of our goals, like similar to existing packages. small set of routines. >The naive user never has to read the chapter in the manual... I hope the user doesn't have to make room on his shelves for volumes of the MPI manual. Al From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 02:41:49 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05708; Thu, 4 Feb 93 02:41:49 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15558; Thu, 4 Feb 93 02:40:34 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 02:40:32 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from gmdzi.gmd.de by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15544; Thu, 4 Feb 93 02:40:30 -0500 Received: from f1neuman.gmd.de (f1neuman) by gmdzi.gmd.de with SMTP id AA08675 (5.65c/IDA-1.4.4 for ); Thu, 4 Feb 1993 08:39:03 +0100 Received: by f1neuman.gmd.de id AA14246; Thu, 4 Feb 1993 08:38:42 +0100 Date: Thu, 4 Feb 1993 08:38:42 +0100 From: Rolf.Hempel@gmd.de Message-Id: <9302040738.AA14246@f1neuman.gmd.de> To: mpi-comm@cs.utk.edu Subject: Re: 5184 bottles of sends on the wall... Cc: gmap10@f1neuman.gmd.de I was glad to read Steve's proposal not to allow mismatches between different types of sends and receives. If we don't follow that line, we will most likely cause many problems to the implementors of MPI which we cannot forsee at the moment. Also, the requirement of matching sends and receives could be used to check the correct behaviour of the user program in some debugging mode. Al's counter-example of course makes sense, but if you have to use a blocking send, you can get the same by using the non-blocking version followed directly by the wait-for-completion. I'm looking forward to seeing an example where you really need the mismatch. Rolf From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 03:16:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05912; Thu, 4 Feb 93 03:16:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17215; Thu, 4 Feb 93 03:14:56 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 03:14:53 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17205; Thu, 4 Feb 93 03:14:51 -0500 Received: from id.wmich.edu by cs.wmich.edu (4.1/SMI-4.1) id AA15627; Thu, 4 Feb 93 03:08:28 EST Date: Thu, 4 Feb 93 03:08:28 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9302040808.AA15627@cs.wmich.edu> To: mpi-comm@cs.utk.edu Subject: Re: 5184 bottles of sends on the wall... Rolf asks -- > ... > followed directly by the wait-for-completion. I'm looking forward to > seeing an example where you really need the mismatch. > > Rolf By using the wait_for_completion you never need a blocking send/receive operation. The code looks a bit ugly. Right now we have to have a "mismatch" because we don't have a synchronized send/receive pair ;-). Actually I thought the reason that the synchronized send/recieve pair was there initially was to help insure correctness in matching calls, as well as a possible simpler implementation. john From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 07:03:39 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07047; Thu, 4 Feb 93 07:03:39 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01249; Thu, 4 Feb 93 07:02:33 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 07:02:32 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01241; Thu, 4 Feb 93 07:02:28 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA09794 (5.65c/IDA-1.4.4 for ); Thu, 4 Feb 1993 07:02:22 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA02743; Thu, 4 Feb 93 12:02:17 GMT Date: Thu, 4 Feb 93 12:02:17 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302041202.AA02743@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA03570; Thu, 4 Feb 93 12:00:32 GMT To: otto@iliamna.cse.ogi.edu Cc: mpi-comm@cs.utk.edu In-Reply-To: Steve Otto's message of Wed, 3 Feb 93 10:31 PST Subject: ~ (Mix + Match, NOT) Content-Length: 2480 The only place where I can see a need for matching of send and receive formats is in the heterogeneous case. Here I would be happy to insist that a heterogeneous send be matched to a heterogeneous receive. Since I don't want to enforce in MPI the implemntation decision of whether the data type conversion occurs at sender or receiver I need the data type information in both places. I can see NO reason to restrict any other combinations. Discussion ---------- There are two major areas which cause the many sends 1) layout of data in store. (contiguous, strided, by iovec) 2) synchronisation behaviour (return asap, return when buffer free, return when data received at other end) Data layout ----------- Data layout is an issue which is entirely local. Why does it matter to the receiver how the sender chose to layout her buffer (or the converse) ? This information hiding is one of the things which is an advantage of message passing. The data is necessarily serialised to some degree over the communications medium, so where's the problem ? I believe it is actually HARDER to insist that the buffer sepcifications match at each end than not to do this. (Certainly I have to pass additional information with the message if I'm to check this.) Synchronisation behaviour ------------------------- Once again this seems to me to be largely to do with local issues of buffer management. (Though of course my algorithm may need certain synchronisation behaviour, in which case I must code it using the correct forms of send and recv at both ends.) However I can see no reason to INSIST that the same style of send and receive is used. Indeed there are strong reasons NOT to do so. Consider (for example) a server with multiple clients. I'd certainly like to write it like this :-- Set up lots of non-blocking receives while (running) { wait for a recv service the request get a reply buffer (may need to wait for one) queue a reply requeue the receive buffer } However the clients can simply make requests thus blocking send request blocking receive reply Here is an immediate mismatch in the synchronisations. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 09:44:02 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11973; Thu, 4 Feb 93 09:44:02 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07037; Thu, 4 Feb 93 09:42:34 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 09:42:33 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from [132.175.13.2] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07029; Thu, 4 Feb 93 09:42:30 -0500 Received: from panther.cs.sandia.gov by cs.sandia.gov (4.1/SMI-4.1) id AA12554; Thu, 4 Feb 93 07:42:27 MST Received: by panther.cs.sandia.gov (Smail3.1.28.1 #1) id m0nK7mc-0016ZKC; Thu, 4 Feb 93 07:42 MST Message-Id: Date: Thu, 4 Feb 93 07:42 MST From: srwheat@cs.sandia.gov (Stephen R. Wheat) To: mpi-comm@cs.utk.edu Subject: Re: Rusty and Bill ride again... # Whoa there Rusty and Bill. Let's not play to fast and loose with the facts. # >Most people want to compute even while the buffer is being emptied... # Actually most people don't want to think or know about how messages # are sent they just want it reliable and fast. # Only high priests are concerned with the pain of non-blocking send. Not so; we have those that are not only concerned about being able to compute while messages are in transit, but also wanting to assure that minimal system space is required. That is, they want to manage memory much more tightly than a "buffered" system allows. We have a lot of applications types that find it very hard to fit an application into memory and still allocate sufficient message buffers. By the way, paging is not a viable solution :-). Stephen From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 09:59:26 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12166; Thu, 4 Feb 93 09:59:26 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07696; Thu, 4 Feb 93 09:58:25 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 09:58:23 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from [132.175.13.2] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07688; Thu, 4 Feb 93 09:58:22 -0500 Received: from panther.cs.sandia.gov by cs.sandia.gov (4.1/SMI-4.1) id AA12746; Thu, 4 Feb 93 07:58:18 MST Received: by panther.cs.sandia.gov (Smail3.1.28.1 #1) id m0nK81y-0016ZKC; Thu, 4 Feb 93 07:58 MST Message-Id: Date: Thu, 4 Feb 93 07:58 MST From: srwheat@cs.sandia.gov (Stephen R. Wheat) To: john@cs.wmich.edu, mpi-comm@cs.utk.edu Subject: Re: 5184 bottles of sends on the wall... as an implementor of blocked and non-blocked messaging systems, I have not experienced any "system" problems associated with matching mixed send/receives. While worrying about the "potential" implementation difficulties, I fear that we make use of the system so difficult that it won't be used (so would that make it easy to use?). Furthermore, as for the theme that seems to prevail concerning features that only "wizards" would use, if we do not provide wizardly type features, then the wizards will not use MPI. Then to whom will the "novice" types refer when they finally see the light of low-performance codes running in a high-performance compute environment? Stephen From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 10:28:11 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12742; Thu, 4 Feb 93 10:28:11 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09131; Thu, 4 Feb 93 10:26:01 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 10:26:00 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09123; Thu, 4 Feb 93 10:25:58 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA01127; Thu, 4 Feb 93 09:25:56 CST Date: Thu, 4 Feb 93 09:25:56 CST From: Tony Skjellum Message-Id: <9302041525.AA01127@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, john@cs.wmich.edu Subject: Re: 5184 bottles of sends on the wall... Please remember that overlapped send/receive will only make sense on systems with excess memory bandwidth. Hence, it might make more sense to implement a system where this is permitted by the implementor, but not required. Otherwise, calculations might slow down instead of achieving pipelined improvements. As Al points out, it is only the heavy-duty guys who try this sort of stuff... - Tony From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 10:30:25 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12830; Thu, 4 Feb 93 10:30:25 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09278; Thu, 4 Feb 93 10:28:57 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 10:28:56 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09270; Thu, 4 Feb 93 10:28:53 -0500 Received: from id.wmich.edu by cs.wmich.edu (4.1/SMI-4.1) id AA16295; Thu, 4 Feb 93 10:22:30 EST Date: Thu, 4 Feb 93 10:22:30 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9302041522.AA16295@cs.wmich.edu> To: mpi-comm@CS.UTK.EDU Subject: Re: 5184 bottles of sends on the wall... My initial concept of mixing IO types was that blocking and non-blocking could be freely mixed, but that synchronized was special and a synchronized send would only match a synchronized receive. The rationale for dropping the synchronized recieve is OK with me. I have no problems with the current proposal. Overlappping communication and computation is the only way to get good prerformance in many applications. It must be assumed to be the norm rather than the exception. I would not be able to use a system that did not support it. john From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 10:46:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13213; Thu, 4 Feb 93 10:46:15 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10063; Thu, 4 Feb 93 10:45:12 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 10:45:11 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10042; Thu, 4 Feb 93 10:45:06 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA12579 (5.65c/IDA-1.4.4 for ); Thu, 4 Feb 1993 10:45:01 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA06100; Thu, 4 Feb 93 15:44:56 GMT Date: Thu, 4 Feb 93 15:44:56 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302041544.AA06100@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA03785; Thu, 4 Feb 93 15:43:15 GMT To: tony@Aurora.CS.MsState.Edu Cc: mpi-comm@cs.utk.edu, john@cs.wmich.edu In-Reply-To: Tony Skjellum's message of Thu, 4 Feb 93 09:25:56 CST <9302041525.AA01127@Aurora.CS.MsState.Edu> Subject: 5184 bottles of sends on the wall... Content-Length: 1025 > Please remember that overlapped send/receive will only make sense on > systems with excess memory bandwidth. Hence, it might make more sense > to implement a system where this is permitted by the implementor, but > not required. Otherwise, calculations might slow down instead of > achieving pipelined improvements. True, and all the more reason to avoid as many copies as possible. After all if taking it from the user buffer consumes bandwidth 1, then the bandwidth cost of a send which needed a copy will be 2+1 = 3. This can be significant. (Of course I'm assuming a "fair" memory system here, in which taking from one port only loses the same from another. I've seen systems where this wasn't true, and this can change the balance.) -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 15:33:00 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19567; Thu, 4 Feb 93 15:33:00 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25767; Thu, 4 Feb 93 15:30:40 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 15:30:37 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25752; Thu, 4 Feb 93 15:30:34 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA04724; Thu, 4 Feb 93 20:30:30 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA05190; Thu, 4 Feb 93 13:29:24 MST Date: Thu, 4 Feb 93 13:29:24 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9302042029.AA05190@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: A suggestion for a multi-level MPI I especially like "Level 1" of Rusty and Bill's proposal. I would like to see more detail in this section, like examples of mod_xxx() and do_xxx() routines. I think there will be a good deal of support for Level 1 with all the talk that's been going on about handles/invoices/envelopes (of course there's that annoying syntax thing...). I am concerned about how heterogeneous communications are handled (or avoided) in Levels 3 and 4. I would be a lot more comfortable if ALL send() and recv() routines were heterogeneous. Here's a slight modification to the original proposal: Level 4 ("novice users"): send(dest,tag,bufadd,datatype,numitems) recv(dest,tag,bufadd,datatype,numitems) Level 3 ("dangerous users" [like me :) ]): bsend(tag,dest,bufadd,datatype,numitems) nsend(tag,dest,bufadd,datatype,numitems) brecv(tag,dest,bufadd,datatype,numitems) nrecv(tag,dest,bufadd,datatype,numitems) or [n][send] [b][recv] in Bill and Rusty's shorthand. Level 2 ("really frightening users"): [c][n][send][ ] [s][b][recv][rr] [g][s] Here's the arguments: - I don't like the idea of having to tell novice users that they must change all the MPI communication routine calls in their code before their program will run on a heterogeneous system. I think a lot of novice users will be using heterogeneous workstation networks. The current proposal forces these folks to learn at least two versions of send() and recv(). - I don't think there are any significant performance reasons for having separate heterogeneous versions of these calls, as long as we require a receiving process to know what incoming message contents will be (which seems to be consistent with the original proposal). - I don't think that the additional "datatype" argument reduces "ease of use" very much. An incorrect datatype should be no more difficult to debug than an incorrect "bufadd" or "buflen" (and should be easier than a bad "tag" or "dest", especially if deadlock results!). In FORTRAN77, "numitems" should be easier to code portably than "buflen". Even in C, there is little difference in complexity between portable calls to heterogeneous and non-heterogeneous versions: (non-heterogeneous) float mybuf[NUMITEMS]; ... send(dest, tag, mybuf, NUMITEMS * sizeof(float)); (heterogeneous) send(dest, tag, mybuf, MPI_FLOAT, NUMITEMS); - This gets rid of a whole bunch of send() and recv() variant routines (at least 22). - If we really want non-heterogeneous communication, it could be stuck in Level 1 with all the other stuff that's really dangerous for "novices". Another option would be to allow a MPI_BYTE datatype (not sure how this would work in FORTRAN). This may be useful for library developers. I think it's a good idea to keep "raw bytes" away from novice users (especially in FORTRAN). Tom Henderson hender@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 15:49:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20005; Thu, 4 Feb 93 15:49:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27824; Thu, 4 Feb 93 15:48:00 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 15:47:58 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27816; Thu, 4 Feb 93 15:47:57 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA01577; Thu, 4 Feb 93 14:47:56 CST Date: Thu, 4 Feb 93 14:47:56 CST From: Tony Skjellum Message-Id: <9302042047.AA01577@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, hender@macaw.fsl.noaa.gov Subject: Re: A suggestion for a multi-level MPI Tom, It is an artificial requirement to restrict heterogeneous messages to a single data type. - Tony From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 15:53:51 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20074; Thu, 4 Feb 93 15:53:51 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28380; Thu, 4 Feb 93 15:52:40 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 15:52:38 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28367; Thu, 4 Feb 93 15:52:36 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA04817; Thu, 4 Feb 93 20:52:32 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA05241; Thu, 4 Feb 93 13:51:27 MST Date: Thu, 4 Feb 93 13:51:27 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9302042051.AA05241@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: A suggestion for a multi-level MPI > Tom, > > It is an artificial requirement to restrict heterogeneous messages to a single > data type. > - Tony > I agree. My interpretation of Rusty and Bill's proposal is that messages that contain multiple data types would be constructed using the Level 1 routines-- something similar to Zipcode's invoices. Tom From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 17:02:43 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21865; Thu, 4 Feb 93 17:02:43 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03500; Thu, 4 Feb 93 17:01:23 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 17:01:22 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from iliamna.cse.ogi.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03492; Thu, 4 Feb 93 17:01:20 -0500 Received: by iliamna.cse.ogi.edu (/\==/\ Smail3.1.25.1 #25.15) id ; Thu, 4 Feb 93 14:01 PST Message-Id: Date: Thu, 4 Feb 93 14:01 PST From: otto@iliamna.cse.ogi.edu (Steve Otto) To: mpi-comm@cs.utk.edu Subject: 5184; one small point I got into my office late today, and I must say, how exciting to see all the mail concerning 5184! Jim Cownie writes: >The data is necessarily serialised to some degree over the >communications medium, so where's the problem ? I believe it is >actually HARDER to insist that the buffer sepcifications match at each >end than not to do this. (Certainly I have to pass additional >information with the message if I'm to check this.) I did not mean to imply that the standard should RULE OUT mix+match or that it should check for it at run time, merely that the standard should not make the guarantee that one IS allowed to do this. There is an important difference. I really don't believe we can foresee everything, so we should be conservative on what the standard guarantees. Jim agrees that in the case of heterogeneity, it does seem that we want matching: >The only place where I can see a need for matching of send and receive >formats is in the heterogeneous case. Here I would be happy to insist >that a heterogeneous send be matched to a heterogeneous receive. So maybe we don't want to allow all 5184 combinations. OK, at least we are admitting that there is an issue here. --Steve Otto From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 17:52:12 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23307; Thu, 4 Feb 93 17:52:12 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06478; Thu, 4 Feb 93 17:50:57 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 17:50:56 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ssdintel.ssd.intel.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06466; Thu, 4 Feb 93 17:50:53 -0500 Received: from fuji.ssd.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA17493; Thu, 4 Feb 93 14:50:39 PST Message-Id: <9302042250.AA17493@SSD.intel.com> To: otto@iliamna.cse.ogi.edu (Steve Otto) Cc: mpi-comm@cs.utk.edu, prp@SSD.intel.com Subject: Re: 5184; one small point In-Reply-To: Your message of "Thu, 04 Feb 93 14:01:00 PST." Date: Thu, 04 Feb 93 14:50:38 -0800 From: prp@SSD.intel.com > >The data is necessarily serialised to some degree over the > >communications medium, so where's the problem ? I believe it is > >actually HARDER to insist that the buffer sepcifications match at each > >end than not to do this. (Certainly I have to pass additional > >information with the message if I'm to check this.) > > I did not mean to imply that the standard should RULE OUT > mix+match or that it should check for it at run time, > merely that the standard should not make the > guarantee that one IS allowed to do this. There is an > important difference. I really don't believe we can foresee > everything, so we should be conservative on what the standard > guarantees. > > --Steve Otto > One of the important differences is that if the standard does not guarantee that you can mix, you can't write a portable program that mixes. Since we have seen one or two good examples of useful mixing, I think its important to guarantee the ability to mix at least for some combinations. My preference would be to guarantee mixing in all but specific cases, as few as possible. In fact, I would recommend against including features that cannot be mixed. I am optimistic that we can take care of the heterogeneous case in a sufficiently uniform way that mixing heterogeneous levels with non-heterogeneous becomes a non-issue. There was a good argument for not having a non-heterogeneous level, that would take care of it. The one case where mixing probably can't be justified is synchronous operation, where the send does not complete until the receive does. I've never liked this option, and worry that some support comes from misunderstanding about the difference between blocking and synchronous, as evidenced by Chuck's comment "You can't reuse the buffer being sent until you know that it has been received at the other side". This is of course not true. If the blocking send semantics are that 1) the send completes when the send buffer can be written safely and 2) message delivery is guaranteed, then you may happily overwrite the send buffer when the send completes with the assurance that the undamaged message will eventually appear at the receiver if its not already there. The only interesting difference between blocking send and synchronous send is in the amount of system buffering that may be required by applications. A correct application using blocking send might require some amount of system buffering, while a correct application that exclusively uses synchronous send requires no system buffering. In my opinion, we should focus on the problem of implied system buffering in the system environment area and eliminate synchronous send from the point-to-point portion of the standard. With reasonable care, we can make a clear and simple guarantee that appropriate different sends and receives can be mixed freely. This will allow people to write portable applications which use mixed structures. Paul From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 17:56:31 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23412; Thu, 4 Feb 93 17:56:31 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06726; Thu, 4 Feb 93 17:55:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 17:55:40 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from hub.ucsb.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06712; Thu, 4 Feb 93 17:55:37 -0500 Received: from garuda (garuda.ucsb.edu) by hub.ucsb.edu; id AA06657 sendmail 4.1/UCSB-2.0-sun Thu, 4 Feb 93 14:55:27 PST for mpi-comm@cs.utk.edu Received: by garuda (4.1/UCSB-v2) id AA22402; Thu, 4 Feb 93 13:56:59 PST Date: Thu, 4 Feb 93 13:56:59 PST From: ambuj%cs@hub.ucsb.edu (Ambuj Singh) Message-Id: <9302042156.AA22402@garuda> To: mpi-comm@cs.utk.edu Subject: Asynchronous send operations Hi: Here is a suggestion regarding the various kinds of send operations. From a semantic point of view, the synchronous and the asynchronous versions are meaningful. The further division of the asynchronous send into two kinds based on whether the buffer has been emptied or not is an implementation issue. Implementors of a system where buffer space is not a problem may wish to return without waiting for the buffer to empty whereas implementors of a system with limited buffer space may wish to suspend the operation until the buffer space is emptied. As to which version of asynchronous send operation is being implemented should be transparent and should not affect the correctness of a program. Ambuj Singh Dept. of CS Univ. of California Santa Barbara, CA 93106 ambuj@cs.ucsb.edu From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 18:06:52 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23575; Thu, 4 Feb 93 18:06:52 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06726; Thu, 4 Feb 93 17:55:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 17:55:40 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from hub.ucsb.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06712; Thu, 4 Feb 93 17:55:37 -0500 Received: from garuda (garuda.ucsb.edu) by hub.ucsb.edu; id AA06657 sendmail 4.1/UCSB-2.0-sun Thu, 4 Feb 93 14:55:27 PST for mpi-comm@cs.utk.edu Received: by garuda (4.1/UCSB-v2) id AA22402; Thu, 4 Feb 93 13:56:59 PST Date: Thu, 4 Feb 93 13:56:59 PST From: ambuj%cs@hub.ucsb.edu (Ambuj Singh) Message-Id: <9302042156.AA22402@garuda> To: mpi-comm@cs.utk.edu Subject: Asynchronous send operations Hi: Here is a suggestion regarding the various kinds of send operations. From a semantic point of view, the synchronous and the asynchronous versions are meaningful. The further division of the asynchronous send into two kinds based on whether the buffer has been emptied or not is an implementation issue. Implementors of a system where buffer space is not a problem may wish to return without waiting for the buffer to empty whereas implementors of a system with limited buffer space may wish to suspend the operation until the buffer space is emptied. As to which version of asynchronous send operation is being implemented should be transparent and should not affect the correctness of a program. Ambuj Singh Dept. of CS Univ. of California Santa Barbara, CA 93106 ambuj@cs.ucsb.edu From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 18:15:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23912; Thu, 4 Feb 93 18:15:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06726; Thu, 4 Feb 93 17:55:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 17:55:40 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from hub.ucsb.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06712; Thu, 4 Feb 93 17:55:37 -0500 Received: from garuda (garuda.ucsb.edu) by hub.ucsb.edu; id AA06657 sendmail 4.1/UCSB-2.0-sun Thu, 4 Feb 93 14:55:27 PST for mpi-comm@cs.utk.edu Received: by garuda (4.1/UCSB-v2) id AA22402; Thu, 4 Feb 93 13:56:59 PST Date: Thu, 4 Feb 93 13:56:59 PST From: ambuj%cs@hub.ucsb.edu (Ambuj Singh) Message-Id: <9302042156.AA22402@garuda> To: mpi-comm@cs.utk.edu Subject: Asynchronous send operations Hi: Here is a suggestion regarding the various kinds of send operations. From a semantic point of view, the synchronous and the asynchronous versions are meaningful. The further division of the asynchronous send into two kinds based on whether the buffer has been emptied or not is an implementation issue. Implementors of a system where buffer space is not a problem may wish to return without waiting for the buffer to empty whereas implementors of a system with limited buffer space may wish to suspend the operation until the buffer space is emptied. As to which version of asynchronous send operation is being implemented should be transparent and should not affect the correctness of a program. Ambuj Singh Dept. of CS Univ. of California Santa Barbara, CA 93106 ambuj@cs.ucsb.edu From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 19:10:08 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA24655; Thu, 4 Feb 93 19:10:08 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11923; Thu, 4 Feb 93 19:09:15 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 19:09:13 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from almaden.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11915; Thu, 4 Feb 93 19:09:11 -0500 Message-Id: <9302050009.AA11915@CS.UTK.EDU> Received: from almaden.ibm.com by almaden.ibm.com (IBM VM SMTP V2R2) with BSMTP id 9078; Thu, 04 Feb 93 16:09:16 PST Date: Thu, 4 Feb 93 16:09:15 PST From: "Ching-Tien (Howard) Ho" To: mpi-comm@cs.utk.edu Subject: re: > Please remember that overlapped send/receive will only make sense on > systems with excess memory bandwidth. Hence, it might make more sense > to implement a system where this is permitted by the implementor, but > not required. Otherwise, calculations might slow down instead of > achieving pipelined improvements. As Al points out, it is only the > heavy-duty guys who try this sort of stuff... > > - Tony A message passing system with only blocking send and blocking recv is hard to implement a shift or circulant shift efficiently and safely. Either you write an unsafe program bsend (right_nbr) brecv (left_nbr) which may deadlock depending on the size of the system buffer, or you do a two-phase or three-phase algorithm based on parity, which may require pointer jumping and choosing a leader. Regards, -- Howard From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 20:10:38 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25133; Thu, 4 Feb 93 20:10:38 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14638; Thu, 4 Feb 93 20:09:30 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 20:09:29 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14630; Thu, 4 Feb 93 20:09:28 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA00403; Thu, 4 Feb 93 19:09:20 CST Date: Thu, 4 Feb 93 19:09:20 CST From: Tony Skjellum Message-Id: <9302050109.AA00403@Aurora.CS.MsState.Edu> To: ho@almaden.ibm.com Subject: re: my comments on overlapping Cc: mpi-comm@cs.utk.edu BTW, I do not mean blocking send / blocking receive only in my comments. I mean asynchronous reading of a specific message (aka intel irecv) going on at the same time as processing. The message appears in the buffer, and one can poll for that to be complete. In practice, message passing does happen in background on all these systems, but the algorithms are not explicitly using this feature, or encouraging this overlap, necessarily. They might process messages at high priority, and only compute when message input queues are empty. Lennart has always repeated this warning that the overlap is only useful from a performance sense, and therefore from the point of view of a programmer trying to ask for this, if the system can perform better with the overlap. I DO NOT ADVOCATE block send / block receive only. Best wishes, Tony ----- Begin Included Message ----- From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 18:18:36 1993 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 19:09:13 EST Date: Thu, 4 Feb 93 16:09:15 PST From: "Ching-Tien (Howard) Ho" To: mpi-comm@cs.utk.edu Subject: re: Content-Length: 831 > Please remember that overlapped send/receive will only make sense on > systems with excess memory bandwidth. Hence, it might make more sense > to implement a system where this is permitted by the implementor, but > not required. Otherwise, calculations might slow down instead of > achieving pipelined improvements. As Al points out, it is only the > heavy-duty guys who try this sort of stuff... > > - Tony A message passing system with only blocking send and blocking recv is hard to implement a shift or circulant shift efficiently and safely. Either you write an unsafe program bsend (right_nbr) brecv (left_nbr) which may deadlock depending on the size of the system buffer, or you do a two-phase or three-phase algorithm based on parity, which may require pointer jumping and choosing a leader. Regards, -- Howard ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Fri Feb 5 18:05:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14845; Fri, 5 Feb 93 18:05:15 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03769; Fri, 5 Feb 93 18:03:08 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 5 Feb 1993 18:03:06 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ssdintel.ssd.intel.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03760; Fri, 5 Feb 93 18:03:04 -0500 Received: from fuji.ssd.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA06572; Fri, 5 Feb 93 15:02:57 PST Message-Id: <9302052302.AA06572@SSD.intel.com> To: jwf@lion.parasoft.com (Jon Flower) Cc: mpi-comm@cs.utk.edu, prp@SSD.intel.com Subject: Re: 5184; one small point In-Reply-To: Your message of "Thu, 04 Feb 93 14:01:00 PST." Date: Fri, 05 Feb 93 15:02:57 -0800 From: prp@SSD.intel.com Folks, here is an offline exchange Jon and I had that you might enjoy. To: prp@SSD.intel.com Subject: Re: 5184; one small point Paul, I saw your message about blocking and synchronous messages. Without wanting to speak for Chuck I think there is a pretty important difference between the two, particularly for applications such as Oracle. Using the blocking send it is true that the send buffer can be reused as soon as it is "clear" AS LONG AS IT IS GUARANTEED that it will be delivered. Unfortuntely this can't be guaranteed in the case where system buffers are required because the system may simply be out of memory at the receiver. In this case MPI and other systems will simply drop the offending message on the ground. For a database I assume this would be fatal. On the other hand the synchronous scheme has the advantage that you know it's worked when it returns, on both send and receive sides. Jon Flower This is an area where we must be clear about the semantics. We have agreed to some extent that message passing in MPI will be reliable. If I remember correctly, this passed on a straw vote early in the first point-to-point discussion. When I say "agreed to some extent" I mean that not everyone has the same understanding of what the standard is about on this issue. To me it means that if I have a send, and a receive that matches the message sent, that the message will absolutely be delivered. This is how NX works. If there isn't enough buffer space, the send must block so that the message waits in the send buffer until it can be delivered. Flow control is required in the underlying protocol. There seem to be a lot of people that expect no underlying protocol. When you say "send", the system sends - whether the message can be delivered or not. Apparently some systems work this way, and therefore it is expected behavior for people who have been users of these systems. To me, it is not at all compatible with the notion of reliable message passing. If you have reliable message passing as I understand it, clearly the important difference you mention goes away, leaving the lesser difference I wrote about. Without reliable message passing, I see no point whatsoever in blocking send. In fact, I see little point in having a typed send/receive standard, because such an interface is loaded up with friendly help for the user and it seems ridiculous to turn around and expect that same user to worry about reliable delivery. Does any of this make sense? Paul Hi Paul. I think what you're saying makes some sense. I have two more questions though.... a) Having the sender block until memory is available potentially invalidates some algorithms, particularly as the number of nodes grows. It's quite easy to come up with a sort of "ring" algorithm in which everyone ends up hung. I would agree that the trivial case (a single one-dimensional ring) in which this happens is a "user error" but there are more complex multi-dimensional cases in which it is quite hard for users to guarantee that it won't come up, even with quite reasonable programs. b) What are the implications of the flow control that you are using on performance? Is your routing hardware doing the claiming of memory blocks on the receiver or is there an initial software handshake to get started. The latter probably costs a fair amount that the "high priests" don't like??? I agree that the definition of "reliable devliery" in the MPI picture is not solid. I thought I remembered a piece that actually said that it was OK to drop a message if there wasn't enough memory on the receiver but I could be hallucinating. I wonder if it's worth trying to straighten this out? I have a nasty feeling that we'll get down to implementation details. If we can't get IBM and TMC to agree that Node 0: send to 1 read from 1 Node 1: send to 0 read from 0 is a reasonable thing to do I don't see much hope for anything else at this level....... Jon > a) Having the sender block until memory is available... This is inherently a hard problem that is all about system buffering. You can't solve this problem by careful or clever definition of the semantics of a message passing interface (whether and when sends block, and whether message passing is reliable.) If you push it down here, it pops up over there - depending on the semantics defined for the interface, the problem must be solved by the system or by the user. My preference (for a typed send/receive interface) is to push the problem to the system side, as much as possible. That way most users won't have to deal with it. But you can't completely solve the problem on the system side, because the user can always write a program that will confound any system buffering that has finite buffer space. With a reasonably good system implementation, it is actually rather difficult to come up with an algorithm that breaks it. Also, it is possible to define straightforward rules that will guarantee correct execution. That way, although the user must be careful being careful is easy. > b) What are the implications of the flow control... There is a performance penalty. People have claimed that NX overheads are too high, and it is true that they are noticeably higher than raw hardware. But there is quite a bit of overhead from message matching too, so for those who want ultimate performance there should be an alternative to the typed send/receive interface. Active messages is a good candidate. Such an interface is outside the scope of MPI. > I agree that the definition of "reliable devliery" in the MPI > picture is not solid. I thought I remembered a piece that actually > said that it was OK to drop a message if there wasn't enough > memory on the receiver but I could be hallucinating. I wonder It was in the first draft. > if it's worth trying to straighten this out? I have a nasty > feeling that we'll get down to implementation details. If we > can't get IBM and TMC to agree that > > Node 0: > send to 1 > read from 1 > > Node 1: > send to 0 > read from 0 > > is a reasonable thing to do I don't see much hope for anything > else at this level....... We have to develop and agree on a way to manage system buffering. To get Intel, IBM, and TMC together it must allow for the case where the above works and the case where it doesn't. > > Jon Paul Do you have any plan for how to reconcile the difference between the three camps on the simple program I sent you? My impression is that: NX (& Express, in fact) want that program to work as well as it possibly can. I believe that >95% of Express users code that way and are successful. I think you mentioned a similar fact in Dallas. TMC is prepared to say that the program is incorrect and won't work at all. IBM is not convinced either way. I don't see how to reconcile these camps in any good way. If we allow for the possibility that the code doesn't work AT ALL then the 95% of Express codes I mentioned above ALL have to be re-written in order to become portable. Furthermore, if the only way to guarantee portability, even within MPI, is to assume "half-duplex" communication then why not toss the other kind completely -- it would certainly reduce the complexity of the draft? Jon I think the answer (if there is one) is to provide a mechanism where the user declares a need for system buffering. This is a job for the system environment group to work out properly. With a declaration, you get: - Portability: The code is written for a specific buffering model but will port between environments because it tells the system what it needs. - Efficiency: If the user declares no buffering, the system can eliminate the associated overhead. (How to do this is a trick.) - Bloat: Every vendor must provide system buffering in case the user declares a need for it. Paul I agree....... do you think TMC will? Should we attempt to make a formal suggestion along these lines to the entire group. "Prepare to be flamed!!!!" Jon [I'm ready] Paul From owner-mpi-comm@CS.UTK.EDU Fri Feb 5 18:59:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA15545; Fri, 5 Feb 93 18:59:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06448; Fri, 5 Feb 93 18:57:54 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 5 Feb 1993 18:57:52 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from hub.ucsb.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06440; Fri, 5 Feb 93 18:57:50 -0500 Received: from garuda (garuda.ucsb.edu) by hub.ucsb.edu; id AA18688 sendmail 4.1/UCSB-2.0-sun Fri, 5 Feb 93 15:55:42 PST for mpi-comm@cs.utk.edu Received: by garuda (4.1/UCSB-v2) id AA23498; Fri, 5 Feb 93 14:42:13 PST Date: Fri, 5 Feb 93 14:42:13 PST From: ambuj%cs@hub.ucsb.edu (Ambuj Singh) Message-Id: <9302052242.AA23498@garuda> To: mpi-comm@cs.utk.edu Subject: order preservation Hello: The issue of order preservation of messages when there are multiple threads at the sender was not clarified at the January meeting. What do people have to say about this? It seems that there is no need to require message orderings across different threads. In other words, the order that the messages are received should be consistent with the ``program order'' which is defined by the order in which statements get executed. Thus messages in different threads may not be related and there should not be a need to preserve their order. On a different note, there was some question regarding the need for a synchronous send operation. One reason that they may be useful is that they might be the only means to ensure true portability. As illustrated by the following program that we discussed at the January meeting, asynchronous send operations are not portable across CM-5 and Intel. Process 1 Process 2 --------- --------- send to 2 send to 1 recv recv In other words, the standard could inquire that programs employing synchronous send/receive operations be portable across machines. It would be difficult to meet the same requirement for the asynchronous operations. --Ambuj Singh Dept. of CS, UCSB From owner-mpi-comm@CS.UTK.EDU Mon Feb 8 06:54:25 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18563; Mon, 8 Feb 93 06:54:25 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17241; Mon, 8 Feb 93 06:53:03 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Feb 1993 06:53:02 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17231; Mon, 8 Feb 93 06:52:56 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA03743 (5.65c/IDA-1.4.4 for ); Mon, 8 Feb 1993 06:52:53 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA26581; Mon, 8 Feb 93 11:52:48 GMT Date: Mon, 8 Feb 93 11:52:48 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302081152.AA26581@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA00721; Mon, 8 Feb 93 11:50:58 GMT To: mpi-comm@cs.utk.edu In-Reply-To: prp@SSD.intel.com's message of Fri, 05 Feb 93 15:02:57 -0800 <9302052302.AA06572@SSD.intel.com> Subject: The Jon and Paul show Content-Length: 1971 While I am no fan of arbitrary system buffering of messages, I believe that if it is to succeed MPI must allow the easy importation of existing Express and NX style codes. This necessarily implies that system buffering is required, so that code like this Blocking send to myself Blocking receive from myself works. (This is an even simpler version of the test case given by Jon/Paul). I agree with the approach that the user should declare her buffering requirement up front. This seems to be the only way to ensure portability. (Even if all it achieves is ensuring that she gets an "MPI-INSFMEM insufficient buffer space available" message on some systems, this is still preferable to a crash or hang). This is exactly the set of issues Marc was addressing in his discussion on SAFE and UNSAFE programs, and the amounts of buffering such codes could assume (he also addressed issues such as "How many outstanding messages can a code assume it is allowed ?"). I've a feeling that a lot of users (most ?) don't actually know what their buffer requirement is, however I still think that requiring a user specification of required buffer space is the correct way forward. (It might also be nice to have a system enquiry function which could return the buffer space currently used and the high water mark. This would at least let users find out about the behaviour of their codes. Unfortunately this information is rather implementation specific, and would not be accessible through the currently proposed profiling interface. Is this an issue for the profiling committee, or the environmental enquiry one ??? [As chair of the Profiling committe I'd vote for Environmental Enquiry !]) -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Mon Feb 8 07:10:19 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22055; Mon, 8 Feb 93 07:10:19 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17649; Mon, 8 Feb 93 07:09:28 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Feb 1993 07:09:27 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17641; Mon, 8 Feb 93 07:09:23 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA03842 (5.65c/IDA-1.4.4 for ); Mon, 8 Feb 1993 07:09:20 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA26809; Mon, 8 Feb 93 12:09:13 GMT Date: Mon, 8 Feb 93 12:09:13 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302081209.AA26809@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA00724; Mon, 8 Feb 93 12:07:22 GMT To: mpi-comm@cs.utk.edu In-Reply-To: prp@SSD.intel.com's message of Fri, 05 Feb 93 15:02:57 -0800 <9302052302.AA06572@SSD.intel.com> Subject: The Jon and Paul show [more] Content-Length: 2029 One more point I ommitted before... Paul says :- > to work out properly. With a declaration, you get: > > - Portability: The code is written for a specific buffering model but will > port between environments because it tells the system what it needs. > > - Efficiency: If the user declares no buffering, the system can eliminate > the associated overhead. (How to do this is a trick.) > > - Bloat: Every vendor must provide system buffering in case the user declares > a need for it. The last point is not actually true. It's entirely reasonable to have a standard which contains a buffering request and allow it to be refused. (There's not actually any point in having the request unless it can be refused !) I don't see a huge logical distinction between an implementation which permits system buffering of 1 byte and one which permits no system buffering. If the user needed 1Mb, then neither system is any use ! So it's merely an implementation quality issue. If someone has been able to sell their current system and keep their users happy without system buffering, then they should still be able to keep them happy with an MPI implementation in which the amount of system buffering is zero, and any request for more fails. Such an implementation would be standard conforming too. (Though such a vendor might expect some moaning from people trying to import code onto the machine, and maybe market pressure would push them into an implementation which did support system buffering !) (As an analogy, I believe that it is possible to have a standard conforming ADA implementation whose sole action when presented with any program is to raise the "Implementation limit exceeded" exception. Not useful, not saleable, but standard !) -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Mon Feb 8 08:54:49 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26134; Mon, 8 Feb 93 08:54:49 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20868; Mon, 8 Feb 93 08:54:03 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Feb 1993 08:54:02 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20822; Mon, 8 Feb 93 08:52:39 -0500 Date: Mon, 8 Feb 93 13:52:17 GMT Message-Id: <16939.9302081352@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: buffered/unbuffered comms (was compiler target (was Be brave. Be sure.)) To: mpi-pt2pt@cs.utk.edu, mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Hi Here comes my 5p worth on the subject of buffered and unbuffered comms. For completeness: by unbuffered I mean that the sender blocks until the message has been (or certainly will be) copied into the space of the receiver, a la occam for example; by buffered comms I mean that the message is copied away into a system buffer somewhere and the sender returns, a la PVM for example. It seems to me that there are a few different issues in this subject which discussions may have touched on. a) Ease of programming. I don't think anyone can really disagree that many programmers will report that they find buffered comms easier to use. These users have been fortunate enough not to have come up against the boundedness of buffering provided with the system. With high probability they are programming (or adapting) applications by inserting message passing calls directly into the application source. b) Portability of programs using MPI between different machines. [Here I give away my bias.] It is very difficult to make statements about portability (and reliability/correctness) of programs which use buffered messages and rely on system storage capacity and structure thereof. This becomes particularly difficult when the application makes use of substantive libraries, which themselves are written using buffered comms and place requirements on system buffering. It can just become to difficult to work out how much buffering you need. We played with a system which allowed the programmer to configure the system buffering, and this feature was only used (properly) by high priests. In my work I will only be able to use MPI if I can write substantive libraries in the confidence that they will not be subject to failure due to running out of such buffer space. Therefore we use only unbuffered message passing with a mixture of blocking/nonblocking (like Intel NX isend/irecv/msgwait) calls. Therefore my bias :-) c) Portability of existing applications (using existing message passing interfaces) to MPI. This is a different subject to (c). Following the discussion it seems that an argument in favour of adopting buffered comms is the number of existing programs (eg, lotsa programs written using Express) which use unbuffered comms. It has been my experience that migration of applications between different, broadly similar, message passing interfaces is not so difficult, except for this point. In a nutshell, the surgery you have to perform to move programs/libraries between an interface that forces buffered comms and one which forces unbuffered comms and blocking/nonblocking (isend/irecv/msgwait) is grevious and error prone. Given the above, I come to the conclusions: i) MPI must contain unbuffered communications with blocking/nonblocking (irecv/isend/msgwait kinda thing) calls, for reliability and portability. ii) If a goal of MPI is that existing applications using message passing interfaces (eg Express, PVM) should easily port to MPI, then MPI must also contain buffered comms. This seems to be a matter for the full committee, hence I have crossposted. Incidentally, to pitch in 2ps on some other lines of discussion: * I can see no particular benefit in allowing a buffered snd/rcv match an unbuffered rcv/snd. I can see considerable inconvenience in forbidding a blocking send/recv to match with a nonblocking recv/send. * Yes, we do try to overlap communications with calculation. (Sometimes it doesnt buy you, sometimes it does. Sometimes we have specifically needed to avoid such overlap in order to maximise performance, but that was a weird one :-) Just as important, we frequently "overlap" communication with communication (ie, use nonblocking calls) to avoid deadlock. * Please, oh please, let MPI decide that communications should be adressed using a rank relative to a group/context (0 ... GroupSize - 1) or (1 ... GroupSize). We do this all the time, and its very very convenient. In fact, when we can't do this we end up having to create arrays of the task identifiers - we end up doing it ourselves anyway. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Tue Feb 9 02:23:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19163; Tue, 9 Feb 93 02:23:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06697; Tue, 9 Feb 93 02:22:48 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 9 Feb 1993 02:22:47 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06689; Tue, 9 Feb 93 02:22:45 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA20792; Tue, 9 Feb 93 01:22:44 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA04771; Tue, 9 Feb 93 01:22:41 CST Message-Id: <9302090722.AA04771@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: The Jon and Paul show (was: 5184; one small point) In-Reply-To: Your message of "Fri, 05 Feb 93 15:02:57 PST." <9302052302.AA06572@SSD.intel.com> Date: Tue, 09 Feb 93 01:22:40 CST From: Rusty Lusk | | Folks, here is an offline exchange Jon and I had that you might enjoy. .... | Should we attempt to make a formal suggestion along these lines | to the entire group. "Prepare to be flamed!!!!" (not a flame) I did enjoy this exchange, because it is a tutorial on the buffering problem. Paul says it perfectly: You can't solve the system buffering problem by careful or clever definition of the semantics of a message passing interface. I agree that most users would rather not worry about it (and don't) and we should allow systems that try to handle the problem for the user to do so (the example programs in the discussion should run). I agree with Jim Cownie on this point. For portability, an enquiry function can be provided to alert the user who checks to find out how much (or how little) buffering is provided. I think the environmental enquiry subcommittee should propose something along these lines. As Paul says, handling this robustly has a cost, and some users certainly want to avoid it. After all, the user can know some things about his program that the system cannot, and should be able to take over responsibility for the buffering problem if he wants to. This is just what is done in the case of the "ready-receiver" communication. I admit that this is not a particularly euphonious name, but some way is needed for the user to relieve the system of buffer management if he wants to do so in the interest of greater efficiency. Rusty From owner-mpi-comm@CS.UTK.EDU Wed Feb 10 01:16:47 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13938; Wed, 10 Feb 93 01:16:47 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11091; Wed, 10 Feb 93 01:15:37 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Feb 1993 01:15:35 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11083; Wed, 10 Feb 93 01:15:33 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA02367; Wed, 10 Feb 93 00:15:31 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA06777; Wed, 10 Feb 93 00:15:28 CST Message-Id: <9302100615.AA06777@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: Multiple levels of collective operations Date: Wed, 10 Feb 93 00:15:27 CST From: Rusty Lusk A few days ago, we posted a suggestion for a multi-level MPI specification as a way to manage the complexity inherent in providing a highly functional interface, and to make MPI accessible to users who did not need or want all of its capabilities. The examples there were only for point-to-point communication. We thought it worth exploring whether the same ideas might be useful in the context of the group operations. Here are a few thoughts. The idea here is not to propose specific syntax, but to suggest how simultaneous adoption of multiple levels of collective operations might help provide both a simple interface and a full-featured one. Level 4: (very, very simple) Here there would be only one group, the default group of all processes, so there is no group id in the parameter lists. MPI_barrier MPI_combine(...,SUM,...) MAX ... (That is, only a fixed set of combining operations) MPI_broadcast(....) All operations would block until globally completed. Message types would be just like level 4 of the point-to-point operations: either contiguous strings of bytes, or arrays of specific data types specified like (...,datatype,numitems) The main point is that using just this level of group operations one could port many existing programs that use global operations, without introducing any explicit management of groups. Level 3: (still pretty simple, but more functionality) All the utilities proposed by Al Geist at the last meeting for the management of groups (creation, inquiry, etc.) Operations at this level have group id as an explicit argument (Again as Al suggested). User-defined combining functions. Gather operations. Main restriction: for simplicity, still simple contiguous buffers made up of either untyped byte strings or arrays of fixed types. All operations still block. In fact, we propose that there not be non-blocking collective operations, even at lower levels. The necessity of cooperating in the global operation beyond just making one's own explicit contribution puts non-blocking global operations more in the "threads" domain. Level 2: (More options for buffer structure, as in the point-to-point routines at this level) Here buffers specified for broadcasting, combining, etc., could be completely general, including mixed-type, non-contiguous buffers described by vectors of (address,datatype,numitems) triples. Level 1: (Full functionality, plus separate "setup" parts of the operations, like in Marc's point-to-point proposal.) Here operations would have a separate "init" call. Being separate has the same advantages as for the point-to-point operations, in that resources could be reused in loops. The "init" call would return a handle that could be further modified (setting buffer type, address, etc.) before use. The payoff seems potentially greater here because some of the "init" calls may turn out to be inherently non-scalable, and therefore reuse of their results is important. Summary: Multiple levels might help deal with conflicting demands for both simplicity and functionality in the case of collective operations as well as for point-to-point operations. What was generally discussed at the last meeting is very much like Level 3 above. An even simpler interface (so that groups need not be explicit at all) is provided by Level 4, while the somewhat grubby aspects of multiple and complicated buffer formats is pushed down into Level 2. Finally, Level 1 provides the opportunity to absorb some of the difficult-to-scale parts of global operations into separate, reusable calls. Comments? Rusty Lusk and Bill Gropp From owner-mpi-comm@CS.UTK.EDU Fri Feb 12 04:14:26 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05929; Fri, 12 Feb 93 04:14:26 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23402; Fri, 12 Feb 93 04:12:57 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 12 Feb 1993 04:12:57 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from enseeiht.enseeiht.fr by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23393; Fri, 12 Feb 93 04:12:50 -0500 Received: from marylin (marylin.enseeiht.fr) by enseeiht.enseeiht.fr, Fri, 12 Feb 1993 10:12:45 +0100 Message-Id: <199302120912.AA08776@enseeiht.enseeiht.fr> Date: Fri, 12 Feb 93 10:10:29 +0100 From: Michel DAYDE To: mpi-comm@cs.utk.edu Subject: Re Multiple levels of cllective operations I like the idea of multi-level MPI specifications since it is a nice way of dealing with functionality and simplicity. Also the fact that Level 4 of collective operations avoids to introduce any explicit management of groups will simplify the life. Michel Dayde, ENSEEIHT-IRIT and CERFACS. From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 07:56:19 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02001; Mon, 15 Feb 93 07:56:19 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14309; Mon, 15 Feb 93 07:54:13 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 07:54:12 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14301; Mon, 15 Feb 93 07:54:07 -0500 Date: Mon, 15 Feb 93 12:54:01 GMT Message-Id: <21675.9302151254@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Mixed F77/C, MIMD(not SPMD) To: mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Dear MPI Colleagues I'd like to raise a couple of points which I hope we can discuss at the meeting later this week, after lenghty discussions with a variety of local users. In general the objectives of the MPI Forum are very well received. Two specific concerns, of a very general nature, became apparent. a) The introduction says that MPI is for programmers who wish to write portable programs in C and/or Fortran77. The choice of languages is fine, although some may wish to use C++ and Fortran90. The concern is whether MPI intends to allow, and if so in what manner, mixed language programming, i.e. programs written in a mixture of C and Fortran77. The observation is that C is most useful for library programming, whereas a large number of applications that we deal with are written in Fortran77. (Users did not expect MPI say anything about cross-calling and such murky business, that is outwith message passing.) I propose that the make a statement of intent, possibly in the introduction, that mixed languages will be allowed/supported (I would hope that we could not make the decision to not allow/support mixture of C and F77 :-). This seems an easy thing to do. b) The working introduction of November 24 states that MPI includes (temporarily?) \item A simple way to create processes for the SPMD model This kinda carries the implication that MPI is for programming in the SPMD model. The January minutes state that It was agreed that explicit remarks that MPI is intended to be usable with multithreaded proesses and with MIMD (not just SPMD) programs should be added somewhere. We at EPCC most strongly support the proposal that MPI should provide for MIMD programs and not restrict itself to the world of SPMD programming. The SPMD model has been useful, and can be expected to remain useful, for tweaking serial (usually Fortran77) programs in a strictly data parallel fashion to derive a parallel program. However, this is not a string enough argument for restricting MPI to SPMD. There are a lot of applications out there which are not really suited to such simple tweaking. Further, we believe that MPI needs to provide support for library writers, where libraries can consist of a procedure library linked into user processes along with one or more library service processes. The concern felt by users and gurus alike is that our perception of much of the discussion in subcommittees is that it seems to be proceeding with scant consideration of issues in MIMD programming as opposed to SPMD programming. I suggest that this should be prioritised within the MPI. I am happy to write a proposal or discussion document on these issues, but I have no time to write either before the next meeting as my travel schedule starts at 6am tomorrow. So, I'm afraid this point was simply raising a flag that there is concern, without a proposal on the matter, for which I apologise. Looking forward to meeting and discussing at the meeting later this week! Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 10:21:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04616; Mon, 15 Feb 93 10:21:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20711; Mon, 15 Feb 93 10:19:40 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 10:19:38 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20702; Mon, 15 Feb 93 10:19:36 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA15992; Mon, 15 Feb 93 09:19:33 CST From: gropp@antares.mcs.anl.gov (William Gropp) Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA27181; Mon, 15 Feb 93 09:19:31 CST Date: Mon, 15 Feb 93 09:19:31 CST Message-Id: <9302151519.AA27181@godzilla.mcs.anl.gov> To: lyndon@epcc.ed.ac.uk Cc: mpi-comm@cs.utk.edu In-Reply-To: L J Clarke's message of Mon, 15 Feb 93 12:54:01 GMT <21675.9302151254@subnode.epcc.ed.ac.uk> Subject: Mixed F77/C, MIMD(not SPMD) X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 07:54:12 EST Date: Mon, 15 Feb 93 12:54:01 GMT From: L J Clarke Reply-To: lyndon@epcc.ed.ac.uk Again, a few clarifications: b) The working introduction of November 24 states that MPI includes (temporarily?) \item A simple way to create processes for the SPMD model This kinda carries the implication that MPI is for programming in the SPMD model. My interpretation (and in fact, the reason that Rusty and I included such a method in our implementation of the November draft) is that there needs to be some way to write, entirely in MPI, a running program, and there is a chance that we could agree on a method for running SPMD programs. It was definately NOT meant to exclude other models; just a position that MPI needs to include some way to write a complete (at the source code level; we do not intend to specify the OS interface to "launching" the program) parallel program for a common and simple model. I'd be quite happy adding text that indicates that MPI is intended for MIMD models as well. I would be very interested in seeing a proposal for an interface for MIMD programs. Bill From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 10:21:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04617; Mon, 15 Feb 93 10:21:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20721; Mon, 15 Feb 93 10:19:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 10:19:41 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20713; Mon, 15 Feb 93 10:19:40 -0500 Received: from jadoube.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA15995; Mon, 15 Feb 93 09:19:37 CST From: levine@antares.mcs.anl.gov (David Levine) Received: by jadoube.mcs.anl.gov (4.1/GeneV4) id AA21322; Mon, 15 Feb 93 09:19:36 CST Date: Mon, 15 Feb 93 09:19:36 CST Message-Id: <9302151519.AA21322@jadoube.mcs.anl.gov> To: lyndon@epcc.ed.ac.uk Cc: mpi-comm@cs.utk.edu Subject: Mixed F77/C, MIMD(not SPMD) X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 07:54:12 EST Date: Mon, 15 Feb 93 12:54:01 GMT From: L J Clarke Reply-To: lyndon@epcc.ed.ac.uk Dear MPI Colleagues I'd like to raise a couple of points which I hope we can discuss at the meeting later this week, after lenghty discussions with a variety of local users. In general the objectives of the MPI Forum are very well received. Two specific concerns, of a very general nature, became apparent. a) The introduction says that MPI is for programmers who wish to write portable programs in C and/or Fortran77. The choice of languages is fine, although some may wish to use C++ and Fortran90. The concern is whether MPI intends to allow, and if so in what manner, mixed language programming, i.e. programs written in a mixture of C and Fortran77. The observation is that C is most useful for library programming, whereas a large number of applications that we deal with are written in Fortran77. (Users did not expect MPI say anything about cross-calling and such murky business, that is outwith message passing.) I propose that the make a statement of intent, possibly in the introduction, that mixed languages will be allowed/supported (I would hope that we could not make the decision to not allow/support mixture of C and F77 :-). This seems an easy thing to do. Let me second this, if for no other reason than the lack of dynamic memory allocation in Fortran 77, and the relatively small physical memory available on the nodes of distributed-memory machines. In porting Fortran 77 codes to distributed-memory machines I consider the use of C for dynamic memory allocation to be "mandatory." --dave levine From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 12:20:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07231; Mon, 15 Feb 93 12:20:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25990; Mon, 15 Feb 93 12:18:19 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 12:18:17 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from chert.CS.ORST.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25979; Mon, 15 Feb 93 12:18:15 -0500 Received: from sycamore.CS.ORST.EDU by research.CS.ORST.EDU (4.1/1.30) id AA03629; Mon, 15 Feb 93 09:18:12 PST Date: Mon, 15 Feb 93 09:18:12 PST From: pancake@chert.CS.ORST.EDU (Cherri Pancake) Message-Id: <9302151718.AA03629@research.CS.ORST.EDU> To: mpi-comm@cs.utk.edu Subject: C/Fortran Compatibility I agree with Lyndon Clarke that interlanguage compatibility needs to be an explicitly stated goal of the standard. The user community is assuming that this is true.... Cherri Pancake From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 13:17:38 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09917; Mon, 15 Feb 93 13:17:38 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29053; Mon, 15 Feb 93 13:16:12 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 13:16:11 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from NA-GW.CS.YALE.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29045; Mon, 15 Feb 93 13:16:09 -0500 Received: from YOGI.NA.CS.YALE.EDU by CASPER.NA.CS.YALE.EDU via SMTP; Mon, 15 Feb 1993 13:16:06 -0500 Received: by YOGI.NA.CS.YALE.EDU (Sendmail-5.65c/res.client.cf-3.5) id AA28124; Mon, 15 Feb 1993 13:16:04 -0500 Date: Mon, 15 Feb 1993 13:16:04 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Message-Id: <199302151816.AA28124@YOGI.NA.CS.YALE.EDU> To: mpi-comm@cs.utk.edu Subject: C/Fortran Compatibility Ok, folks, What exactly do you mean by "interlanguage compatibility"? In particular are we going to ask that F77 programs be able to call C routines? Although I agree that this is a good idea, I don't think that dictating compiler (and linker) standards is in the jurisdiction of this committee. On a slightly different issue, it is not clear to me that a C++ interface would be different from an ANSI C interface. Although it is possible to make an object-oriented interface definition, I think that such a definition would be unwise for several reasons. First, if an ANSI C interface is available, anyone wanting an object oriented C++ interface could write one in a portable way. Second, I don't think that enough object-oriented message-passing interfaces have been written to determine what a standard should look like. Of course, I'm willing to persuaded on either point. -scott berryman Chairanimal, Language Binding From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 14:00:27 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11570; Mon, 15 Feb 93 14:00:27 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01123; Mon, 15 Feb 93 13:59:09 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 13:59:07 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01110; Mon, 15 Feb 93 13:59:04 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA00836; Mon, 15 Feb 93 12:58:50 CST Date: Mon, 15 Feb 93 12:58:50 CST From: Tony Skjellum Message-Id: <9302151858.AA00836@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, berryman-harry@CS.YALE.EDU Subject: Re: C/Fortran Compatibility I agree with Scott. I think that a valid, related point, is that there need to be a C interface which is convenient for C programmers, not just a FORTRAN interface, which is usable by C programmers, but not natural. - Tony ----- Begin Included Message ----- From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 12:41:33 1993 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 13:16:11 EST Date: Mon, 15 Feb 1993 13:16:04 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) To: mpi-comm@cs.utk.edu Subject: C/Fortran Compatibility Content-Length: 937 Ok, folks, What exactly do you mean by "interlanguage compatibility"? In particular are we going to ask that F77 programs be able to call C routines? Although I agree that this is a good idea, I don't think that dictating compiler (and linker) standards is in the jurisdiction of this committee. On a slightly different issue, it is not clear to me that a C++ interface would be different from an ANSI C interface. Although it is possible to make an object-oriented interface definition, I think that such a definition would be unwise for several reasons. First, if an ANSI C interface is available, anyone wanting an object oriented C++ interface could write one in a portable way. Second, I don't think that enough object-oriented message-passing interfaces have been written to determine what a standard should look like. Of course, I'm willing to persuaded on either point. -scott berryman Chairanimal, Language Binding ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 14:08:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11680; Mon, 15 Feb 93 14:08:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01543; Mon, 15 Feb 93 14:07:50 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 14:07:49 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01534; Mon, 15 Feb 93 14:07:48 -0500 Received: from id.wmich.edu by cs.wmich.edu (4.1/SMI-4.1) id AA01566; Mon, 15 Feb 93 14:00:38 EST Date: Mon, 15 Feb 93 14:00:38 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9302151900.AA01566@cs.wmich.edu> To: mpi-comm@CS.UTK.EDU Subject: C/Fortran Compatibility Asking for compatibility is a lot in some respects. going far beyond the MPI interface - such as asking a malloc from a C subroutine to "work" when called from a Fortran program. (library name collisions - need for initializations for both environments, cleaning up both environments etc.) I want to see them together too, but.... Perhaps we should not make it a full "requirement", but rather be careful to set up the MPI language bindings so that they could work together from C and Fortran. and indicate that this is intended on environments supporting mixed programs. I know of no standards work on mixed programs and recall the the discussion of it (ANSI?) as a general area for standardization a couple years back being postponed as too "hairy". cheers - john From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 14:24:08 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12649; Mon, 15 Feb 93 14:24:08 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02252; Mon, 15 Feb 93 14:23:11 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 14:23:10 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from chert.CS.ORST.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02243; Mon, 15 Feb 93 14:23:08 -0500 Received: from sycamore.CS.ORST.EDU by research.CS.ORST.EDU (4.1/1.30) id AA06780; Mon, 15 Feb 93 11:23:02 PST Date: Mon, 15 Feb 93 11:23:02 PST From: pancake@chert.CS.ORST.EDU (Cherri Pancake) Message-Id: <9302151923.AA06780@research.CS.ORST.EDU> To: mpi-comm@cs.utk.edu Subject: hat is Interlanguage Compatibility? To me, multilanguage support means that: (a) we provide both C and Fortran programmatic interfaces, and (b) operationally, there will be no perceptible difference whether the programmer uses the C interface, the Fortran, or both in his/her program. Cherri Pancake From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 14:36:03 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13372; Mon, 15 Feb 93 14:36:03 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02815; Mon, 15 Feb 93 14:34:51 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 14:34:50 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from NA-GW.CS.YALE.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02805; Mon, 15 Feb 93 14:34:49 -0500 Received: from YOGI.NA.CS.YALE.EDU by CASPER.NA.CS.YALE.EDU via SMTP; Mon, 15 Feb 1993 14:34:46 -0500 Received: by YOGI.NA.CS.YALE.EDU (Sendmail-5.65c/res.client.cf-3.5) id AA28237; Mon, 15 Feb 1993 14:34:44 -0500 Date: Mon, 15 Feb 1993 14:34:44 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Message-Id: <199302151934.AA28237@YOGI.NA.CS.YALE.EDU> To: mpi-comm@cs.utk.edu Subject: Any opinions? In my earlier message, I suggested that a C++ interface was not needed if an ANIS C interface was available. Are there any objections to this? -scott berryman From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 14:44:24 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13826; Mon, 15 Feb 93 14:44:24 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03257; Mon, 15 Feb 93 14:43:38 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 14:43:37 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from NA-GW.CS.YALE.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03248; Mon, 15 Feb 93 14:43:36 -0500 Received: from YOGI.NA.CS.YALE.EDU by CASPER.NA.CS.YALE.EDU via SMTP; Mon, 15 Feb 1993 14:43:34 -0500 Received: by YOGI.NA.CS.YALE.EDU (Sendmail-5.65c/res.client.cf-3.5) id AA28255; Mon, 15 Feb 1993 14:43:33 -0500 Date: Mon, 15 Feb 1993 14:43:33 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Message-Id: <199302151943.AA28255@YOGI.NA.CS.YALE.EDU> To: mpi-comm@cs.utk.edu Subject: Perhaps another place? Cherri, Perhaps we should carry on the thread about C++, ANSI C, F77, and F90 to the mpi-lang mailing list. At any rate, I'm very much for having the interface consistent across languages. This does however, limit what we can do in C++ and F90. We cannot, for example, use methods in C++ or optional arguments in F90. Will such limitations cause much gnashing of teeth in the user community? Will we hate ourselves in a year or two for not taking advantage of the truly rightous F90 features? -scott berryman From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 15:11:20 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14568; Mon, 15 Feb 93 15:11:20 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04587; Mon, 15 Feb 93 15:09:52 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 15:09:51 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ssdintel.ssd.intel.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04578; Mon, 15 Feb 93 15:09:49 -0500 Received: from tualatin.SSD.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA25287; Mon, 15 Feb 93 11:39:31 PST Date: Mon, 15 Feb 93 11:39:31 PST Message-Id: <9302151939.AA25287@SSD.intel.com> Received: by tualatin.SSD.intel.com (4.1/SMI-4.0) id AA06755; Mon, 15 Feb 93 11:39:30 PST From: Bob Knighten Sender: knighten@SSD.intel.com To: john@cs.wmich.edu Cc: mpi-comm@CS.UTK.EDU Subject: Re: C/Fortran Compatibility In-Reply-To: <9302151900.AA01566@cs.wmich.edu> References: <9302151900.AA01566@cs.wmich.edu> Reply-To: knighten@SSD.intel.com (Bob Knighten) POSIX Standardized Profiles are one standard arena where some effort is being made to guarantee some expected level of interoperability. My POSIX working group (P1003.14) is producing the P1003.18 profile for "classic POSIX" covering P1003.1 (system interfaces) and P1003.2 (shell and utilities). This does not include networks or any of the serious problems regarding interoperability. In this context we have been struggling just to find reasonable requirements to express such expectations as that a file written by a C program can be read by a FORTRAN program or an Ada program. We would not even dream of trying to stadardized the way you might link routines produced by different language compilers! -- Bob Robert L. Knighten | knighten@ssd.intel.com Intel Supercomputer Systems Division | 15201 N.W. Greenbrier Pkwy. | (503) 629-4315 Beaverton, Oregon 97006 | (503) 629-9147 [FAX] From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 17:26:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18114; Mon, 15 Feb 93 17:26:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10980; Mon, 15 Feb 93 17:25:12 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 17:25:11 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from chert.CS.ORST.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10972; Mon, 15 Feb 93 17:25:09 -0500 Received: from sycamore.CS.ORST.EDU by research.CS.ORST.EDU (4.1/1.30) id AA11917; Mon, 15 Feb 93 14:25:03 PST From: pancake@chert.CS.ORST.EDU (Cherri Pancake) Received: by sycamore.CS.ORST.EDU (4.1/CS-Client) id AA10017; Mon, 15 Feb 93 14:24:59 PST Date: Mon, 15 Feb 93 14:24:59 PST Message-Id: <9302152224.AA10017@sycamore.CS.ORST.EDU> To: mpi-comm@cs.utk.edu Subject: Response to Bob Knighten's comments Bob said: ?We would not even dream of trying to >stadardized the way you might link routines produced by different language >compilers! I don't think that's necessarily the goal of multilanguage support. I should be able to write a program that combines routines written in Fortran and C, using MPI for the message-passing. That does not mean that the function calls look identical in the two source languages, nor that there be a single set of library routines capable of servicing both. But it does mean that I should be able to send a message from a Fortran routine, calling the Fortran library module, and receive it from a C routine that uses the corresponding C library module. Cherri Pancake From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 17:45:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19007; Mon, 15 Feb 93 17:45:07 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11672; Mon, 15 Feb 93 17:43:36 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 17:43:35 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from enet-gw.pa.dec.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11664; Mon, 15 Feb 93 17:43:32 -0500 Received: by enet-gw.pa.dec.com; id AA27000; Mon, 15 Feb 93 14:41:53 -0800 Message-Id: <9302152241.AA27000@enet-gw.pa.dec.com> Received: from rdvax.enet; by decwrl.enet; Mon, 15 Feb 93 14:43:30 PST Date: Mon, 15 Feb 93 14:43:30 PST From: MPS ENGINEERING 223-4656 To: mpi-comm@cs.utk.edu Cc: benson@rdvax.enet.dec.com Apparently-To: mpi-comm@cs.utk.edu Subject: Some notes on MPI 15-FEB-1993 Some general comments from the Massively Parallel Systems Group of Digital Equipment Corporation on MPI: The MPI effort must strive to Keep It Simple !!! The multiple levels idea is good as long as it is clearly stated what the model is in terms of mixing the use of levels within an application. Language independence must be worried about.... C and FORTRAN are a MUST today, C++, ADA, and others will be of interest in the future. MPI must be a reliable message passing system. i.e. The user must not be burdened with running a protocol on top of MPI to support retransmission, etc..... (An optional non-reliable mode should not be ruled out.) MPI should be thread safe. All functions should return error codes not just a fail/succeed status. All subroutines should have a status argument into which an error code can be returned. Groups and communication contexts discussions and specifications must include thread usage rules and guidelines. When in doubt give the user the choice about buffering and other less than obvious characteristics.... But, consistently warn about portability issues WHENEVER a choice is offered. FLAG ALL CHOICES THAT CAN CAUSE PORTABILITY PROBLEMS !!! Be particularly careful with the homogeneous vs heterogeneous environment issues. i.e. Packing verses raw data, XDR etc.... Models / guidelines should be included to address the issue of sending structures and endian sensitive data between heterogeneous environments. When the syntax effort gets serious: All names should be readable, descriptive, and consistent. No numeric constants should exist in the MPI standard. All constants should be defined symbolically only !!! (POSIX inherited a lot of stuff from SVID and should not be use as a strict model) Data hiding via opaque types should be used unless there is a truly justifiable reason. A message type registration facility should be considered as an alternative to communication contexts. From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 18:56:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20138; Mon, 15 Feb 93 18:56:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14806; Mon, 15 Feb 93 18:54:48 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 18:54:46 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14798; Mon, 15 Feb 93 18:54:45 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA01171; Mon, 15 Feb 93 17:54:04 CST Date: Mon, 15 Feb 93 17:54:04 CST From: Tony Skjellum Message-Id: <9302152354.AA01171@Aurora.CS.MsState.Edu> To: benson@rdvax.enet.dec.com Subject: Re: Some notes on MPI Cc: mpi-comm@cs.utk.edu Please elaborate on these points so we can discuss them at the meeting, in the context subcommitee. >Data hiding via opaque types should be used unless there is a truly >justifiable reason. -- what does this mean???? >A message type registration facility should be considered as an alternative >to communication contexts. I see "current typing practice without any registration," "current typing practice plus context registration," and "typing registration with no degrees of freedom left for user," as the three alternatives. I think the latter is too restrictive, because user cannot encode semantically meaningful bits into types at all. In the first (current) option, this can be done, but there are big risks of conflicts in real programs. Hence, I continue to favor the middle option. - Tony From owner-mpi-comm@CS.UTK.EDU Tue Feb 16 08:18:41 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA16998; Tue, 16 Feb 93 08:18:41 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19183; Tue, 16 Feb 93 08:16:05 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 16 Feb 1993 08:16:03 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from inet-gw-1.pa.dec.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19171; Tue, 16 Feb 93 08:15:51 -0500 Received: by inet-gw-1.pa.dec.com; id AA16702; Tue, 16 Feb 93 05:15:46 -0800 Received: by mpsg.mps.mlo.dec.com; id AA14150; Tue, 16 Feb 93 08:15:38 -0500 Date: Tue, 16 Feb 93 08:15:38 -0500 From: linden@mps.mlo.dec.com (David C.P. Linden) Message-Id: <9302161315.AA14150@mpsg.mps.mlo.dec.com> To: tony@Aurora.CS.MsState.Edu Cc: benson@rdvax.enet.dec.com, mpi-comm@cs.utk.edu In-Reply-To: Tony Skjellum's message of Mon, 15 Feb 93 18:57:10 -0500 <9302152357.AA13311@mpsg.mps.mlo.dec.com> Subject: Some notes on MPI [I work with Ed in Digital's Massively Parallel Systems Group.] >Data hiding via opaque types should be used unless there is a truly >justifiable reason. This means that, unless there is good reason, the user gets a cookie and must use cookie-manipulation routines to perform actions. A major example is process ids. We think PVM took the better approach by having PIDs be opaque integers. This allows processes to join and leave, which MPI seems to rule out, but which we think should be reconsidered. Having the notion of 0..n-1 or 1..n may have some level of convenience, but we think it is too restrictive. (Additionally, there is the paradigm problem. 0..n-1 is the C way of numbering, but 1..n is the FORTRAN.) >A message type registration facility should be considered as an alternative >to communication contexts. A big problem we see with communication contexts is that they aren't thread-safe. Their purported purpose is to ensure different pieces of code (e.g., libraries, user codes) don't conflict with each other in the space of types. That's a real problem worthy of solution, but the lack of thread-safety in communication contexts casts doubt on CCs as a solution. A type registration system (compare (very loosely) the X windows color space registration system) would allow users and libraries to allocate and deallocate a range of types. With appropriate allocate routines, you could specify alignment constraints in order to bit-encode semantics. From owner-mpi-comm@CS.UTK.EDU Tue Feb 16 10:29:05 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25120; Tue, 16 Feb 93 10:29:05 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25032; Tue, 16 Feb 93 10:27:35 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 16 Feb 1993 10:27:33 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from chert.CS.ORST.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25024; Tue, 16 Feb 93 10:27:25 -0500 Received: from sycamore.CS.ORST.EDU by research.CS.ORST.EDU (4.1/1.30) id AA02715; Tue, 16 Feb 93 07:27:19 PST From: pancake@chert.CS.ORST.EDU (Cherri Pancake) Received: by sycamore.CS.ORST.EDU (4.1/CS-Client) id AA11099; Tue, 16 Feb 93 07:27:18 PST Date: Tue, 16 Feb 93 07:27:18 PST Message-Id: <9302161527.AA11099@sycamore.CS.ORST.EDU> To: mpi-comm@cs.utk.edu Subject: Continued comments on multilanguage support In response to Scott Berryman's comments: >At any rate, I'm very much for having the interface consistent >across languages. This does however, limit what we can do in C++ >and F90. We cannot, for example, use methods in C++ or optional >arguments in F90. Will such limitations cause much gnashing of teeth >in the user community? Will we hate ourselves in a year or two for >not taking advantage of the truly rightous F90 features? I'm not convinced that optional arguments are all that good for the general technical computing community, anyway, unless there is VERY GOOD interprocedural argument checking going on (which lets out most current systems). Consistency is much more important at this point in the evolution cycle. Cherri Pancake P.S. I tried to post this to mpi-lang, but got rejected. From owner-mpi-comm@CS.UTK.EDU Tue Feb 16 15:03:44 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08057; Tue, 16 Feb 93 15:03:44 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10592; Tue, 16 Feb 93 15:00:35 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 16 Feb 1993 15:00:32 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10578; Tue, 16 Feb 93 15:00:29 -0500 Received: from s307.cs.wmich.edu by cs.wmich.edu (4.1/SMI-4.1) id AA07659; Tue, 16 Feb 93 14:53:15 EST Date: Tue, 16 Feb 93 14:53:15 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9302161953.AA07659@cs.wmich.edu> To: mpi-comm@cs.utk.edu Hi; A late pre-meeting post. I hope we can talk about the possibility of adding an io specification to MPI in Dallas. A couple of us (Leslie Hart, ... and myself) have been talking about possible MPI io specifications. Here is a brief outline of IO specifications for one option, a loosely synchronized version of multiple IO based on the ANSI-C librtary. Placing "mp_" before all names is fine (It is my preference to do that in all libraries). Leslie and others have a seek and read/write primative whuich has merit if processes were going to read from differeent "random" positions in a file. Assuming a message is sent for every call this could be high overhead, but it keeps life simple. This also allows IO request to be asynchronous (assuming the seeks are absolute), although we may wear out the disk head actuator ;-). Perhaps we should set basic goals for an IO system: 1) interaction with a terminal 2) writing and reading buffers to files, which should correspond closely to sending and recieving messages. 3) three modes of operation - single_io(), ordered_io() and unordered_io() 4) some specification of conditions for deadlock free use. Some questions: Should all IO be loosely synchronized? Should all IO "require" a message be sent with that request (implying no buffering / queueing)? If not "require" then, should the user have any control over the buffering / queueing? I'm not sure about how to address hetrogenious access to a file. I have seen a couple options. I would tend to not want to force a "binary" file compatable with all architectures. Text files should not pose as large a problem. It does appear binary conversions by the message system in pt-2-pt has supporters, if that works out then soemthing could here too. This is still a thought exercise, not a proposal. john ---------------------------------------------------------------- The Three MPI IO Modes The three levels of routines below and the three IO-modes are orthogonal. All IO routines must be executed by all members of a group in a loosely synchronized manner. All processes must invoke the same primative on the same stream. The following IO routines operate in three modes. Initializing IO and switching between modes using: single_io(group) ordered_io(group) unordered_io(group) Terminating IO from a group is done by: end_io(group) Each of these calls must be executed by all members of a group. In single_io() mode the IO routines should function from a process as described in the relevant standard. On output or in setting the file position the results from only one of the processes will be done. On input all processes receive the same values. In ordered_io() and unordered_io() the results of setting file positions are done for only one process. Input and Output from a group of size p results in p separate items being read or written. The ordered_io() mode requires the p IO operations to occur in process order. (If we have a group order this should be group order) The order of the p IO operations in unordered_io() mode is unspecified. **************************************************************** The POSIX IEEE Std 1003.1-1988 C functions From Chapter 8, 4.9 (Which is a pointer to the ANSI-C standard X3.159-1989 which is essentially the same as ISO 9899-1990) Those functions not mentioned in POSIX which were later included in the ANSI-C are still be be considered required by POSIX. Below: the functions marked with a "*" are non-ANSI-C the functions marked with a 1 provide a complete set, without access to buffer cantrol. Level-1 IO -------------------------------------------------------- 1 fopen(), FILE * fopen(const char *name, const char *mode) freopen(), FILE * fopen(const char *name, const char *mode, FILE *stream) 1 fclose(), int fclose(FILE *stream) 1 clearerr(), int clearerr(FILE *stream) 1 feof(), int feof(FILE *stream) 1 ferror(), int ferror(FILE *stream) setvbuf(), (not mentioned in POSIX, but it is in ANSI-C) int setvbuf(FILE *stream, char *buf, int smode, size_t size) fflush(), int fflush(FILE *stream) rewind(), int rewind(FILE *stream) fsetpos(), (Not in POSIX, but in ANSI-C) int fsetpos(FILE *stream, const f_pos *pos) 1 fseek(), int fseek(FILE *stream, long offset, int ptrnane) ftell(), long ftell(FILE *stream) fgetpos() (Not in POSIX, but in ANSI-C), int fgetpos(FILE *stream, fpos_t *pos) 1 fread(), int fread(char *ptr, int size, int nitems, FILE *stream); 1* sfread(), (a possible non-ANSI-C stride version) 1* ifread(), (a possible non-ANSI_C iovec version) fwrite(), int fwrite(char *ptr, int size, int nitems, FILE *stream); 1* sfwrite(), (a possible non-ANSI-C stride version) 1* ifwrite(), (a possible non-ANSI_C iovec version) Level-2 IO -------------------------------------------------------- setbuf(), void setbuf(FILE *stream, char *buff) perror(), int perror(const char *err) 1 printf(), fprintf(), sprintf(), 1 scanf(), fscanf(), sscanf(). Other ANSI-C IO -------------------------------------------------------- putc(), putchar(), puts(), fputc(), fputs(), getc(), getchar(), gets(), fgetc(), fgets(), vprintf(), vfprintf(), vsprintf(), (Not in POSIX, but in ANSI-C) remove(), rename+(), tmpfile(), tmpname(), ungetc(), ****************************************************************** Remarks: Deadlock ------------------------------------------------ Something must be said about deadlock, this needs to be addressed as part of the collective communications and pt 2 tp too. Some options are No deadlock will occur if: no messages are "active" while any IO mode is enabled (too strong but safe) no messages are "active" in the IO group. no messages are "active" in the IO group, during the execution of each IO opeation. all processes in the IO group have free receive/transmit buffers during the execution of each IO operation. Some type of inquiry function might help. All processes in the group may execute a probe to find out if it is safe to call the IO operations (humm....) Remarks: Sizes of objects ---------------------------------------- Restriction on the sizes of objects generated by the different processes could be made. Possible examples: In fread()/fwrite() size and nitems must be identical in all processes. In printf()/scanf() the same format string appears in all processes. These would make implementation faster and easier (and should be true in single_io() mode IO.) It does make slightly irregular data distribution harder. (Why can't P always divide N :-) ). Remarks: Buffering ----------------------------------------------- There are two types of buffering to consider, queueing multiple output requests before sending them and buffering the response to input requests (as the normal ANSI-C buffered IO rouitines do.) Buffering works in single_io() mode, but it may not make sense in ordered_io() or unordered_io() modes. Queueing input requests could make sense, for non-blocking requests. In our distributed graphics system we have some asynchronious commands, which are not loosely synchronized. For these commnads each process gives an independent stream to the server, which it must de-mux into a single valid stream. Any calls which change "state" in the graphics server are loosely synchronized. So the server dose not have to keep state for each process, just insure a stream can be interupted from one process at a valid place before switching to another processes stream. I don't think that type of IO is needed most. Since the most heavy use of IO would be the ordered_io() / unordered_io(), and that would seem to be large size chuncks of data, buffering multiple IO requests may not be needed. For terminal IO, the usual line buffering should be OK. What I'm really trying to figure out is, if setting up things with no buffering/queueing specified has a negitive enough impact on any applications to worry about. That boils down to how applications would use IO routines. Remarks: Higher level IO ----------------------------------------- Mapping r-dimensional data onto a logical s-dimensional process array is one fundimental high level IO operation. The ordered_io() and unordered_io() calls above allow a fixed dimension data file to be operated on by a fixed dimension array. I'm not sure how often dimensions above 3 are needed. That is, how often problems of higher dimension can't be well handled by projecting into 1-3 dimensions first (for the data/process distribution). If the size of the array was changed, a single_io() read of the file could be done, disgarding items not needed. For non-parallel disks (from a host or other single source) on an MIMD machine, I don't think this posses a serious problem. For high preformance parallel disks, or disks distributed on a network, that could be unacceptable. The only way to support that type of IO would be to define the process array and the data file and let the system sort it out - as well as choose the files internal distribution. (As in Hart's system propoasal). The EXPRESS model is successful and should be considered. Remarks: Multiple process IO ------------------------------------- What if anything to say about multiple processes operating on files? limit the total number of open files? (eg System Constant 12) limit any file to be open by only one group for writing, for defined behavior? allow any number of groups to open the same file for reading? (requires the server or group to keep state for each group). Remarks: The three buffering modes in ANSI-C. --------------------- The three buffering modes are: unbuffered block buffered line buffered by default: stdout is line buffered stderr is unbuffered all others are block buffered The setbuf(FILE *stream, char *buf) call can set buffering on a newly opened stream to unbuffered (buf == NULL) or block buffered, using a user allocated buffer. he buffer buf must be of a fixed system supplied size. Useful buffering calls, not mentioned in POSIX: setbuffer(FILE *stream, char *buf, int size), allows user specified buffer sizes to be used. setlinebuf(FILE* stream) puts a stream into line buffered mode. setvbuf(FILE *stream, char *buf, int type, int size) sets any of the three modes (using type) and allows a user sized buffer to be used. setvbuf is in ANSI-C and I suggest it be included. The tie in of the buffer and sending a message could be made, implying a message is sent when the buffer is filled. Giving the user some control. I'm not sure I would like to state something like this as a requirement. Remarks: non-C Language IO ----------------------------------------- The functions above are clearly C oriented. Compatioble calls could be available in other language bindings. No attempt to specify native Fortran IO in MPI is suggested. Remarks: The varargs forms ----------------------------------------- The varags form of the buffered output family (vprintf(), vfprintf() and vsprintf() ) is in ANSI-C (it was not mentioned in POSIX). They could be useful in avoiding extra memory copies. There are not corresponding varagr buffered input versions defined in ANSI-C. I would suggest that they would not be as useful as a stride and iovec version of the fread() and fwrite() calls. I think iovev/stride versions of fread() and fwrite() should might be the first extension to the ANSI-C standard to consider including in Level-1 IO. The format for these would be compatible with whatever the pt-2-pt calls become. Remarks: Blocking, Non-Blocking and Synchronized modes ---------- Some of these calls could be used in all 3 forms. The setvbuf() call already could give a lot of control over buffering. It might be wise to only consider "extra" modes for the fread() and fwrite(), which would correspond the modes finally in pt2pt. (though if extra modes are implemented here they should be easy for the implementer to add to calls such as fprintf() and fscanf() ) Remarks: The modes in the fopen call ---------------------------- Limiting the fopen modes to read only and write only would simplify some things. Remarks: The unbuffered IO calls. ------------------------------- Although the POSIX IEEE Std 1003.1-1988 C functions do include the usual UNIX unbuffered IO functions, I think we could take the same approach ANSI-C does and only specify the buffered IO functions. Because of the fread() and fwrite() calls no functionality is lost. If the unbuffered calls were included I would suggest the calls readv() and writev() be included as well. int readv(fd, iov, iovcnt) int fd; struct iovec *iov; /* pointer to an array of iovec structures */ int iovcnt; /* number of iovec structures */ where, struct iovec { caddr_t iov_base; int iov_len; }; From owner-mpi-comm@CS.UTK.EDU Thu Feb 25 12:19:04 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25996; Thu, 25 Feb 93 12:19:04 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06338; Thu, 25 Feb 93 12:17:37 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Feb 1993 12:17:33 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06280; Thu, 25 Feb 93 12:16:46 -0500 From: lyndon@epcc.ed.ac.uk Date: Thu, 25 Feb 93 17:16:32 GMT Message-Id: <2150.9302251716@subnode.epcc.ed.ac.uk> To: mpi-comm@cs.utk.edu Subject: background information Dear MPI Colleagues I found the February meeting both enjoyable and stimulating. It seems to me that the question of process groups and communication contexts remains fairly open, and the committee has taken steps to move forward. I am sending the interface specification for the system which we have implemented in Edinburgh, called CHIMP. It doesn't contain heterogeneous support and the such, but it has a very flexible approach to process groups/contexts. We've used this extensively for both SPMD and MIMD application and library development. It's coming in three parts, first below, and it's a uuencoded compressed PostScript file. >-------------------------------Cut-Here-------------------------------< begin 644 interface.ps.Z M'YV0)4) F=(B")DW8LJTD.$"AH(2)8;(*1.&SALY.D"0L9,&SAP0-5S0R%$# MQ) WBP*9/QX!@7&],XE4B13IHW;HA4I KB2ADR((R4$0,3!@@8.73(B$LC MALT<,YQ""7.FS)R,-/)"W-OWB1PR93"""#)G3!DW9)P*>5,'C6 M;8T<-D#@H"'#*9$W8^JT>4S'"%@Z?T'L95/Q\IL64MZT">-&-M#:;FY/9H-V M-O#;2>@ ]>FT".23;5:[@?V0B)4D Z'OALSD,MFL'C6BQLJQ^O6!1^O(<9P1 M!)4R6$!0I@.GSLN8>!LRC %#!UP9.$A6QAF702$':E.404=&=)2!APMP'/A0 M" J\\!X61/CTD@PUN-73&"\A9H9[\&4((@@)$=C;"T[L(6(?(H+P@A![B'$9 M6B\Z(>,4>S@X!AI]Z/@"%GM, 8(3?2@@A(Q42+$''7+P-@=M#08I8QI3?F0& M4',4):0=@:51&HZZ,2 (C0%55)QSNB7I M'(@J"I8"8(I95*&-@I!F4$T*.N*HI5;:6Z23FMI;JF."T *KF,($ D]HO>J> M% KL!F4:>("0FAP33:?L2LV240<<&ET+ @T@]/72@96AQ:V/:(!07XC:6HMM M2=Z" "YD((% K@+G7?2RY[W9U &5& M&6?M,8=R=-0QQXJ*.6*=XF8RN34=%'@"#,,.18+\F\[)X*&#&='(TN_-K/;_1+,]+ZCCQ M2IC)*$8878(0'U-1YB&C$%')'%Y\,]91-1E#H,'4'@I$SI00CA.7F+[.^=E3 MWTFC_1B.UPILQAMOV$O'Z=/UD1/A81A>LNQVR T"&V_P9#/1]-HG8]-+;_%R MU +/,:)=0KX-Y=UKHQWUU&%4/;P9<[;Z@O&"RZA\W%M@[Y!;V&MZ!M!"/T\U M&R\7L0<<*(& M;H:E6&&=ZR_U.7JK[W\8PMWI$$&'6C8P_[(4)$P* !^4\$, M .,UASJPQ5OZ0T,+T% &EKA$@!(DH'*RA:T$GF&!W&K@ Q44P1;@ 2EFV$,, M -2G#!:0@[=[S ?+=381=DM!G')@"?. 0@RV0(-A@*$'%Y@3&[9KA3=0@ A+ M2 8\^!"(0I3A NUB1!(J;']IV$U?GOA"^-&!;&6HT-XN\T$PNB]@P&O7"X ' M/+L0ZTBR>IG_GO8".=R!CNNAXQFP9;0*C8&/:'O!$>CHO4!:3@Z8>UQB)!8& M.Q3E;'8Y$!N"8B3XO>!O@6M73F[DH!O200%&FE'BPA">=AE)C0-\87S8AZTU M0NEI VR66_9W0C.,:'\\M*6S)$A!"W[2AK1$X2[YY[\%OA&7PKS?3WYTEC)P MQ#'#[-__RK6_7K;D)1^C&= 4@"=?N< NP=0E,"68RQ%]JXA.'=P M0XV6BH:F+O0%4T@"IQKI)1"TB26748">VO &1UI$1E9P546HY2SU1(L.^*J6 MMNS2+O@9BUPVJZNVC$4L!;C@)K>C@\"B8(4^"$P*A1U4&>14%-SI+D:L-!=* M]%6&HF;5GA.[2%'F4-4[P,&=(#CJ8314H0*OD6BH^_(&!AX+U M NLD08O+#1V'0@,[P^G(3B#@4 T4L(?0&8F*!1T?_-QR-DXVJUYCL$-O0G8Z MI]F@!C? 00Q<<(/06(]718G/??.[W_X>D+IH"%.O7"8D.$BLL_J2PMJTJ:=N M!DU(0CC-AFP OO0&4@YJZF0?ZQABPQE-1G80&(BG8CC&K;B3\2FLPJRPAYZ P4G0KG(^AC(Z.H@21]UPSVXY0:>O$R#Y& 'H' 0 M!1F: QQHDX<4E"$.\L+#CVQ& P5X2\J)J3(;4. $^%#!RV"^B&$CNR7&RFJQ M73)LG?.L6,;*"S)[V,,9YL#58WGS3M]40).$1<6?3,5=(6[6BPVWYZ)(N")[ MR/ ; M7.OLRSGF>82&;S:5A!$]J1AFY!HE5MET6_6-(EYI2CR22L2NMD;9G6 M,#S=^>ENB1J?[[RSGQ5&6,H-NM!0DE)]V^ NRL +K^]""UZC$B4W&.\BS":K M65/W:IO)@0W>.:M#)GW&\7T[W-R.=+F]#6Y-G75^D_3U/37[LC#L0=L*>L/P M%KO!$9_/1=IZ 6+8L,'XN$4.^+;(RYI0)'/%C^!!?./YAMIP;$W\BGMH ;>: MD"2%D2'C9^.XP@*V$!"(7'L9M\O)7W &)YO\90%4N1SWD).5JV$/(7_9&O:P M\815Z XN1WA9\_TRJV%+XR"XP\O:L#Y?G4WI"I/JT7,"=1F]H>G=K/H+XM!T MNVA=#D4.;])?-B=L/?UE@L46M[3N1+<82>C;?ED><"YV208%6V$8ZAOFA-6. M01"$@J(HDND(10- MZ&MU3.'\(%._*"JPI#?5VDMXK$(0N!=L22#"&<7NW@I5P0UIB,/!8C][WGS\ M1>MK'RN%G>?Z H4-VGR]6T+33;N'N0S^= M>Q]M\1 7EN+52;G-U=2E'&4M\ MY'M!$Q2TDC%\!*\QVO'GW:" SOV8C#T2<[F\:,:!36= B1%H41!JY/:9E"WK M%4/!!P>&)3Q]EF>4$S87M5;,DEIR(C,P4 ,P:,R 9$(#]AH":P(3 )!@(VL$*!!"8*< ,X$4A5 M14=VD#HG]@()YE.OV!C"(B2\,3ZY. 1,@!UXU#\)\A@@8&M'4T=HD ;)V!O, M*"-300=35F-"$AP-@HTR$@8TP(TO,!5G !38^#) T"E341E1T18C!@2[Z"E[ M<(ZSIHX5Z"BZZ(LNHS! ":S%H_H6 ;TV'THAH_FF&"JHHHBR#C$:(S(8XX M)C!@LEP+"06_:(YC &X.]@(3*7:!-S)594OWHXJIDX\5LH\H%)(O<(N"8X[! M.!4"$XPKJ8]VE 8J-%WV.)/+-9/16# @AC R^8SDQ1\"Z8QID)//N),? T?Z M.$E.%(YLT"SY.#*3-'=.V6+FJ!Y-B94Q69+J095=&9,RHD7_R),'@X)_AA;[ M(C.'2 =5X&"UU7D:@Q:[U5MO"2(5T@9HT2[Z!S4:LI=?9E1=,EHG0HEX&4.G MUQ:I5YB[@0=#5"Y;%)?)0CI*=I@RX!9OE%2S@X(+=8*%28GAJ"#7J"_:&##Z M(H[DJ #Z\F-/@@?L$S@UTP9T,(1;@HIQI1')9FW+9A3#HAQN\ *?E3V:XED5 M04T8^5D E"Q,L7.W>6T(!R?+F9L)IV^&]9S8YIS5QIS,UFY$9UC,EQ-V%U#7 MF9O=>8-!09T(QXJ^XIW6QY/*5IW4UI[G&1KJ.4E>:)[,)I_DZ7KVB9[S&13/ M$G>E=I%[5P;&"21]4* :R 8@50;"N4!^4AG6J!O).2S0$CO#,FI%P7+^A =1 M(UG8A+J<3).HSRNYP:.!%QF M0!LMMQ=2)@2T,0:O-2CZ$INM&3!P@ LE!X@*78D@=F.BR %EE3L*#CXZ52 !,X<'";-A;" MTDU.&B$UV"Y/ZC1TA4/@9T/@UZ=BQZ=ZZA#M$J@G!7ZQ$E-EBI$@@*:0JHUJ M^G%LNFYPZJ%R2J<4 S)ZFA>&ZJ>>M&Z"NBENT:@C(JD= 0*/NJJ4ZGOKHZ=_ M^A*#>JB>Y*;'$D=[<*48.2=?*GSM(P,WP*EV&J:R-J:.FJ:JBG> ]JMP0"_M MDP/$VB!W*J:/1JAG$ZK"DJB;4JKC0Z@YH:UNX2VXZJVYFJR0NJSNLVF5N@?. MJJG2VFR=:BZ?.JJ$.JOENJBF>JZIFJ:MBBVOVJQQVG_8(JS3ZJE.$ZXX1*B( M"JC=JJ_"J)NH2GR;-6MBD >@$:M.(Z6Z$UGJD2=]5E]82[,]FOTIB.C$T=$FHW8QB7]N'VV M\B@C&RS8YUSL!RR9TJVSIE:#UH_6A5U0.&L+!3X*$FJQHR\>D11S8B0#IE_\ M90,'YBA((B2R!5SO@0=&P!("$[8D, 5<1093,*^GUE5U^1*+V9B9R&-WR1=C M:6LQ$AP(9[2>TE;0PAKL(V6&YIO&0P)?&;:P(2-TJY66*X*2:P8D,)7F8CR7 MV[F?^Y2A*[:8.[F&L[FIZ[GO);H?X8*.^P:02[ICT&(50K=C )6M2P)S\+JN M>[J4&S?"RY0Y!(#!TF"3.P>KJ[K"VY7T,KF@:T.MHB-T^[O"R[S'>JV3N[O9 M6RW!0K?&:T.ZFP?1Z[G:^U*3^Y4VM&C7D[/N!(Z) 3@$&K\TBQ)D8[\-5H,B MH@! H):3UY;Z8A48N3[2:[KBRP;-Z[F:N[YR(#LD.!V/.QW\29X+VJ#EPBV9 MR2W]Z2I#=U;,9[+Q9B3:>58DC&ZZ6<*I(Z!=4J!],JD,6J >''=",CIO&[.A+?SZK/YQ"8\IHBB>)J;.(GKMR.A.(H\O%F%IB/'AFJ(NQMR,K6R M*\&T2\%-XK^@N(A 48JS27"QZVP9@QIDO%QD\* 4K,9T@#&R):1P%%L*0L2C MN <3J0!V\9?!*<-C]<&ILXH'I\(V$Y'GYFY__!'@%\@HS,+U>YP7V1$+58LJ MN6B[" +\N+U% 9,'\L/("#%2A9-:.;XGU2IR<)2>3#FE7)1?.;VCC"F&I2X4 MB\FW8[K@![J,*@6&EI'%H//,OF2[T6^+)3:[RUK,#K-K$8:!+%2)$Y MH<0:/QC@JH3N,@P=1@A:,L[>U&540,S,*/03J'0/LO=[M_=[N'=_T#=\.,=_VC=_N MC=[H/:R?/=[G+5W=#0-%\-TW4 0Y< ,Q, ,^0 ,X$%0\8 02/N%S.N$4+ET8 MGN%BE>$_N$@;N$27N$B/J$#+ETMSN$][MT[CN%!_N,\/J<8;N1$CN0;KN1"?N1)/N31/ MON11#N4^+N5!+N7>K> V< -$@!=X[.+8'51" 3;^1 .B+0,XKMV??0-#<. ' M/@/J7> %'N P( 2R.!HQ8 ,-7F9$P ,W,.'29>$D/N(@WNB.#@.* M#ND7?N@2[N/\X><(7B9H^^)V4>C2)>4QH.=)+NH^+@09'@.FCN$S( 14ONJM MSNJJ_MU$[MZS/@2U?MK<3>L"KNL$WNN?[NN2/N02ON.A7NGKS=T^;NO#>NS+ M?NBA;NL2'NKJ[>Q&8.MX/ ,Q<>@R$>@O7A*%3NE(/N$[3NFB3NOEKNPF;NN? M?>SK3NOMCN[,[N-&L*+E/NSR;@3$/@2AGNSAG>[A;>NT'O /_#'+O#L3O#= M3>XD3N\"+N&V[MTRD.U&L.V"G@3;'>JRSMT\$^Z,ON]&,*ZD?MIX$;_ ?%?7Q_>NZ7O6]?O7'GO4/C_5>7_!13_ A MS^P*X.\?K_'&+NY+C^0+'A.@7?0^8/'D#NT3KNQU#_9X+_5#(%8&G^0_O]ZZ M[MZ![_#W_O#%;OCZCOB!__=:W_A[;_5=O_73KO86/OGVWO8)/O%P;_&5G_1* MW_F._^MX'/FA[_>GGOC)/OC57OBL#_B*C_BGK^&\OO7\;OHQ?_:)OOJY3^68 MK^V;O]UT;MI+?_OE7N#S?O+0?O)SVNQ\/O/++O,P_^LOW_+A??:3;^OJ/0-G M+^I)U/S,7_,5COP[+^3[GOND?NC+'MY%L.PK-/04'_='_^R2+@1*?^AXO/JT MSOXY'^_\G_].__] C^DAN@!X[F:>_UMO8D7_&<#]=P ;( /I_FXW:<;]A)N!55!-3>NJ-Y(/ #BD D-P*5GPD,@20P!9[ $DCSQ K*TX P M\/-ENS#W^PA?[OMY-I#VZ<#9QP/_WNB+?;:O]@G!'$ R&-R& QV#=JA.# M9] ,]@PSF .$P,AK@V_0#8(\.#@'C8#-*P)WT.;1/#WXV5S@S=N#?S +7L%! M* BCX.\[X29L+UM@G=6R=D M;_IO_24[4;C>OERR,X7K[0;H/U5XVMS;#-!_KS#9Q4+ U^R,0/F3<#F ^^FY M^$-[HD[[*;T :/C*1((C>ACPZ.$^M:?Q&-ZQ.WONSM^- M/;V7]\2>V#-[)FX %CMBA_3I_N"W3]$BCZN%_J^=%@0 M"^*(8XEZKB#2P\$8]NHA C2,A!$Q%L;$V/>R'Y)SC'PN_241H[@(#2+;JP%N M#R[40&6G!0DA9[2( -#=^;@_N.X4 !\DC>GM#=K!.A@'VQT<]'&OT;O%QIC@ M!F%C;92-MS$F++A6^ 53':13>4ORU&_*S?J,ISV\XZP3L"%QUSW\Q3Q?UH'F_=KOMYKD[5C$-\ED1WW\4!DVQ-6F8[;);CD*.64HA5DD'$101(X$/GIC)U8 M>8@M\N&)NA+IZWYBB.R)2/)(*DDBJ>>,7)#\'@^Q Y*Z&V?X8.+;2X<-TN'M MPY#7XKK;)@1VZA$H"KOOX?P2'H@T?N[Q['U'_H!$PIRF@W_"D"YZ1YF7_IH? MD=-P,^_?I3\)Q_X&()?DACQQYAG)0=G\"*6@]'Y0LOF%0WU7[WJ>BN1^<:Z] MP<2]:/34W^X[?%QR_27#9H"O%(;';BL81 M3@:_AX?^6"2]BV^>,"I&0VF9#:>EX,.!N*_8)3DA,.2B/S(Y(["T?NV&2*"W.KTNCEP)E9%_G>KAR( MP;)7XDQ@R3._A\X4ECTST@G-"R?TSN'[XP\!HM!-O"'P CL@W;-W(2^_9#\\ MYQB'5=4TF)$0:UK-?KBAP"9A,/XKG,MD) 6[V< M.YA^\-7MXXBZ+U,6PVY8**]AM;R M9)%2:!Q)'Y4BWB MQ=PWY(:G=ZL+L*YTRLSCER8K(Y\DEGH.";9++#W*!+?P=/B.:3H\2HS=/UU( H< 5"P")')( M?KP65_2I+[+D]O6<"I(MI;P!:N *:)[UG WV@=1*">K<[61J1 MW0?T;T@NOZ2_RBCODB+;*Q.J<.*U(J.'"+.NB7Y+"=L)IX]=(^SK^O2"&K(R#\HJ>QW^%!$Y<+A5],L(.V MT8P2 3]W"=FH=UNCE="-_L8=]R.!8T:THN23A +10Z?][)[\^XWED5]"T>HW M1:G?/W1R!'*0W@!^V43/XA!5A/"Q N)1+,H7@>@&S(6,$.$=R/0W'/DCL&ND MC31$ <@<,$IYWX!4$D-1YJE2'1G]F"@I)7)"H#SB/D*JZAA?R?)SE-3HB<,J M>/?FGO-;?DL1V3DY?CGENJ.JTW)4;F!>.:C'[,S>W;-P3Q/_1;PKZD//',R( M##6@)*R0-K?=KF+M$WY[3KJ<3!&X/=,?5#R78O)W KNP*5W.J614=:J.[?V\ MGW=.P2FO>WA!( 80@1L0!&Y";FMP#J'043EZ"D]'HQ <X)3%LA056 ( MQ'7OM)W^SL!' C'<.?UY0W%5[M,AX$]I@+)TJ.ONQN$ M!FBI"2^,XV,;APJ M/7&8YS1@216IZ@ZDACO^P!4CGD>5>P"SVB%.6Z?O6&>O>W@<)@C$(IE0%SPJ M#3B?%^_O\'54ZM4=J5]. M:+43ATRSK][5O;I(EY\7+:S<;;#R3X='X=AD"'VK-Q7XV<]@YPH+G&#MJS[1 M)][/4,E.T:EGO9^9]=.MNYYZ+5^@^/N-JA#!M:+("B\?8G76W02DC99RN8O,$GD#F*EUOJW=]@2F5:!97$?I/,^"GS*N)U;"B5Q2( M0?U;I1.'8M-K-M05*%-SGWV=H#$@M;K5X]HJ0QZ'&7:5];O95V%CD8W%LDP6R3O;']KJM.:STW/4KD,\QB:C* M(_LH'V5=#:S_+6#*U_06WG) ]GNE ;*/5K]3.37;;'S-FJ^5=,=EV.*W+9^]@932,6#(-J$ WV M1DCK!7M&I66#UUMQG+^5YL MASV3&M+4*4-9ZT2A(:T=@V2VS.;:$+5K2:DR]+4G#]CVN_7>VJ7;70$EKT"F']JOJ\FWUV M!')70:M2B5VBS;,T5M@AO?_V\+XLBXV*7O2W-MS@6OUP'WQ5?B-0USG;I4VM75S$A(3RU["_B/=;CNMD@'*0; HSRK[9+#A/RWJ=\([8X M5ADJO1]9[^2I=OQTF[7<-DFT*6<1W?]$K8HVPRH\SY<,7>&LG+!--[ 2.%?H M75UX$'LAS)VT:+:07@:\V9N+;5GU]2JW;1K\\SN'\R/ M2-3G&5/BF%]!&Y=EE!4.[WI8.FHMQ^JU3'VO#_!Z52K7544KX76*6'6J\KCT MEP#KI%/UCD]UJ3I.(TR](ZCCNC]39]K,8%D0DZ^EJ[:^5QV: MU&^KW]Y;J 6!G'2B_LJ.*V73Z=8;K.O6P?;+&(@%X^V\M7?V;LAUU@1K@#TK M!;; ZO0"JT<%F8'5*00>AU]PG]Z$5:EGL6!#U, =.$&BX J\@C$P"^[ 'UB] MB940K.U(L _ "Q#NCZ*].SG@3"*XO)G#4R-R26:9!)G@U]-UPXI=5MS5&6-I M:A'(<_]T1BK-I A_=V63J\+9D@B,PWW'_K(?OWMWBC$,2[V2&0^?G!FULI>P M+A)='Z=R[6Z&S9*9-_&M.^QZ3CEKJ$RHSE#PF>!3"4"YVQ.VA9$U^"I?5HH@ MTZ.!A',*-]_Q2--FY&@GD7S$KQ,2PTYD9XC)&W#ED=).]Y5-@*=R+6^KO)&5 MCNHRXDU(;,WM_A5WG14/4T,]+.GX\&_TPU X$$_/08C?/AL2G*^4;O_FS4$; M<'OQG\VVA,\6KMPW7#;E';V[>6GV6_(Y)+SUV.W)4\;]3@121B4L0>=DMO24 M#7<8!M,4BSF?X?>3G.30_GHW9!F+X9^X-(A5.&=6X0&G3[?PQX.*/VY= L ? M*(_[WP,$C?9XYOU%$YSGVANV6[5'EKP=Q0Q:43TF75R.OI@7_]E!RT+!*]HD M?)9552HX67Q.Z:9%-.+2;FIA?@C>GEP]AKOY^"O? MXWI,CW??5^S'5-DJ V(XR2ZU9=3,FO"3VYKB=#N7'>R\S)UO65O.8EL(B]&1X,2IFFO?CSC!(U,*(+UNF8YYYA0WB+#V([0\' MJ&7O:_P2W?K#??CS ^[/H&@066GWS)' DQ 3O]5\?S5>.3:?,6Z[_636.>)H M[D@=MOWN_*E)8V(5U"CFR9V_?ECI%F5 M_;*&3:O?EZS*9X^(??EJ]36\%A6<2M_%RY_;KW]FJDFN3+3#JTQO>_/S_(/< M>,6666(K2HD=*35\M'?VV@##%P0F=).KT+B7&E/? OM!::K95+HG-/A^Y^*[ MH?I6. M/$X_S\*35R>5'-\#=>]7'7=7 ><4U2_S/75EXG 2Z%T:BR9<:(W+%U0 G\ MS.Q(7BT\M&S2-?OE]@,VBH)^V0*F#23( !7TX-8T@# ;=@+*R("<$!-@ IIV M"V\Z31]5F""G:0:4,6T*X$W' !54$M+TG+;3'*(MH&D1-:AGP-E(TX.Z+L#I M\+(%.H39H!DB2E)+:A82J>V"4[73(>I1$.KP4A(2M:DNU:B:5*OJ1#VH75%^ MW18U8 O4F*.J #I$0^!P,L(X>(7@T *& UJ0 V/("!P/D% 2PG81DQ'A2G-Y;#*,@S&1@(G'[)<^)T2 +G<<>M!;$?%G!0L;]8U7:Z(84? M:9U33!- X[$WRA6_8 WP 37@?%I3(S ^KALW_:\[#RHB01H YWSJ$(#7#Z[0 M<;GJ>>H&=@(F 9B: M4K.Y\"*B/G7,A@DMFS2 :OE4J1_<#%!!9\/!@6HZO064-FG 8[%Z+O#LJ3T2 M9+7K.!N:S04XZJT=&EZ KK8-O1I1_.I@793.]*&.UWV;?;9K! M+6KVMF#3H-I.Y #&C9B.JM6> >.E+K@ &\"Y.?>#\]J_859N#01(. MAJ>U:9O."W'[:+?M5FV[(?>;%MIIVG&;ZKQMM&=VH7[4C1MF%^K[9KL;=Z'. MV];;;.1M[2V^61'-P--K.WV'%_6MM].WVE[?JEHH+6\\3;F7=TX(459[D6Z+ M_=*Y/7>N#MU@.S=LAXP6K.N)V6[=;MIPY^[:+;<#':(NW*KZ%<*$YKV\RXSN MQMUKVW@C:K]]M!,UI#[<39MP4VH'CKG-QNQNU-M[:9MOH82S]_;KAM]Z^WQO MBY9=8T*UYN[?-N!S?VU>[:L).-@("@MN<@LK$VXV##<-* E.6P5Q,,B-!$%U M[P[ABIK#@(1*[1#6M"RZU&_:!F3J2LVH@S;3%MIPVRZH.:M=IUFGUM[:GQIT M4XD +J'8S^F&;))-DKR!,T"(W,#DN19.02U4!/5 %E0 TE(NPH\,F(Q(9-G6 MCUX@)8EA"JB*C! $RHSBP5L* H\/!G>2$?PV'K,91 5;W !M6L(-2!P: HH< M,5"P/4 "D@ 9R @$JI^X@(F1!UB H]@O+/LNS( 7 -D "V_VS5 !WPV'2"T MQ8 :< -WI QA"Q+0!U"0 O 57PYHBZ@@ @2&F$4#X@O.JQBON7 V^(4"J]5- M' AHI0'C5;Y2#G<+;:)_>)7[\='J^&E(#=*!\J3R,2 &[@#N. -OP 40*!$4 M$ 9"03@("0%/- 33@!I4 VM00+$!"2P6L^(3# A$@ I2@2P4 2@ @&$SZWIOD?2$V?!2(,!G"!1G1 M#)4\FE-SD-#$!\PCUPZL@9X_!*' T(? 7; +[SQUH Q8 +N0 J0%W"@G[3S MK"[//<)#4 58G[LD;V3 !) 32)M,*"8]Z\7L'=F35OG.=NB#R2 _E4A)@)B MN.MN0A [&&+CK.2&N[W7Y42$*.X&ZZYYM6]R P<[8Z\A9..QN/9DK M=L*>*-8"4W#K_/HM4'8ST-B9PAUHZS9@L9/V2\(4T$!;U\RJ_2'XIO,!V6^ M"TC4FYVQEX 7<#Y.>P*P 2*A!L3VV1X]V$!MU]HE(;>O]MZ.VI'[<#\?KCT! M8.[8+D:H6!7S% D@OU#WZV'=R8L"2 O0*"S 8+^$Q+ Z(D:/"@!S 'K_H_4 M^ZRI'ZCI@##V"D$;Y$!?\.[@7;R3]S!@WF\$?$\ ]3VHV_5_!-^3!%^_))V] M&DT9_,Y9>$M\/_!^7<$G!OR>E'X$4W!A!GZ^"YPH<0<<^]^0 _A=%]&!!$ N M8CMXOP-D@,3+'^J> %!\ H !":"0Y9OO#N-=0HQ78=\=Q9>;%W_CT M%*(3:BL#7SW"@Z8)OS=22QC1\L,IHUUVJ6814-L8"/-G0<*#^#*O@]#Z43M. MCMV?>'D4\.83@&.7\TH$2IAY.U_/T( "<.S88L_#@:_^Y\<\B(_IQ9W-/Z+> M7C"*^_SA\!Z>*1AY^H3F\7QG__"2_M";=A31VC4"IO_TFOZJH?508WC.? FX M\R5%J>/@Z+0MAKA3Y19DX-0?#%!/Y\]\@Q=.]_T/089S[L^GPS''9_[BF0-M MDFT37H=7<>9>#)J#<^,5"Z4Y,G-P#D&;$Z\;D,V_TB*%,N&\*)VC.?;,S7D_ M3^>HO Z,@5\#(>BY0" (!@$A*(09T!"H^A0P BV WCL$B, 5-+IB, (3H0P@ M =\# E %4 (AJ,)7(25<'Y> E" \W["VA@,$'$64(3A4 )O WT!LR1T$'\ M08\*4R$CK/M[ A.^29GA]QG=5JQTLG #7@ .> &K$B;$ !V 7X9%&Y ,SN9S M9 :[D=39R[#:%G/J431Q%+ 4/D)E"B^L$R0DP M&-))5%@45@8QG(^;P$-6=/]Z7_M%?+,B!>D)4P@538!GR01JI"3;0 OP$[E /6X7@N+8R M0/Q;O\Y'$7;#NO(2V'^^7\5W&9A_EHSN-_,9!0K"&- &Y %K 'H-!P5: &E M M@;"$2!9A@JU 1B" N=\W,0-D!%8 XV^Q_ M6@(-(@),?6>? ,<;B !%H.=A+H@:K 'A5R%,,5:&].&$98=*8@60'\C"S&(]PD944%18"D$$9\@'/,W""FA1LY7(2@%9LS]YS3P M+])@P7 'K#"]'_[7 @B!E0J*$*2L 6G=_$< XG^X7CEH!LQ_!HTX:/<9!:I" M"S 1+'E4RH$0^24M$9^@)R?("R[ &=#_7036PD10%A03Y4+W%^ U%@\=YV(, M*AA%00\(_:D0@?Z""J%]6J!5NA:>?:>0"S 1.%0YV/\R M4P:X01;N@TY %G@&?@U;0!= H5@:1\"2H&8<#!\!"@ *#=IASK MIB%BB,?A?'@@%G$/SA'P(HJ!)>(]&&^@B 7#,_(%%@5_ WW2(M8'D-^CT1YF M-B7?=N@D;@=,HFOH8QXJ]&']N$,P"UD 8L%[G 'O "&03!R)0:)18&6F _Z?C>*IK 2 MNA-BHAM %$P2O(69:!ZZB:PAM[B%PB )A:R MD M0%!WS(6$G&)(^/Q)?P0?9W$R$ =]XF!X%MQWA=_H!\9X!ZJ# P@!2H \8F+0 M!GP$2(&[< :P!3$%I2*#: J>( 2#U[P$: ;D-Q%,&>9?#T('N(H18._G- @9 MB4%1$-!LBM+?'G#6-( /8+*(.X!XG:*T"#\()5\&"&"S7("WH@28)'2*V>+_ ML2T("\Y'7D NKHO>8@WR>KP1X$< 0G!XBY%%7.=ZC B0A 7#+I*+Q=Q TQ5 MB_LB^@#=D+EE0LEHW=0U*@.VHD:D,6 *;A95!/ M*!,%8Z^'FAQX!X$0< 6,C(R-5##E080LH\I(2@2-FH++^#68AFN 87 %/"/4 M"LUX+>(83$T8(894C)^%E$$Y 'EI7A031CB-%\$R4C4H**P>AG(S:G@CW\20 M5-@ N%[GP=>5C6T,$J,B5(Q>XE30)82)E,/S%S=N%AW#1[ .5HW301#($ Z& M=@$*P 0@ 2E 32 #U 0T 4A0$YP-+$"9L1 =U &:'#;A7H+@6U7$I $Y\1" M0 -,CI%0O;<0T'M0QDP@$C NVP8WP>9PCINCB((GV'9/1@Z0'-X,(4%.H I] MA:3CC(()!<:C0.^ICF7"Z+A(_16U8YO'.YX39P+MB">\AE#& M\)@3+(_U'K>WN16/C\A"\!J**)H9?W$[^'B:X^28 VR.;L'C^#HF.)ECS*8] M2HXJBWI0.BZ-U:!A0"4FB2 #)_A)P'IH'9Z XR.THKVZ.79C[1C<_@ZKHZ< MHVWWX. )(0'V:-MI,.JC[9C9$(^B8TEP/(8&9@N4<3RZ!7,*[4C3&(_\P>LH MBVB/OL+]*#L"=W:!KV#;Y8XA 91Q03XBR!SMN,E-CB(D\;A!AI MP/WXX%20 M#\X)R4UDD!0D0_"Q!(]VP6A /!Z//21PESY2&0KD3"@:N(XY 8Z')]![$*3K MZ$ N.--C_CA YC;XXWZ!0!*1&.3F^."L2MKC[3@^;HX2)!6Y1<:.U.,-"=RY M!=_C_YA?T8XXP>C(/W(+^:,)F4"V>>QC@^ ^5HG7BOS8Z1EZQYSY&!I@C@^. MEQ=>') @0!H9&H07FV-.(#KF!'/;](@#%)!EI$W@2#($#PZ?=DZ8+2'D BD[ MVG9#9.DXP6F0D*3,IT)NCIDD+"?TG1,VI"@I.^I[T9_K>#8XC]'?_9@[XHZ1 MI&@P/4J/A^0*$3T*D="?+MDZ3HZI)!19$FR0C.0:.4,R!%"&)7E,:H^I9&4R M.F:/W *.M[X]DZZCB!)>, 1_I.X'#=)[E^-^$$ER"%$D+1E> '>7(W#'+8R3 M461(4$/.D8]('5D&W)'PXR8H9>R1:)TE^4>^AH&D'W/,?3?UGO^82YJ3H@$G MV4N&!!0D<+=.8I$HY ))08:2&>0,^1H6DP_EH^ Z7I&>9"&I0GJ0T9_E^"CP MDM"?0?DHJ).UY!GY4;*0CP+RB$3&DOWD]5A+GI2;S>O8%FAMH@%!V4MBB*(! M0!E,[AJ2.>7T*!.RD2HE5!A%FI+!)/A82.*4O22]MT5JD;VD48DY M4I0S'DQ 1AJ2O23K6$@:D?9D(8E/EI'N)#QI),PM!).C[H@FZ),W MPRUY/YZ39.4,:45"?VVE: Z1I)Q9>=X2+8 S&,AF3K"E9LC'+E7II(M0$CP M1X*1<.5:>53ZE"U -7E8@A/LY VY5?J/)4$'>4F&DK^"(YE(TI4CI&A 2<*5 MD*-FR4*V "(E#4E7A@3#I!,)5\Z5*Z0U259^D,!D8]E5R@'OXU?)-\Z/A5X] M&5X@CR"E(%GR<0OY1;W72TZ2OF7)!T%>EJFD<"D:8)2^PFMH%Z21E:39$$Y: MD(TE],=44I707V;)14:6%N5R:=%%DCCD$3DZ5I?/I$#943*0@F5(R4!>EM ? M)-E2AI @ >>85Q:73.4&>4XZE\,D:1G]>9,KY$&Y/C*-[R1LB4?&C_,D_4@2 MPHU"8D)7)%H-H]_=2.K1)\_?$V ?:">< APP9(01!.3HB-W0COOC LDM:''G MA&0IHDB/C"1>Z4(2E]%E26DFH)=09&C 86:23"0KLED&DRTFR^92FI@@P7X1 M8JJ4F%%KB5".E?>CB))C/H\:IM*G1H8$&^:+Z5KREX:!CNC)@)6?!'@QLB * MW@$J^(RT!.#&-;$I(I47IEV@6VZ6%R9VR66RDM"?4LF*T'N=9%3I7#XXK0CM M" U.CQ>F-4E?LB+O98G98G(8)"5#<#9DF=$?ENE-VI QHQM03\R61P+ 804F9%\",$@>N@?\&(P'-N@)4Q1@ ,(@CT*#O:F*&> MK\!47IB,I"WYX'@VQ"/T)T."!+\C=ZE-RIC1I(^77+*9#$&(J4..F;MC!KEI MZIBE8Y_Y9^J1X(5=(#)&F6,E_W9*GI6>I(@Y3$:40N5U\QKFDF0FG#D]PIBT MXXTS.4J3MJ2(OX9'":U.:[ MF6:&, IG"7DSA)S,9&-)D+1DFDEL1I65I8V#7IZ9ON.,*68. MCS1AB;E5#H\B2KZ)3+(BBF4+:6WNE5ZDBLEI\IFXAY\I;AJ:9YRN0D?>F MEHE6ZG-0QKUY3@J3K,A^84UBG6FFB )M7C?E);798SJ;D65@.3E&G7 EU+EG MFIQ)YZXY3S*=4*:Y.6!^@XAF,W$^S FCG_)7W.F-6<-Z=WQL@HNF7W!EGIVT MY)9Y=LJ4:^:^:5^&F8=GR;EC,I[[YLQYW!!KV"9#X&,VF\4E*WEO-I=(9;SF M4Y9\D>>SR7;*C$MGRMETUI-FIUV0M4F=Q28I"1+@J:I^=8 M65YR;M-CP]ER6I6>)INI4=:>*J9M)T<^GOD>F\E1:AS1)B%9>UJ;9"1@*6:B MC6IFZLF* '=WY?0)?Y:1)^?;:7K&G6=>PCDY@IJ48_!XW?R;5G\8>W$EH M;HJ&9=96$NB6$^7K66I:E,%G"?I2IJ"BY^.900:?R*5%N6^>E"_E[HE*1I>V MII"93%:;)BAV&8+"GO2G\UEPWI\?J*D9;?*2).@1RD%:E&BF>-E0LB*$9#"Y M55*?M*=%.7':=IEDFIAN$H]2Z.O(?F*7)R7U&6J>FF)H$$IZ#J'0)_YI6W*: M"@Z&B5;>EK(C!IIX*BU+:+.9/$*@CZ=S>6MZF=$C-GE?NI!::.89/6J'>:8* MB53^EK>F?@EN"J$H9QKZ@2Z1V.:%&1IDF'TEK1D^4I$+Z ^Y0$Z;<>6AB"0G= M;G 94 YBI=U8=X*):,%\0&CF>\MG(>DZLIX*) 7*82:1KJ9%N7P4H]*D4\5& MGII*Y&=S3MP$YP2.QX$VF41HE'DWSHTF#2JH8$X,\L18Z8J:#1AEPSEK\FO3 MIJM960:?R*;9&7R.GT2FD(E[HI#;PO18?&Z/ALRF6%Z.!"2E \J.AIH"J4!9 M7 J;UB@SVH]NH^/FDTEHVJ)>8$)7-R*($X.S #A\B<)HE$D_PI41CQNIZ>B/ M;^A"\%;Z"N0CE$%MZI]LI>1HDC:C*21;64&NI/UHGMEB=I!Y);6959*DA*5- MZD?"E9@C3&IFHI,A3*"C/=JD[650ZGWRHS^I<[E#0I$])TOJ7-Z5V*A/FGM& M?S-I8NDZ[J!D94IZE49_6>G- .(G,5E>N*19I!8Z:S).W:6'.E8VEDBG>(A M@-F!"@L.*;!9&\(+$>FCD9&:FQPI< <\0I$,9TAJ/JJ6::14BH\"D(#I7'F2 M!H^JY>PHF *,Q:@(>93:I% D#)D?+*;WI3:)DPZE+&E'&J;PI4CI5,IX+J6& MJ5/*@'JEER516I7*!"2E/@>8HJ5IY6KJ>%:4K2E/RII6EJXI5DI(+H^]8UA* MFWJE(B=<:=OMI7.E;ZJ)9KG)V"DH ,:G\5V"#V9B<-/= M!#* ?=I9X*=A(["!"GJG[D(8$-S,%B((1VI8(G-YI6ZY$%RHOX+(>9+* )6H M"+F5ZJ29I8B*F<:DT* LR5G6I"QI\'A7TIF4:2\Y?R)SCZEHBEA^.2=J4JJB M8J@YJF<:0WZ7H"6+BJ("GYQEC8JBX@D )8W:HPZFP2-;":-VIH\(0ZHGT)31 MY_,W.#H+T4*<6!M8"B+ /W@RF $BP*;XB7*6O:.&BG4B<[WC8>$1C,@YJE^JE)R]!1RL D@2JX M,:B*J> D6[E"]J45Y><(I"JF4"JT"$[>E:RJ3EI>(G,C:F9:E5*?C>KI"*MZ MJ#Y>>$BG IH"IJ&I(GH%1,;H]P+^&'%BQ3>L[G]3W520%*R#!)^;X!),$ 1? M@5(7X@%?7:UX&?B)G,(SLEA(&YEB45 AX 'A8_XHHLQU>$ (R47ZFO.=27@Z MX'_!8I>@"%H*$\SE@@)T&=MI_D>L9C+P0L@"+S@-IZ+@^0+D 6>#3*AQ&G,# M:ZB)7[R. $.2FL5:99<'7*(@*L/:KBXC?")BL!+T,/9$"\ :5!LO08[B M%5"&12

FK4MK MV^JW$G9V0.#:M*:M,VNK@+4>KE%KXAJT>JV-J[ZPJU0*JMB%QRM M@HS-B+XBHV>%\CK>,:\HC/SZOM*OW,;."KBVKS:C A :D*\,BOF*OKZ<]>O. M>K\"L()>G5>S"JZTGNWZO\:O#6P8@;ZZ!;0>\(JUXJ_AS /KN^:O%6RA>> 1 M%:G%1'H)-A60('#A$_@%Q!\=<$DP""C!);%I6 1M0$V0!M!U&<&VFAC0=?9! M8D#\L0%UQ!J1$:RL=80OL49\K,R@5) Z@!Q)*K:3' JM/J.WF.\1HF9#-0G% M^A/M(FMH30HK_ 44NQ*TBTXL[?D*_15>[!)K-F"A*6I#<+7B#E*L-/JQS 5/ M+,*:A!0E&0<<"SPZD5?K@0#&@I#I28]YM8HB6>R[*3[B 'DL'0NG=9'.) TP M=XJGR"G=-Q:,?B1L/7&EF,6;(?F=%(T&[!*,Y"I<1!):( MEJIA& 80?O2C$AOJJ;)X10UQ4N 5.!YY&J*,CF=#+.MTU*,?;#\JRY:@)0>. MEX2D#JULU(I7F(N^+!M+S$*OQND\:1(*-LDI*AC)Z@1(0+=:R=X!]*2@M\KJ ML<'L&P'+^G@F(1)T3M2RW*S9P,?FLC->-RO(@K/]:&\#S,H?-8@VR\X>L^FL M'@O/+J+)K+&WS-H,OJ8:"LFBL//!8'B7AA&I+!N[/XZQ\R4JZG3HI:Z73X8>>E@VM-%J%LH8PI$5;2[J>!VUPV5YFM!.M=-EA.K2S M1&.ISK:JUFPT2D[>EQ>JV6!*AIF$Y+)02_ZF5*EX^5NREP#"ZW@LUI*6*=+G M4D*/\R7F1CM&L9ZH$[N#SK*T9[X7T8IOQ*,J:T.^EH^BG%AZ(J=5*M[WS$H! MT:P\@REJBMOE6%@-GAMG86]3E'AY)RV.!YZD#BKMC)G3!K4O[1D[.LJT-J1) MB#4>)U!MG%@H#J%4K1JJ5-XX6EP-<2]\M69A5.#,HK ZK-/PSP84X$:H-\>V MJ@M!T&D]!I<.IVH)13XX&6!14ERBL:VHFGG(JI;.9$D@UI:1,FTTVM-:HQ>M M. FKU7L[+6"I@LYPOF5C>X)ZLVX!47MDMH^P): XU8H=5:VE*,E&LWZ"'. G M'GH"K3ZYV.)X>H(1F=M>M;SM1?#;GK1D[1=+W-H,QJT^^_P]LSL*]+$&Q'V? M25(@.< +[.IJA[)NMVA!-W'@5;=""N6@BQ!\>PAI)YZFLM=)W=?'E(YD*?0' MWKH*;0AK2*<)E/.ML4!^)I/.))21W[8AT&-)NV/RDXSD?QL45); &_%HX*:0 M5>C]Z-\2?,9"]JF(YIZ](PBPX,:5DJJ%"^$"N&3DJYD3++C0W[M9G"J=:"AK MB..HH0VJ9C$VDG;=XR&)3CXX*^M\NXPD@O6$E1K-6@3/RD(0$K"73VB,2_#- MN-9M?8CW7:EY*ADP%9B.L^K3.6W*#P#$CTO>UKA#;C3;LM*P?213F9_2CM\M MP??D"KG0;,G:1X8$J.L3FN7""UON\X?5N@K^@Y&KG(*3S:79Z9WX$O-MF8OW MG;E3KFXPV%Z*6D:4^5O.ESDM!ON&CJ0^;14J8N*TV^>K2>A"E^"G%MI+WH^" M;E[)DV*;9.5=.::H21N(TIN2K=6+0I[YFJU 6N%ZJAAFS5;O2=(NHFX[N099B*> M+BVN"^PZL08H"IGKSI?.917*ZTJZ]6B8*4[JN*-C*IGK,KNC8S0ZT4*[0:UI M6U*&FK=N4#MYCKANI[%G9F@T(JR&)^?!'W*?9Z ]YHP[8Y3)7."X4B64$2/4 MAT!;=ULA2 $MZ\WP2'JMCIZQ4>3]IPV"G)HTR@'N9*NH+%8()PD.D=ZFNZ$> M%))>G(R:BL^Z;B2\?LLUOSJKPD[]%1IZ&\4@#2F^\ED38O"C %2+U[&LH+ M]#:]3^?FR/.&O!POR]L^@)9?;]'K%W*]OP*,:_.B-D_ RIOO=JCX9M5K!+B] M[,JAJO2B=2DOW?L2Y'L]9]5+!.B]Z4G9Z_-NO?ANW1O-H;PXKTZ0[X:X=R\* M( 0 OE)EXQOS*K[LRJ0&]4*^:D[C>P1 OE(G+KO[@9JYT:SJ>V6VH,CKIO+2JL>JO$@0DK*GZK\D$/"["&JU*?93@" M?A91PFHP98 (@!&* +4!!1@+EAI'@AW1(E(5!P&?)P"\!&^A-TBY9!8&I;; MY1O!CM".9*L^.?^.CO4O4-%!8I1]*V.+FMIN]=X;P3M>E@'PKQ!1]K_A[4=J M'CXX!K"#,SI&E 9PE#8Y;I7'1 1:ZKZGYT0%W)'2::F'%WD<)L!EPHXK+-B_ M(J:;V0/>CAAH -Q,>I/OI@W!.W*: 3"I]CK&P"=%POENUL!JSO084TFP0%D#.\%-8M3:04K!J4C^NY0:JL-* MK2F4AH\2\%**49ZBO&S[\UE2D7NI"?QJ H^D@0:*0DZZFRFG"0W:ESIPA]D# M!Z5[<'1I!%>@'&4AF5GRCN(E%7P%&\+'G"8<7=".83"2*0=$M7>M'@D"XWIG M\+V(9[AZ>-_T2RH6MELMA6FNMI*I9AMLZ9IQH8$2.0OSCG8F,@?8JJJ+RC":Q_[J7QJ""[&[VF4:50&L+ !2SDZ6B>T^4%3NI#.J=6@N&F+]K.20&9M]TP!D("Z'='A!*&@L+S@![4N#$Q5Q]L>+:>45+4""=]@;= MXBA(W%1\6[&\$&7!.IX,8^GFY<&X?"M"W2*&X& MJ_U+XN%'_!KF+4BX\*8DD: ;,G^TJWC?KE@4H(B!(NG)+H*#0&(*2]&)Q=YB M<+CD%7/EA77X#DJ,JS#"BA CO^UP2.@5_W[^H7:L7,2_OFHU"![KFL4<]+?, MS'4'L=*I'CL37_$SZ!Y;"O!QVUD?=\?L<)6@"OL9!]YKJ1"'A-AQI?@?<\5@ M1GCLUPW(X7%;2AXCR!?KX8>+/AK@*.?R6L:6CP;ZIR4"B6%Q@(PD6HF&17QL M(-_'%^L0T!2"@:#@&M -A@L50GX\\"[(JNLYF!V7R&#&B%PI\LCRP4Y\, [) M&J&$?!90R#;RA1PX\'4FH=VR(C*#Y_'TVBGZR.+@@\P=A\>.77T<'LN, S*+ M/.0UR= O6C 9L+\LHA@LU8K(\N"/W!P'R(.BG)@B%\COQ(&\)&?(OZ@0<'S4 MR**PDJF,](!4,I\()(?'K-YC,!_#R4IR]G>Q3@:7(F$@L'[(MBV?G"97R7_R MF @H#LH6LIQL*'_))8!I>)'4 73(:Y 1- %9'^3']I5]5!_:QQOL@T$ 9F Q M/,@ZP.CW I@!=H$"DB%,!"#"16 XB!^A\M;7,.@6J\7H%RU^#.#Q;K#^>8LH M0/TP!^#*:Q_7]]6A !*BU[BGCC/R0]4 ,K0/P#*.80>>!;[RP2A 3 6SA80R M+%<:$F.AG"UOA,)R/2$P@,K)\JY\WI4>K_)K(/LARZ(RUV>6T!E<0IR<63PI MV/*GR"ENRQ0!<.$MUQ/5,I'B+0;,$V.7,"Y[BQ:"V5?UV8+2B']'@[3+?:,' M0@LNS%??BW O@HPFH8X8!L@31<$5@#J@!5MQJ_PJQ\JOP:P,?> 3MS*Z'"_O MRG,)^_$K]R3!\H2R!Q3+K\&QK#+KRF' LMPLPQO3,;3,,VLJU?*]? M$\RK<-.<,)?*#'/4C-Y1,!"S5R Q*\RF,J- A%S,3#+>=^YB!FZ#7>"RF<2? M6M"@5N@@AZ /PH,$:OE!"9=-L8XE@2,;\+*-ZL?;6.1!?V'Q>MPE2!L1&"FSG4G3@P[;\AVY]X),'JG ./SAQB( 77 &: BC _:24:@-<(AM0*A8KS?:>@,%.RK &,DFSM'QVM![_P[(Z7"\]Z2-4R1F,CR&-NV MMW;*6/+,,L^;(OG,^"Z_YRK^R>6.SJ6B8?M@1IGD;N+1V&K&#N3@W! 4SN!S MP#MWFH10@$<7_UF#(X(Q6!,TR.)@3> XGP4U09^L'DQU]T-H@2UPOQ*J@G & M=G\B -N("SH:"LH:\6N$QPC?L@P=<3:!E1SRA[ M2-#WMLRE!6O&$( $) &@<@H0MXD!1F\*D%BB""A )*A&VP5L]''C9?R\H6%S M ^9M*@!)W,*4"; T5Y!0J<#I "2&EM@!*@!*( 9C49# M 8:T6\!&+P%4 "6]R9F\9<8?'4AST71 "H &H ")=,C'2-/1:0$D+4FGT4() M&WWT!I9PM!Q=)AQZ*( =;0;@T !(1K(TP' 2G +2T$@!B.00J &='29@1'!WQ3 PI@(C"1I:GE 4NU1 M,]6 0TC]5&_3)35JU0U'1U6 Q!*=8#*&S0,#TP*< :$U7B!(1W-L=$Z M-5;M44L*' ືU1##KG8&.-:HS5H=5V/4X/1M9UC])L!2[/1T(?&5]"Q 94T+7T26!NHB!825V?71]S[FD\' < % M&J >"--OQIHA(;X!<;17MV8 V()C"N &W*N0 5@ 6P\!]9T.0E_3<6Q&!8%@ M[]>@X&E0:1('FS47D/)6=5Q X2A,)P&\P1WM%ZX9SL9G_5C;V#, ?CU*=P=D MX5F08TO4>@A*<,CU!1E!WA:H"0N9W$=YQG+7C_2-#7,YTYET-#U)4]/6-#:= M FC3W+3*.UG3U;]U518@U-$2M94=K_%IV$X-XBADUU-0$&$$C-(+]AD"&XS3 MI1IH$*F]@*HW5@ M4&:PT;^U!.?@0!E_-"2]'S#2!+75FV:LL)(U'!T$]!-^05S]6$,!/$U)]89]KV* M::8 J1N]]FN[VP*WM:UO']S"M@)0<-_; [?!77++VQZWR9URH]S%-FJC-------------------------------Cut-Here-------------------------------< ---------------------------------------------------------------------- / e||) Lyndon J Clarke Edinburgh Parallel Computing Centre e||) \ \ c||c Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk c||c / ---------------------------------------------------------------------- From owner-mpi-comm@CS.UTK.EDU Thu Feb 25 12:19:51 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26012; Thu, 25 Feb 93 12:19:51 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06370; Thu, 25 Feb 93 12:18:21 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Feb 1993 12:18:20 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06357; Thu, 25 Feb 93 12:18:10 -0500 From: lyndon@epcc.ed.ac.uk Date: Thu, 25 Feb 93 17:17:58 GMT Message-Id: <2160.9302251717@subnode.epcc.ed.ac.uk> To: mpi-comm@cs.utk.edu Subject: third part of document >-------------------------------Cut-Here-------------------------------< M3.MBW=4X7R!M"=FS[?T0"!P-9;F G&V9^JE.Y=8!$JUT,N0>,&HSGU:"W Q? M74UY1V+0@-I&:2=3'@V7AD_#)65_,*=O/LPF1:NN;XC-4J5D)^?8$M:NA3:K M"JU#[-7^+U+W_SMD+6*\V'R6?,_EX%(7H]==I.4%TIZ]]\L%WYZPI@PL;"Y# M\6[*!V29\*AQJXCLI18RE?UQ_\_4(;KYB%Q()ND9^C!Z,E7XW-2X+"<&H!H\ MAR/*J5^U0UQE3F-M5LCVZ?)IC.2I[.!4M4IV35S8F'>%K>)XH6Y8(E8X?WL[-+F'>J/_4/4R>)/\,.!6NS.9[:G]!9O.JD2DG MG&M*B9N:\W[V/SP 5 4+B).>B[H4,]%GT;BA2Q"#)PVG!F0T[.//ES MD$N_#>7*7^Z60WQ?E=\!]3H$N=;M*764C\8AF((8<4<# M:2W%D+3K[E.21LS=M1%G.*-VQMUU[IVW@ERS=)YG\PW%""J0A2XX]UCJYJX4AN*:Q-+E]]V*$8E87R73MPOSM_^ MB_&[>F)[,C:P3B.LF()*8I1;K#-P>?KVISV)/9+18UENXJ20J4ZW%<=_MJ+88R\8MKMQZ MB[EZR^=.7K$WW%QOQM!9 65U.%UGKRHYVIM$(-U6C(LX\D?*+R#UK"D-?1<# M3CNDW=YY<<$XEOKF0-Z2>Y..G.R,:?,6Z61,E%+1EO?$_&1T*7+:U19;PU9-\>H%_&#\R8\=&W!UVZLQ'7 MC&&^*<44'^ M,%>$,&@@J:L2F+H/36+"G^?&1%R[,;9CRG8BUKZF$*_/>^-5#1+5\-I]%AS[ M?8O%Z-=EI?BY(IK%_?:]7YO%043'\G)$(;9KIHE#1] O[/?Y" M.\.;U6./K+S%ZY(KR4^4V8<._D33RK_DW,)?^!7AV MCU?1;LU+HL9U(+J-V^4&5.@-/14&!\,^O1P]L8QK?Z/BVT?(&SJGSR*9Q:I3VJ M[\BJYM6+85Q.E0P##0UB&+N6V(8F\K+-]A=R;"2,'%^9XDS)J9!QD28@P@M4 M>56_: NO] '6") $>)3"%>FU_T_9Y8J79HNRJPYO=8ZQ #[4!FXXTNK9E0J# MI/=[_+$:(,OW?EMDS:EF9B>4ES=R,$[:]UF"/"1_Y:2 _\9ZE1Z8ENPKQ0XS MAVO2F$\'J;?6++L?@J>V!6* )>9A+[Q4BAS3O3>^6_D#7M_G87T3-#VQA:.B M#M=LVUM"]-Q9O9S:>TMGEV5\N&&08:.. -KIRS]OUO+2)\8.](0X+[>M^Q8N M"@?3C\)KI@E2@N:9MMVP.=:M8&F9D/10J#@:O@($ >*'.1QS+VP.L;R65B][ M13]T5N2:[F)!=??_PR\3**1_M;VHG>W/(IVMW;W>+],$0F;+=-OD6?CI ZOA MIB>?Q6-)(T3Y'(R3'B+W[9S.FZ-S%@T_ M 9S3\4.:\ALY)-R'.P>CA57!6+Y(F^XUP<<&XB\?>9=TJ^(&8,:9[:?9PU^" M/Z&30N9DW7V3SG<];0L[A?/3(-2[].%/]5>@-C0J6F'%"[[>G\L0;@#9#9J. MEP>A]&DV)6AR5#=Q!NXAAM>'P<0R,F NU3<<;;!FIC]])Q_TE+DZ]&&!.M'ZNR[F#\*@P S^KZU#:G MD:5W+KL%'DMX8?CF6>PAJS3,H,TPP1D ]\!BML@Z$J8#5]JX2EYM--SRX[4] M>^_+3]HX]7_:Q/RA)>H6"=/-M,M:GU* U,9K3+2:[X ,8]>38\KQU"Q@,=%5 MI0V&%S8[\ \ULJ,#K2:J4TV:U&?,33&5B1JL/6DZY'#2SDL4@:AR$+II;!T* MJ1O+H,MS7A[U27P&?$=W+EMOBIPQ01TU*]W2:U3_IJ74Q] ='C&RVBHP63U2 M\V"J56('Z3WA]7C,Y)Q"I>=XIYP'K\.VDS8C;=6&B>&M%+\)M;:3WRN.3JR] M" .:1%ZYW(8U_\PI[D^#>8"U643D=,;5)\CM:?VJ $W31%>XWM%UM?:P3K3> MI2O6NMDI[YU94-S?/=?6;+:!PL#F]'/ZS?#P-7UVF/V,W>HSX[IPMH?7,\:% MY7(,-]2"()LCEK-$!2(.K5F]?%>\,1(7$XWW%3S;>E4UTM3O>*I13['#C&#O/GR>WOMZHYMFQ]39^"/^M M54-QB & @5FU+;HCY@SK/7LV+,$ K\+Z^-IZPX.Z M][S5E.FO[I>:+,>WIA@;KY);VW0C,=-F\>:S9#S,[-AH#&W-[XJL7U@M7Q?/EE3 M--ESD#V6,>J:L B%CEFWKL&1XDC?9V:RGE!>!2/4F9ELW!SV+:H0!>@?[ER_ M&7R??=8:=>EZ_];C4S>O;XO&Z&)/( ,78?VQYH42 5K7L==0Z!5U<6V(F[=) M5P_)7^,09O;:VO2I"6#N*9%Q@]24<"$5 MU4O#@I< F=.9R/(KM884P_9!C; M7JN9IU"(,<$W>JME45];-AX8YA?UJEE_.<^3F>(QZJQ:,[U%?= MJ''!$.BM(>OD?MUYG6&'?1&=\.OLH8[TL)LV3E7B< '8;>-%]) 4;LP4'5O; M:PR'61/JLR4ZN<>T=C[>;*^AZ.[,=O;1I7+@+79/N"]>GPH^VG\7' .F8C_]UB$-/59.6 M?=FD!=C,L9TS#72+[N32<7FPP&,RXAYQQ?:?[@569>XWRF@JXNN7C]V!'0PJ M?P&0A]R=*$-'']W[?>3V_EBPP6/@L> T.C"#3?X22FEIS%^4WYK0>2R$1?G] M8*>_TV-7A'FC,BR/YF/GH\N_605^M%#TO=E!--C>;^38[]]D*,,SYUC_/4C_ M_9C5^M]@+K3TI4L!M,#VJQ-W;%C2-:V:E[DJ_N!N/R^-VF?D&T_-N_=%I0P! M:2O.R@)E*4=3]P=D_2KZ_CS*C^8[M7UWGAQ/[B6_>R%I8E2NL,]5,V?CBT,U=@LCW8&JQ"CBAW M8E7!ZF N=#Z-;]I@A.V.B!4GYXTC*E1(;ISI_(_F7UW$9^YV M=XF[X,S[;(*CH\@2O!M229O$\D?'-? M7BF\+&MW("X4K3VQG45^7/-I?#Z4W4V3"ENW"2'B K^21MPE-J\8^]RF"1"\ M:5:!4^SR:Q>3BAOX/1Q;K>MH[E0^ MH]&(Y^C:L0@V+#C^A6774@-SC-S ')#/.=?[Z^."U8S'@MP_(A"VX'G\?67O MHX&BLVQVM"-[VHG]W5@&MTNYTU]9;A+3P:$#M$Q;(),^H-*IZ"BR)/!(VBQ: MMU62<]^-&N37C&L2G?M",B_;-^@^'-O.?N!,C52*GW7(LQT=L %B M(N:^%S6O RD-W]@K/1V@ME?'J^TOXF_;M0W>U.+("1_9LFU$;L^OMNT9NFU; M):N2CJ^+7V!.-;@U@&+'*+>Y_ MN-?1X$WGY7+;".N$O6D.K1_;I,4;=[$4GYS37."-*W/)6%>)HS0%,0%4.XQ=W7*S M(^S#)6V$(I@[G[8W<,HU"?%S"N63F9HU3DN^MC%WE[O#TUMX''UJG'^$G[<0GH(=?/5JJ^;X\6X9;HGF9KQ=8CAYB M6I/.9FKE7><.NRHEY#3C5W&R5J)Y6XO.)P:7)=CM^ 1YF(33H:YNTDLY7;2^ M92F@L&7LW4HFZRPB?CEZY*Z.T6KUY/@T6QP30%O(,7O/2=LOIR^GD/:TJU)+ M:,O(A>1-'YP8Q/?%0VWTCP'(/N1,JO;X:/L$SGJW >5WB.(]HC'4U?@%KN)I MO!]K+P VIS:X8V@O*F=[+FY3X4AQ!S%=_;5>X1FHRK.9U[6T[%#7P M=I>WJV)I8<:Y>GV"G"QZI&>V$F*ZJ_!&2RF2)/(9<3L$#L_";5!SJBUX=>J5 M.7RM(A6QPU?24 ++9Z=_0R9B/)0WTW MMH@-O%,1=7C2D^OMT$ M!MV_<,SI9@!VLEW'-(E:MO?8_F[UMFC[CUJ.'G*'MC7;A[19]Q\R#92!E0*J M !BP?MP_*2;;=4QZ4P'D<57'5.ZS)*Y[V7WE/DLVNP>YP^U":20;][OF+O_^ M_.9>'+8G0'M3_!.8 QYC^C2_W49*;NI8*TS(WG4?CS6Y>VZ$HY\;EDWM'G$; M$F]KS]]0=F P'LW(L;X0E^O7[G MKF^SI=:TO]FMQ7A0X M*#^K?MT.9K)ARO?0I%G:+XO/"(<<-7?; GQ>+B277(/5@0F\;J8PHNK0=GI7 M% F'6<>D3YBV-' )S6P,0?/#66_,P5VW]>CU+IPRP;/9!%3D\XJ6+FJ;ME%* M1\VTB%48L0=2_PQ':W.&00W.D^4D-)Q62ZT!_()'2X?*06(6]A891RU08P@* M?9&C/>H*WFURM?9)QBJ7'!]K3H#[&A.@3UJGHWQB^0+5W6R\G/E20GD&7#P? M6\&>"N'Y 1) Q _1.VID/AO%!?="D\*?RA&U6GPI6'2)"J*P!7$/[ [I\: MPH>7B/!YF^V/GKC+0Y>&L.-U%L?;G?_81@P,/Q<2XH;AP;E_AF,)".%+7B1P.)\M^ MY3 MPI&FIW"!^*!:=9DTI/96BA'B$;UON 6O(8[W)"F\G%.\@CR/L]]S(UO.W2V' MAWNZ]G_4RLG55FO[%+GB0NYV<;97?JW6[+JR-7&N74,_-/O0VBP,=&N# M0G_:8LX?KMO#<9O55@CZ4Y38P6\F-O/QM!B]$;'@:;1J@6TI+K&8L'TL=GYC M<7MRESAFL:U2_LI$@JX%U_AP^43W'B!.0)G@$_9R]MS89^3WUVF %7*;^2Z_2\3,+^D8=3S_WNV]\O*X/VN46C]OZ,#_.TM^ M!<0 [6_9W73@* FX"8 GN*G1/3]%KG7@_JC;SC1_8@'9(+;[-P7<*>@K!;%A MA%"_NW&2*@:<=NPIU/RJ5-=KSU],MDKUE"T]=L$.8>W1\FAM=XZ-JEH:"&V# M?WVEK5SR.-"IT"V/AHFNH\GCL0A)WO-W#*!.0SINQL4.G?%"X,PY\2N+2"*$ M-.NNT4V5I&8&WCG>[G2W20?D>T<:K^47-9[P4XW_L7FWQM_7^.FXX#VBW-'1 MQH5IL%^$ #N3M;W'FPY\H^O;1;_[0@\)IPVO94:'?VFB"0[:[_\1MS:-)H!# M 7#'O-_Y5''<&\WY#61/P /<[@7GN*I OTH?'T?#J&6_XDY+)X QF>23#^%G_NPU9T:'QT_ MR#'9@NS4L7]PY'VZNY"'N6_C$?+87'3.[X\666' !R MV#2_F#G@KW*\=6PW=1#H5Y_CDKDUN:,;NDDD5Y+7-YW:M>?W*(1-0"3?='=& MQ:#DG&XIN4GT45XEIR(NR+'D?L!DSI8\=-PEKXZOQC%^R/'-[R!;ZDPFMY!S M_S#D:/+X-Y]G3<[M#KZ>H]_DMU^+*HQ<=YR5M),/TR"Y>7+@L8+WX\:G7P4'NG&_6;M!N=$FV29*G#LW0,-CIUM _9UO/&W_EF7C9?)3^9E<0ZXJ[Y&3 MN;4X*G(V(JP\V$V ''8[N+$,[\6AZ'8.R9>P/(XOQV]U+ @=^9B 1RXLQZ$V MRPOES/*]J^%P(0B^S*Z!!B*0A7)N-]AY6" :)V]3RR?FH>XM]\28O##,HN8()=[N'M_%/+9N+H(%XN30:3EXOUT?K?@-S5H!\.:V49D8CK^16P'WES]%AXE8.9FXZ?G82 M0A_F3')CN4V <[.X79B'-B3:9,<60NO;77?G,]T9O.,K37QRV+#IT?8,OF=^8$=N77U'WJKM6, M:TWG$O;)_-SN5,N70Y]&/ZUS'/C'?+)N1_Q2ZLSAV3' MRGOFY=^?.8..\3XX'0)E'SH/EPG.R;7PS0 Z5.ZIH MSD_$+)N?*L23W1VS>:V^N]]T3TMY]TJ/WLWNV\/F,Q;"/.N]MR-A-_D$M@.2 M#U/ W6DQ*M>SF_M>KDLW=.^*^&P.L>3S+7>>Y#K#S26N9(0(8+T6YET/GRV3 M5P&?"MVM6%D84HV:NXW39 >A>[R)LYZVFZN_>\S5J^"R0\\-[ P!-S??742@ M#:.TI9K^ IK.ZZ'-5/1YPPU'7RM9:$_ MBM%^35V\7IC [N<_S WOIW=I2)\ ,[^42^>4':72XBKHC6F0[$_\3XA 59Q: M!JS9?6]5)@CP%XO0QJ!;&R. ML)%H=(WGT82ER('E@,USF;8Z@B[:E@7WDBG M09NZCSF,4+!/5QG\9O;=?9V/YKV/@GG/DA88O]MB2:W8S6\L-ONU^ :M/$4O MQBGFGD_WGCMYD!>,:!@$=2C56KQVMEB4C#Y%QVHZ$TF0O 5J*?DN+#<@2!Z> M5I_ H[];GLB3.*Q\IKQIP:])*C_;W0WT*%H&[Y[^#".)-&XI3F P47-S;&#: M! T"N9Y4V[Y<*PGC:+U)"MIX0-[3UMC+G>9B\X*55UJ-M2H+W$T9>';:RM>17QO!QO%M\ ;AQK5@LX-I["! M0;O5;DIJNAY:Q1RJ53BJ+G,XX_3&M DUJL>M/ER[*2ETUFPQ7=293LW1M4/C M@L&@,Y)%BAJ50>KVKJ<_GBNX8X*Q76M.K+K/G@!2!&B9*<_[&WRW/-R8#JA; M7,O5ZC1N>+IZ>2=6)H":J!$")665J;X<1AQ3"OENS5,O9]S'7]KVQPWUG M1H/62\K\:>62RWLWSHOOM3/1CIN:C6F$*MB)5GX/QIG?HFC#>!8;T[8L-J0/ M;AGCG#@QMB>N8>C=LXR[T_&OM@9R.,?0JNRL?1K&U=7>2\LJ[[6PF8 MYY[1:@Y/-G4\<&X['L&68"&EMG+$*H?MXPI"UM$Y,RY6. MLFNPGG(?[/J-E(W*566#QTO@*O!T$Q,V/.X"WW9#,OM:U>/)7L+/%5%8B:[K MUOV(^075^AI15[=7UWS'E CK=9I^WXN;QAW JUQN:DJEE=(4.<6\0#XM7RZI M)TWC[_/HHB4,44GCY9(WR+WDDW4(N:A\ I[:KG(ONS?KZYHG\1<1>YX;%ZT/ MS/VXI[R].IY;U'A2G9>WUL."PO$I *V\3EX^<K%Y86<1^Q%1_1;U^YJWUV5"RSIXJ[39U U9OZ][ MRK_D$G#/.FQ\/@7@)OT"V'ZY[R)XTW[#O MU?6K)W)"J$*MW?ET'$@K4W,;GL MY7D;O>ZJB=1\;_DU&#)6'R97C_?71[[*[5@-8\ 6:Z(3L,//0NFX]39Y@-ZVC M' ?=^6L]]VH=V/U@U[)SV2?L0]'8.JYI0\S_Y1R'1V%,GFX6^.>0F'6^XOF)UC M"T3FF$-F[;A\O[W^=KM=UOWKEES4,:K]R XH9ZBB6L?DEG8R-T[ I'KF9JW/ MMC>!A'/<]M7++.DG:(#K@+Z>O'+B\6==8+Y;'R.*$H'@DS\SZ M6A-$47.0:!MUTVW'-I![O]NH&7,0.9%[Z7TVCZ^[SINHF[38N=OQNVQ\[_=HU+SBGMDW8B.W<]V^XW%SV^RM'

);9_=V/Y^HCEG>9?MI>VRK\0=TT=QC[:# MRM?M*C7 @D=1$^=9Y["CCMWM2O-"=M$/!CBWD>*X#TWDSNCAN3S\X^YM5W"_ MR+?L5@!(Z9M'$MP 7[GKQL?1J78SNZ4=S9YV)]ORW%7L(LG#Z^=;*.%K57#X MM6_M>9N_(=!=L5[9_AL&W)7)>_X[LYX;)[ CV#/NH?67^[L=CLG YQSVK_M!? F@/+ZH7*GF.^^*,F+V!V<]Q$ MK0?;<'7B)I?T,(YI4QPS=:;?7FSRL2\;*H??"V:#L+<$.5G>'8%3]YT$1^I1 M"ZGI4R'G7.X/['Y,W$QSB)GH._&4U]'.U'=)7TS3>4^F>D]WWYROKP=) [M; M/@_%?CHFZV-TZ(Q8JT^^HR^&1LHRY[)MT\=^OUTWO%FLAK6=M,5S">@Z%1/. MW]U]^+U!,6=2^M>3;4TS7>&[R^:8^A(PA9L/+WH3_U3JU3FQK-OSH2[76[7[ MA'UT]%W2VW:.9"E/I(UVU^W3K9NT..22>[IGHZZ)5*KF1'6]S3_TYIC#,9\Z MY'#IHH8HA!LP]>CE_*5K;D6\Y3N6'3$=0\>@V[\%U-&-RG1/[JB8GGXH=O02 M.*G.QMRH;4XY!9SW-GHW6M^6JLO\@M>W1\[;U:9W;3^]H]XS@+Q8KA3G)*C7 MJ^':2T CI#B3!9^8E<,'TV8DFIDZ?'<];)B!MGKEX:M\XDS_^\!0!5]+[*2; MP3N(. $=*CF-;>/4YJVFGP&OTN)M'7L[$1JET0&NODTXZ;>#8+3OJDZARZHC M/[?J(DJONNY[EMN)EVA':=:VQ_>"Z!T=6(DH8A[6XK]I??1A\1]=;QM(G[[/ MU8MOY6?I]_DY^Q[OV\3MXIYUO;C$P!SG,5!9(, M;R#;*DF>I(MX=.YO_[9IXX728T34L7+YH;O4D%R (4(,V" ;F%91ZW7V>.)ZE^)7WC8<1%3Z44P_O\3:[7 M#:WK>>LDIC5^)3E>N"3ZV1B>#ASM.P^Y>BT?!UT7NHB9;\\,[JEXIAGQXE@%@G_XICQPC@9-Y$31\$R!0MH M_B)Y1<;EW=-R7.B?^\E.&_VG(C6J)[<7AA<#!@&&&IJ"_S1\;!>:C[E/O&6. MO2EI\/>GY)RP#7Q40M/=:3N&/^(=\:86\=)D'3S<9Y95 =X+OKM:=;RRVN\H<:V8*GR0@O$"M]UG6F<_#J9@C=7ZRRL M/K=[2NK"ZHHW&NO.W5EWD-F&'K@>+59S,9JP95O2E1N?=^5,88!9S0AT_-*9 M6]N>3NHP:%QX=RLW5(50U,C,.V(-AZ8,H]6@V:V]-+ADE97:ULW23=8]?,- M&2,>34L9Y^J44H@"+2U3$':[4+[=:?@+.]JCI$9_.[K)-40!N^"( !!-V _1;&H9 "H"7&Z6I M19FS(U[>PK249CLR'3DGM1UVQ\S2Z>EY+SV6LWESV!Y-#0*XJ@R/7U@$_WI. M&M6<:UT:?><'..@/8.0RW'2ZU^ MUN9J.&\P^J!X>)GIJ['%T1)"9,9V,M@63;T4>%+P$\.R=P C9GW>7G>?5]C= MDY.&'&HRH,!;!3@%H/F]WD2,/$/ WO-PJ"JD 2 Z#[N"0Z)TL#/\Z.VI\VV9F46 MU\J<,W3U'.7OQBP[#+YM-A2X].ICZ_YT">I1*\$C4-_'YE7-W:TI=.9A'O0&OK]PK&NS_I9 M"6EIMK]W/9S.*DNEOX@G2./5W=70.!]U>6B!!J5^ R>8%-S'9!(Y:/SEH?>Z M4GEX:O%K4K<>9JQ [-?KB]G$Z]V?,?#V-FVQYX=^ZQ&OUM?R>0Z0$DU'E\4W ML6GQA9HA7Z&&H):+'QS_?4/1TO( 7:E>T)ZB"D3G9C6RVNSP^Y'YZK/D5 M 9@ "_"4>QN@D\UV/T87!$<"9=)"X$A@J#Z%I8KVLB?RV#LO!_I80:\^MNK& ME96-<;6!<\94;U=<[;&UYF;7U>4B)?A6T*Q=AL8VO+^#=M5D>*C->%H)F^@:,^5C=KU MR[/2@;F67G2.PV9H=0/Z(=U[;F?4N0[N(ROFKP,QTU;?O[947F'6D;Y<5JE_<2:MB738%';M$5/QOUHO[P= MA9US -DA MJ6-\NU5@]@Y^Z[QRO([> ,/.HX$Q;?H->UFN_FQGO8?UPE XO+'1[%N&54E3 MDC'?9-@-O=Y;@;QD1+ YNK?E1,U#8K+<44/_35DR.86:GGJWYHD=O_"6)-5[ M'8\J#D\1*0[_ .L6SV%_.7FRQ%XX&E?8(V\B-![8+\N*Q3RU>/9.KBE4K&TS M3?F]?G5^X&H-RY>[K^%)%FO4"[Z_(M0F!YJQOW:32,/:[^L&-%D<7NS1-)+2 M\"^7Q$?\@E,\^XI\G\6?Y$(#_,M/30GG^;[\GEKWXG7V@W2G2L^^B\W43OQ& M$A^#(%'M-U(0K294PPL:.,RX-,GOMW9S3'@1(*C>Z* %C7Q!J<.N42\$B$K> M_#R5E;2_9,^/@9V M+&!>=""]$5F&05/K PU/ .8HX&P%%Z)X8&@;$BQ1--= M+&U^9$F-)5^0E ^D1#K^\466:OO ^B419:GN7C!N!N?V,DY3YN049RF>W%E. MW-S3:_1EJXI.U"NB3R_<)^O@N6&QEOP H5<^-BLOB&,1R,$-MJFO8WC!QOX8 M,PNKR"I-Y/&TE=Z#_QE>$UWQ7RF@8.!P.YK@:.$[Y!CY:+7TJ0PQF5^=E[J? M":?(U. Q_'*TPPB'O$8*(4/CI7PC*"LP 2EB,_C@3#>->F@N=6>.4L?@Y:T3 M^%CQZ7QT/AKS*6Y1S-;?1QV'RNF&N8G><4O2M%&:%,^ 7U"_%2.FL?0]\)# M&NGYP>, \$VS%^**K]E(HIND#T&W[5X_S%]BV.+LDI9=H[?LW.WDV/NX'\X#OYO6R?6OO8*KH_YB2_"E#)3R]> M\E5>?<&PH $RZ.?9#LRMW@_KT_$UHK+.GJ_:O&0N[9^BO#FVO9O-;?_PM(IB M[[*BXNN('F 2%UFU6S G336.SL'0KF./!$U=G<>B&1_8_TKB\^Y#>[ MYJ3)I^88([CMU/KR[%UJ(]1T]^-JKL]T-:Q7%#D>##F>['LF'\O&R2>:%&Q& MTE:\#M8[&F[4XC@_1_"Z&2%VM[S4X G P]@=W%G:_@##"6"9=I^1:M<<)''B MD;.-'TK#KA760@B;@Z3M57=]D],7(0N>4!UA]NX%[+7S/^-J)EOYLVO8;P0; ML\V3OL(.\.\O.@=-].P/325$#CD&??AOO?,$GP,3#*W2EWT9868?Y=CDNX$" M4?-J9L;;7G;V?4H7RH7_;U5TLV&3\OO4?\"5AB,I (W&F:\2X &V40]S*0), M :Y>#H[E[8< ./!9"W,6\?WGGV5W[KN5N\%2Q>^+-AC\$ULCP'.:"5 %. 4R MTM#2IJ$!_]6PN+8EJ!HKG^JL=QF"'@U&MOM=@ENMY$.Y=JG789E-=*J ZD.+;?R7_4G.:!,@D.1) M<.CX;W4[?EQ=D/[J_&P$]5'10_V!:%'_*9K]GAS3&)/ZD\.E/A-7CZW='-': M'_>81\&H_FN[NTC)M^3__+#Z1\%-8!! "-#S) HZYVRE@NX+N';=,#C6C[A7 M.@N"4_ZX9*'F4SH0]62^[;6PYF.XO@//2M=)QO.66%ET83N&+H91L!I0"P-? M9-V0$L[#K$>87I@>KN%B1D.[_T9:WMT>:KR7!>)FZ;BJB\:+(9+O0<_$$U#Z MZ>*@3\+('&$>H6F4!M&QU EW(_E,^&=O.FWQ9.SV8!']B=9N?D#MTE7 M&46-AEFU<>4UCE<=;.I>Y87.^,D:*V@W"IY(MOV=6GF10.OEIUDN8CI>CO5' MU!"MC3JF,^1>5:VKOT$NJ/.W(?A^MXV?DDZBQL[.(8?S'M#JOB8NO1_I'<>R M]]-WDKWY\*>:GHR1O1(OTO"_5Z6T=D\QO\^V"?#R]]D;__T+O^11P^^8-1O& MQ9>^1T+Z9_.M*WW?1Q?G]V^![D,)?ZNPPH_5Q_!?T0G\YL3BSH?_QI_PM_;? M?V& G'H&/@3NQ]]1S(J+!FJ4\D?WJ$T0OB_C]V#RU$B5\WZ]IY%2X9].A4<" M"*;U'<68#_#;93_3I\7W;&YKH1I)'I)_IX^SM^+Z]'MRT>_K^S#>60RTY\== MXIYUI8@( ?1A?B?-T^,$]$KT:'2+_,:19C9N)?#U=86,O\)W\A="/#S8^3:[?2(JTQ=N>'?E6NI2?P? MY5'<*+_C,G$=>QS>6@[::O,H#7 P@% MB5^>GCD;,!I4X)BA-PIG1O.F8L)7D&2H\+_71UL:@_)^<3^>921OM<9Z=*@I M03OHJE$1Z#X["MY?-4:A09&J'.AEHE(7HNYO1H%FQ"]OY590JW*P'0F&^ZNA M(P.-467?7=J2N+LZ5%%27<%M2 "B6:=345! ?-([ U#B:2T]P0G]6AY/]W][3T)*B%EJ?WLSNM.^%="WRQ7;4; M.7?A2^A'0%"X7<16J'J/;XQ>YW3V=')0KGIP6CAI,%\=@'!AIT/\.>QL4G > M@#X!2S9?2CQM46*,9^9TY4+H=+)DS$+G4@M>RS;\;&YFG3N\>A]"]6N9#7YG M&'4X08QG&W4@;+1ID&E@==IRA5?U+L9W"W-!.5A8?U<7?59'93Q8<]!J4V^S M0T(;:&LK"MUBD5?Y-!4XJ4OG2.11(7D4/IHUF% ]-'WBO>O!JQ&R7 :5+C5K? M7UU\""H-7!U[#WNP2\L[N8 "0R5T+5^?>]12(7M:5$-JMSJE2_Q0)DU3-DE2 M'V(_&SM_,GW[2,!*31PL24T<(C@U>Y]HA'%7;+YX@W1($$]^C6-$?6=]9E*Z M3G9_%G/830%>E0%[?U9[ 7E2?:YW6'8$2#0_C'^+7.U-CW\-2V%]M5]'>=%+ M>7Z:3'1??30X2KU\H4P-;3]P07!S-1)&4G@C1U,"%D0Y'T5M7T^"2;AP[D:W M6,U]NW"D1E]1'SP4.K=,NUC'8$=:?![74BE/5SKJ<-K3?^'Q$]E#IH@&\>HX W38=LZ'P1?,ERE0F90TAUO#U+=4D[ M.5X0-DH;I0$V?.9+$7E=?-@.8W.7;3U\\4L%@"!LC'M&? @VYDN30P=H43&J MX8U M?H%C@("!WDPL-A-[P5LE>\-RCX'S2,1'1QR*@#M ,7U+:SU_S3UB'!\%FAI\ M=#E]?G2$8T5_@E_)8F0U=0'J@&)2?GPA2S%!@7Q6?GAV0G?R@+1HPT[A/(=1 M_$Z7-YIFY47.3H-_'')B2[A2DGSY@'!R:F[V-8E2YT6!.>E%^DZ04FY C4R? M6.U([X%R?FQWVT4O+Q\R%:J?YY$S%+ 7/U3 M^FS$>6UE^W.O3[!0$TZ&6.ML!SM'7A>")$\(?Z]B)X%K=,$THX"W1U)J3EG/ M3&)A\3IZ:+]^P410.[P\D#115B):1SH=4Z-5Z6@;4[ Z5CL3@+\^>F1C5Q0$ M-#:56$)645]%W)+^'3I-M1PJ$PG:9I(F&D0 M/5]_=GM*@6Q?^E N@-)!4(%G?\Y@4X$/ U6!25E7@==:68$">A='*UH:;UZ! M&W]A@9ADW&W[+A=49H%)@)\_2X#^?SL;S6IP@10U34I*6_-/4&J+0D50K'B:;0P[;G8]59]N+7R)@8M 5TI12B12"SS#2)2 )D#? M-FAK U*_7IJ A&^-11QM[DF/:B5Y.'K1(3M"(WMZ17 [R$F+:D!O'(+20)@1 MU%Z;4,>"J8!A?&-M8WP/:&9QK%N9#2U+(E(L&U]\CE2*2CY\*DP47"M2E501 M8YQ']2X?!:&!K0C6@GU'@FBX5#ML\E"]5$%,:E]";$R!=WY+2F@C2%)=1ZV" M:WR80$J0'=2T'/,=@55'U7^0WA<<%Z#?]9V941A /^! ME5*P;8E_6WLNIHSG4C< =NR'4@@P>"CFW7>)%NF1O8;01I)TFI?^MN MHDE)3EU8,DXD:TA&%5IP3VUEQD:O4+Y&&H+X7!R"&$J%/^=1)$^]6EEAK%'/ M@ M>#%7);+II7$WR6%5&-1_S2>9I=EU%<"53'$PS@B=EI3SX=^1PB%,<5.Q, MACSD0HY@Z$BS.ZE6\3I @KA3)&1Y:!R"56>".11"TEV4@EQ&3D89.WY[#S5- M@FM98&I+9:AYDF%E#XYMR%-T62N FIO8TV!7H+N4U&!(DII?_ML:W]6@54\ M.8 J9UJ!*&GA1FV"+4YU>N9\$55Z3V>!DV=64VJ!NU5S9D^ SFIW8#9N'45( M/X""7DG@/3>YJ1GEB1()7-G^0 DIR)6)O=B$*7&OV43!\&'UP M=AI]0!LF?TE2T5U;3/5GM(*76[:"3W@R?^Y2KCK1@[N"^V,/"D&"Z$S^5BT\ MT6$:<7"#LE,T->!=NVMY1Y9PM&AB<6)\Q4/L4T-YRD,$M50X;#-]R W5?-/4D;>0'#2+T<_H+0=TI^.7M' M?VPT@0$%@Y) 6TRC-$YXT0+6;*%%4#V0-$,ZRCBT-#QZ.41\-1<#KC50-7@Y MCS6#-4!F/65* !$T,#Q' #UZ935?(Z [; YJ;GA5WCU5($2 FS5$-8\UVH+( M.^X]QSD_ CLXQD-I.9("QC@Q4BM)@#W_@M TX6<@-D4 GP.\,Y@_MT:"-/]3 MRT (-ALZT 'G OQ$0P ?*[,YOFAAA%@@%SX..38UR#N&-1%#!#9NA#%2IA%* M958D_SM.92@ H#D09HHU5!^)-62$&#IF-3Y&'S_X-2,V&CA#(+ U2@ ]>J@- M*#9-@5E/53P4-@ $)$=6+W@YQCG4/"P['C:H4W](T3RI0PLVE#FP 9Q/!D(D M?7:$8'FH.7J$14A[&B0;P41_A'LA.3]# /I5@X27 86$;D#81GL!40**A,M/ MC82* A):D(1/9 DY(#;>-2L!C3KX ?L%_S3Y*96@:1#KN M-KN$%C9835, 7(33/!HVLSG#A)$\%#N1 UD$ GE"-3X\B@(B3AYY!S4/#'MK MAC7T'NA6S(1I/,Z$L&9-7&PTT82I2KDYUH2T/QLF7V4I+U@ 6W95 #<_5@!% M /D>%#SN/=N$UB%-0K98J W@A/Y,"#;60($4 *2A'M:! HR'4Y7 MD#4, -------------------------------Cut-Here-------------------------------< ---------------------------------------------------------------------- / e||) Lyndon J Clarke Edinburgh Parallel Computing Centre e||) \ \ c||c Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk c||c / ---------------------------------------------------------------------- From owner-mpi-comm@CS.UTK.EDU Thu Feb 25 12:20:10 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26029; Thu, 25 Feb 93 12:20:10 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06397; Thu, 25 Feb 93 12:18:57 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Feb 1993 12:18:56 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06309; Thu, 25 Feb 93 12:17:22 -0500 From: lyndon@epcc.ed.ac.uk Date: Thu, 25 Feb 93 17:17:14 GMT Message-Id: <2156.9302251717@subnode.epcc.ed.ac.uk> To: mpi-comm@cs.utk.edu Subject: second part of document >-------------------------------Cut-Here-------------------------------< M(#?)K;\YTI"T."TE/@J'],.MM=75'0(M/04$ >KU:JTCI@ <-F6PZ#W6I4AC ML!+$=!=!7!UG.]+M]FK=<=/;'S? [7*SW,+VO[URC]SY]M3] 3ZC1XP74O7%+W?YV MU;UU]]M=-]9-+6OSW8?WUYULFQE^-\Q]@S?B+6]KVS0W"D #\-F6 M,*=]3HO3C70D+4T'W3GU;DAI>]-?*1Q= WC3201_4%5WVB@ QIQL]$QHU>P M'! :MH) K56SVWLW'.UUI]Y9]]7-=9?>8/?TC7H'WM=WQWUU<]^ -_;==R?> M"CV]1U^:]^F]]\=ZTW?P!\RX3;-?NMM;G?0W?1?2TD MUE- 'G"'H $;-FH#%F )Z7:HYG]SW-1W"0Y^Q]S5M@(N>B/@*S@ OH"7!@WX M#?" (Q$E@03^>H_;[S?E#7>'X)0M]/UY2]\F>,OM@A/@+3@1'H1[WZIW]VUX MO^#K=I;]W.%F;3D0#7B&0-V#:]RR M]D=!:Y_<2C@+OH07X8:W"KZ&O^'_-@.>3PLK,SAI,%Y(X#: ^ST% I3!N;- M1A\!PPE-C6>[W97WG^IT/]]D^!Q-2Q_@0S@ [^=A=TN_>G-H+SW4HX)&Z$+]N,^"H^@!_ACKC M3;)MVPMW@HI^Q[O<@@2. [C? ET?7G3W!9H""^B'6[W7PD"H1B_4/K@:#7H_ MXJ^X&JZ*B])7MP%^>C/CK;B\/8NWWF[B#(XVBB@2> [@?IO5C8&6&HQ;#$IU M,DZ"I^+,MAONC#?CU/@TKGT+X-4X.AZ/4^(Y 39^?K]">,R9D(MOX@[XSK95 M$R@;=AP-*)P%I39EW15!Q>CW/;WCAU/1<(*_EXG%T2. JFQ9--%CQJII'8465OTA*<,- M/X:ZUWQOTG/!^IU/:\BC=?^- E@%Q5XH?4L7A)Q@7#U*Y\RW-!T3:5O>FS52 M($R_TGGW0TW1\=]6=4, 6YOEC[5@*$Q/W(]UL*A'NP "=:C-T_'?M#18+@-> M!$1W' U[NXFA05ZM=TK+ON%<;7GT"^GV8PV7V]Z3M# =\3D&,=T<0%_?TA*B M I"0/PQ_M44 6ZL@70!>WK:IU8!Y0IX"C '#M!NP>3??M[2)'9=3$)NU"Q*8 M!]/+MFQ1,<3590 *8):SY"0!N"V7H]>WMS"]!)0!7<8M77$7V,'!.]=EC-(& M0F ^QW6_JT%LWF.[#5L=F/=8OZLN]2A]&7SF,T!H#GOS:YJX<,V:U]*;-6(0 M:6L*7CDI#9O?TE/&K#AO4X#"- 1D%,&+0$;^)I;Y\SR;(Y5C])F>6Z>$@K3 M6'DZ)YJGY9AL8BU"= E+B#J=F\NKN_C]&%.&L+>GT$-ODT+BK"UYP#FW=)WP$I@!E[>;Z5K.,H6_G M3?I_7I?GT3WZ8_VCN]5_:IE^.4IP?_0HG: JY?_X>7[\>>A\-'^ -HKJN?FG MS7:+VE0&J7UFG]J485Q]2[/:4L:#K4T%.@\.\4U2C]()0K2>4B_;U'JJ#5MC MZ^KT+6UC6^%+MG:N:JOG*SHI+2MNUO#Y*.V9W]*(@=)M H)Y8+5],QH@3%.#/X(7'IQGUQ%/MXX"1 C.>;6ND*_9*$"7X%+[Z=EY?IZ0 M.P:+WV9-]1$:01U>'DB3U*@YQ%YT5^MP^ACP"@;D"OE7/F]CX:/TQ5)2G]_Z=M!E/-8^NP7N$=CIJC9J4ZHB"5-OM1C>&#K6GZ(3X M#MX@%.E1@<&.L:?MFX::'KF;[K=ZI+ZBHS:4^T6P!N#EG7>:#N8Y<&QT\6ZY MA^6?A;4QH\/FHW3FQ[+SV7C,4854%^ZDN)<.1U\&F[+.AZ'_Z7])-0MX;EN<:N(1Z8+/M?KI6SBQ7!E.#V>>K2Y6:&;,> M;@_K7WJYO5V%0K39ODM;;,W MUU@[P;ZU?^47NP6/P3?7)#R8EYL[[EWXLHV%N^H*0N5^O.OPL@A2?06D &L& MF!=(L^R<.8;NF?OGU#O-C'#0U(1[UQZW/.SS.HX" M4 E]N)_NE:\H8/47;\4?\/)W1!VT0])X^6<#1S?G)[NYGEV/!D@U'\_(9_"7 M=WY%2[_ICKE]T#\@UH$\F#=*-P8H 7[^F/_G^'HZ;;8[Z;ZA=JZ_,^J;NYJN MFK:8GY <@[%V9]$%_=+,^@3_K M$_=H;O"A\1:W0@Z5YS8-?""M=@?:O=SJKEZ'XCIB[(YT^P5U-]/]^RGS(&H- MGD\OC8_!XOY8N_#F^F-ML\/P.3.>'9R3WHH\[(ZW#]/Z^YN.F<_F%8-O2Y"# MU6!YSR[% ^CD_-)]=S\K0CM;SD93Y07Y59[0V]V N;O^MD/EC[S"\9[[! MQ^^XO"!.2LOHI7S=+L7_[R"JV>U)A]OM=S>O'13B\W7%]!+[W[ZS'['IX%(?40- M5N?KY?S=711DV0B[P3ZAP^KU^DY_1]/P\@--#4F_\?"Z,J_9<-K1H0[>O9O8 M0O5V#E8?Z)!A,)V;8]#I^;V^I]\3FW4N_UB+?U9B\\V?6^D]_?M>NP_V;H!H M'FI#]+?(]'1-P@?/<8>RP_3,#Q8@+_/ MI\M!+.^UP^A]_59^?&#Q>'S$GFKCY1XY3M"[!>7Q DGN9 -U9 &W4&8T;@ @ M==V2Y]/-]#/M94_3U71-/F;?Y&;V-XUF[^04^^M-7;_9^560;CV6D%@7]G[!?2U0UZG6_2.>5+0SM_2V6NJC5#'YSR]1-^>>P<8>ESO MN0/=+0 6'HM<;YHX%W^AO^_;;3]]GQOJ@CU9OI4?ZM4(@*^V#^VT=,4MHQ?7 MK'PF$RS[$A_AU_+XN<4NOE/QON$_;^!7 MZ 0^)(V%1P:I^7>>IN75?CE:;[LO"O][A["/PX$).UB-$$CY>[9O'[5S"45\ M=2ZAHS:\<]6@OR_F=+I4 *@+\8\U2A_GZ^I5184MQ-_26W%ZWL3W]2RVG<_9 MD^AO=E[-V(OLROMI+J]CU;GY/7B'6/9V(1%?X,?18WW7GIH'^:[YI*^>8_G; M@9\H3.?R87EMDZ+3+Z#[1 TX#W'EO>HYE)?DS'U&T+K5:N?B2EZH01DNN98] M!,3DUSTTG=V+V60V3N[=Z^1($\7^>X_W0+EY/Y33V4;YG5U\J]1"=32OEW?H MTCL?;=KDXFE!;FY\/^O)]R@"@C??TGU<5>R+'>@]2% HE :5R8/SRKDB1]QH MG!8X^.HT$@%'_PB+GMGQ;1?\Q#,]#14F':EY5&!C2QE\.1NM C %9\ 8L&+S M>9+#7ZT"C/PF)PEQO20C5$+]_+@!:#_N[+9_$U M]G:Q4ROLD?G[_M7?TKL!8O#!/_HG=G NTX?U3?MP#WN/!@X[4PU T.N>^9E_-^D-GIHPV]_IDC\<3_F/#T8 :L[QOWGE0I8MQ/?_^C_[CAW K =)H^6Y MZIQ_V;\925[-EY>;:R04=U!_4X'(7D4O5A>O@U!4#>QT$CY%'?$/@[:YD]'1 M60=33@WGZH@P<9&>\Q5 M_U -, ^W7SJ9#8!_"P(\])\=KU;FJIO7J6D2PN@YE9GF3I(7E:M=V/:*,P= M_&)X9X !(!*P]49A"\Q% 2-ZW+^[VSOO3D>02_7-YW1 6@$0(!BPZ#>GL-TX M:D04V#5.CJ2FZ;<#G."4.:1^QK6QGA<.6157>Z=I4T@V%C6BW":G5#.W.>]9 MPN)=9ZLSH!F-DE;T^PQ@ B-^J(*)GPJP"! $8 )( 4B!1 "_T!"G]99_6_4= MV@QK0 &6'4\@;8=!(\@E./ 8V(V_G2:PY]=FRVQT<78V%[6G0\HK"" %. )< MO9 CK;@P5B_K)R<;5;8*!#?_-'<\AM GN!GL!I8"0PFC,*O ;2 M5)* ; ;G7W'-+]?5,]EYX%!V0< S@2[0"& -Y 5V ADUZD 00#"0XO<$6 +0 M_OIO^;0G0(0HW;9>8Z,5?9Y_ZIYJG8^H%]@'Y--D4\0.-[=^7WZEYD,(1.\- M^XY[(KECWW*O!* HRPBHF 18\0)GWY4MVE?=Z[+-Y*Y]-KDRF[I-N@;>HZ7E MXZ&0*>TY9Z=C>K'^=&PI9E,_NY M][: ,3W:'KU.5E?22]OI[&Y_2KS-7X8/%+2RX_OQZ_A[MS1%'NHN9Y:;P\*! MU?1Y< #DW6K-:)>P.[/A!C=KD@-873WOL6>&N-"9Z7)ND[L(VL$NI"=,\^=5 M[)88K+KY7TVOI1=I>_NQV"!HWJ]671;PUI>> PWB^?!N<#3>H,%.FB<;Y.W9 M_GZ#R[;@X$5OF/1MRZMA@58#Y8<[FJHND::Z\_!!^)Z#BXCH(-?NCO:;0VW( MJ\ "6$#LH 3-OS:@@Z-!]%!SWL%:7T5 IE.9,_5YYB!IK$''P?6.CZ9?T,3) MY7YW8#];G^UOA^8;BN+]['*#8,$D@)7.H]9 0+55A7Q$(#:P'&H#O!4!'*5) M]KQ\Z$'1H)H/UJ<>X!5U^)B#8;FJ@OR/;+=+*WM(9R_!= M"*& ![TE763//H"0$\RI_X)S([WC ^BN3H-\&Q$:UBH#)4*!H)H@1;C?\XG- MYUA" ;7'6F)@/O=5BP[9_9IT;SIRFE BK\:R&^LU?V!O,@$3G8CP=-?/>]>I M(\8_P 4M1)&..;AL2PS2\42#Z#\!GPW/KE?68__-]*I"'S\>GQ=0!A05I.,M MTV)X;L"LG.PN3F?HP]HM?BYOV;7P1I8.-O>J8:/-A4QY*PK4W#70["#)XP^4 M<-HV$K\4( K@"I $8 +\O48$63;4G8=OLF?\Z^GIZO( Y*CQ'VKC^( :& ND MYTZ#?D(-X'M0,:A-B^19Y3!9G;NI'J/0JX:]0YKLXX:%'CR/VA/0E-?J$Q3N MA[2 =\#-VIA@/N>KTXIX O-J$+WK'+O/8Q<[^.#AYVJ!\[]06@,01AW;@.M3$K5-NQ M[0AWD"(9G]ONL<:R6T),#-YU+"' 7+].1N>0\_I9'J1_",-O8;V0"_@"#%% M]49V]AUH8)L-PJ:T:PB.^( VF @?H"4P"(@9@8"= ?^"@0Y+VH1M"C $B.A$ M],8(!#Z2X6S.'!A'L]AYUF!K,<(=VAA0JJ2TJP8N"Q6#VC5P(*$-_X!6"\71 M,

8#F6FS0=*>=ZQ#JYKQ_B\)182&)O,>RN=N< %\YXP45(*O05;@8M-U M1<0."PY+X *OEI,2I SR:31JYS1)@2)P$)>OLP&F]P1$@<"Z&.<&(9CT 0+V M;E@G!C\C !MM4Q>TVZCU &!KE[C\"BHMRY86@@_Z! QI,#@4 (LP7M>?2PZN M!GQ#M\A!0,P/O8P>&!;&!C!K:(1X#*/C*$0:2 HN!QT!\H -O&3B.0+"- M Y^!FK^M84S 3:0 \!K*#7V!-@%$T3QP%$@$J ($W82'D#2FFBR0#-!E<$6P MT6QV=4#NH$^/[Q>["P(2541U[4!.H%2)47,^G!ZFO'Y^0;\4 #(PZX>K2ZF= M_=2#%#[GWRTM(=#.F]!-VNQ]LKH9R3*O=I@^Q!W6Q9B'[L,B0!- @?CXFA]F MV<8/S,+B&OZPU;<]!-3Q#[N$A**!G&_H66'TDQSB)P $@L-237GC;C,9+ 1^ MY"""QC[EWK,BV<=DT+*@)S2"T[TL6T=0)G=[H\EA^[A[(\'O7K>/EB:#0PGN M^T*(4"$4(K\O,@@3!"S,!*])202;(,9/X<=&8_BUV9PJ:YN?X!"1&X@DC-VI M'11%!"%X?D%P05-03(>A"Q;"!;$=#CO"8%:0D496,P+0 M_&Q^.#^=GU5N$S@6+ LB"7]V@;FFX/?O+9@" S&!<%M:D3#(+^&EH8T)/C! M!0")++_"8-3+A-_JU M!Z=R^L(*7H)POP?*\]J)[J67HR0K^>VB]EU&8Z#[RO7(4G'U!<5. ' \- ,'SVPVNJP\N<<7!B^ M"3=IHP%N7'C-:PYV M"#UP<$,0H-R0)&"[B2=.<*:()0&]8:OP[_5(+"@R>X2&H6PFR@30H[8V3:';38C@.<0=,A_&QU" MTDJ'2+$C8.IPF:BYR\UQ"#%T[T0R(+!N%Z@^A!X&#H-OO$-1X# 0>"@_Q![^ MO(B'S4 XVA,P"A@-) /.!)J'/$4#XH(#[99 K!Y>#\&!VL/9W"P0=0A' Q_Z MAN!X6+ZLG0?N A<$# R8 6V'S\-LH%E1'BA4+ + #]\,#D1(VM@06-.CNQ\" M#F!KO+S]H2=C5O<_W/HEYDR :#?GH3MP?3C!>2HF$!>(38 &HE$1@@@V-*Q- M$&.!6D7NH3#M@@BGRR#.[SB(!4&V29L&Z0-$S&P!(< GXK :0 3P#K(1- Y<5A"P(< MO3AL10"S8& N+Z*QP6 M GDH%T5RHGU/@,4DRN]1$#IY[KW3(#PO]]<_$ '!ZFQZ@SR*@&NO4/CU^_"! M_=R)!SW9WJOO".3:V]'MV3"$S<']'I]N?*BUV_5E"_EHG)M-(9C'(6?B2R7V M$W=Y83SC7>^O3N?;FR0T]O2%XD)[86?/D$:@TQ%@CYSH71N&9@NO*.1 M[UIX[L(&(+R0[<='4W"8Z'Z#4L,.'K1@-C?;X^NI!P>&[;Q1FE=Q0$=+>P+H M[1)K?KD)8WIN@KBR\6P., M[ZIXY+N;GI9Q-G=F5"76ZA1-'L8.W]<(MM8FS"A2\6!K#@)0^">Y4P_FYBB+T,0&XY"0Q(A 6]R%Y1P("+J#@3JMT_?54R7* M\Z9V L,WX]UNT3-*&^49[ )YE+\]HF*PUKAHU-4%!#L&>+F?H69F,&BEZ_RA M!E%V^3L&X9[-B'?8&QVX&3]V[@3'W^+.B\,8CGDTQQVBY4Q)F_T(U>;7_W-F.-Z#KFZ_9Z:)VE<:BFP;0.=AH%.@- M?PQIH;V?EY50J;A94^39![.-L#I2070NE <;X$V<^0)[HSC+6TW/ PA0A!,V MA\!M>:!P8UU."HCE\S8.'<@ ]+5^'6S.(1?F0T4TUQ)\"8&"X5?@PC9GA*V% M"\H Y[EGXK9NB",#I"9&=3B*6I'77S810* #%!!9#F5".#5PHAB136=/A*0I M& %S9[]SXFE0G3C;Z^K]!AN*)46[P,R1=1*2TP$J!/>A%0>+/L6Z8W.Q=TCQ^QT: XN*M#BY(E+1>.@,C!0F M#X. K0B[8P'Q'5CO&B\D$-^*146Q8?UP>>=8O"OF#]5^W,85( #1%AA\X]P0 M$&^'C,= !]:QK<@$> (, 4J!*8!D3NNM0%=!=K/#IN*LH,@0X!8B6P$!A\A ?V%DV(HL5]'VD1V4<1I.04 MD@I);I6LP6H1R_:2V[*]%FN((,'MGD@P)S>!VR&RT3IR/[F4X&Y1B.A;W-/D M;:HR*$#?34W0N)CP0RXZ$1U^4<3>35"1BO@R!"[8V&X1ZT0.VU30--)=#"3& M_,*+;T2OH!PQ@@<%* )@ : 3P I@- /!9 7[",B'=U$%IRD7R&0,&=@[/>I M.V'H;@TXZ'P7?=E5/#%^-R&48%WW;Q0^=:HT^2Q M;!!XASNX'&K#"F=,_-!) >-_-#:Y7_I/?!ADI!F"$4M],,:C3P^/-0$S]*BE M\CAY/CH-G#+(.8=[J/ZL$FF%8 T"GZ%0N+?$:]+9&FV0*$.(@09P_0=[%.:! M[IY^WR-P6TQ-,K?9<_&)!J^#LJ*HP'YH8)C+,\VY['J,"#9[G9&PI5=_?.6- M$H&#ED=$WU[1A,<3V-M1"05':0 =1(5QF&BFV].X!V]\JS4U8VQ.%LCJ\^>A M-B)$9*$U)![O*S!+] #R(+UZDX3_G68FFX=1]&UI%!N'CD"]4B0PNR8$!!Q6 M#B^!-X%,X-VQI^@)3$7N':%""@ 58#WP'HA5/-(5U]2,"KE;&NY1F,:\V,#- MYYJ0G,=U36 ?0AZ)"SR!X:1CT>@'US1J#A7U*-)UFAI$\2\8G6N?ZAY]"LJ M#^$"Z\#%HS$2&ZD 2""6 D^!08!48.BN]38$^.I)$"V/%,3(H@5QKTCWZT,R MV3J(90)3C9RBLZCP RWVV8A]R+WEHVG1^2B_N6^@D:)[5K888O5QVG=]_+)E M'[-]W3W:8O>QO$%6<\#E%L5]"D'UGEI@$T? D[C!WS9Y_,"\ !NM_I882-NU M KF C4 K&X<@3G-\O"2&-P2,PL6,FMD "5)&HO8CSXK 1 =:+>4$HX'R1,I!'/$JJ0L!M_-'NEBBOI=!JE^"\8S =9/AIBCZ!XZ MU 9WLD9QGHQPU,B98]GYAO9Z,CK4'$<@9T;0J_QA&25ZFT0YX-L0#PD9F H@ M[QQJ^CW:'[_QEJ:(%#-F,:QUNT+8FRK$8?<;G-!1'.F, +T:X4"/OY>;6R:" MU7)Y]C\IH%AQUHB\FZO-"2MX(D@J9'IN]@C#X^6]&F>#SS_H() .IYC$N\#M M#^T#?[\W35ZMPBBV2ZP)"3U] SG<'6(MUA@LE/4)AEAV2(&/WX.O8J&[$*F MYQA[M3Q'I"LQHA:SVT%<[W9W9;GK)&RM,PEP9"5F,0ARF3WB'=R1Q3AK3!3J M$@]ZF4+"85[-.0B_D]_MAWB-I+S)'.51%OF0=C\E)5!.* &^Z2PJ M..Y;)4F19/(Q'RD17"$V'Z%L,A^9CU8$,3%]Y BZ%CV"V,?8HO9QMD@25-8U M) -M#TFAG$*0M^AT)")*$B&#\4@D8DL2X8<3/.+H!'F"5QSFHN_&N:@"S$E^ M!FB"6D2?I IPAX9=Y*E9Y9B"'[^BX"N/5+=9&_FQ*_N(V4HT8O/0NZA(7$IR M!>&(7\%.85C0"1 $6"#*%]&",KKZHAY1F+A_O/4@$K^+6D&'6B$Q,'A(S*N% M)<>2=\&L)%J2D7843%+^%YU[CD&>#61PT4:M'"(ZZ"R#@#0L UY2,_A)=."% MT@I[B#\)WW[OJ# 0E$/"UG8#M,9B6<#P2$EK1&VP]9AZYSSV7(:O-"E,A-7% M_A!ZO#/?V1.(35?CTQ<*U>Z0P,2EW7R.(] X-#:N!BP\3I@]P) M$]]R]$GB'VYP:)=YXZE]Z=!V9;FU7=K.EP>%G K,_^"%1LHZY))0R-@M; Y) M$U.6,T"=H34Q5)=-]"R:(DF*_(&CC\XQ=I^NAYK2&_3](6KUJX2A,T]DA M^N) @;DVGF9O.L )XB<>";6&8[Y8X\0-*WDBBXJ)R\!C16RKJ M#*6!#HZS8OJ249/ 9"M2$5U^4H"KXNJ14VA8NRFB+F-W33K>Y&;--QE'PP/@ MHNB.#K>H8NC1LP&^?&!"'B,AGDM&)0@2CK:JQ#Q2(RF5UDBBX5,Q4SG"I"IV M*C^5D44@-I?ZN53E9W$.*+6.5O\=ZU_CR%+G ^RQ. M!B.*]T@AHC[25WE:Q-OX)--J[0.6'&M1AGBLI"$:))65",D<8FWQRT%+6R&\ M**N7TLJ5(/EQ AFO(-RD)+.5Q$4E8OO1*O=^%&?%'U^15$0SXA>1>==?A ,T M+/N/^D7P(E-RO!BPG"/V_!"+W!(I *;.^G?0TS_B*U<(IA'\(K]RO_BN9$LF M,0.,;DPV)J[2Z?@S1# B A6,&\C$HZ@.DF;VD=#U*(>V3\S;>@O1N1R>I[.MY$J^,![W<'!LOWAA' MZ[8Z5[M081\SI2@&=+?=)MU[MK]:8)E0@WEYA*WM(AN3ND9'89Q/XZC%XUEB MZ !S.[IBH]X20CE,$C52,X68PK1(X\2 OD9ZHQ02,R> ;SIA9FHR13E',Q"& M$C^#XT(^I.IN[>?F.3[(^,*6Y+O5'L:/< D(3CPYU@VA*.-+M.)N3W391J-Q9;/.^@A'85O6D2BW%?*^%BM)$4& M?;*7BD K9?<22PFBPB82Y2"!(\7RI?PQYA.F'%.J%,>.+$71(3B0L9A/#,5! M"[U_OC\OGS-QMA>GI(EQ!A>/4!M&C9O(;S-_S%,*,)F1!,SCH>%14!E\PVP0 M(].*8LV83P+QO"C!!"52'N%H%DQO9@;S#9"^XV"J,T-I9<4F#@W3&'E4,6J: M,)61@,=DX.0QHM?"Y!&^,/N*X(8 H@E06M&_7&!.<.(5"41/Y0$RASEAVV&V M&GN8'-)&+V)]MYL$HPSTT3J4F]Z2R2;-)[O44H)L\&'QD15"&: MY"HYD(22!22A'I?%?/91'Z5]U#[8HG8OC,E]3+.1,<%I/CD'7;@OC=FW6V,2 M&,V/$LCAXOK1)3G'_%8J%^^8X\KV91]S#G!__&0U( &#_,(!.,*\M.8NN&=9E/HU)* M[\Y^QTM7)A,/L\>S?-?)YA)RTP&V'4%/$>3:R\VA[FQ_Z[]=X4N/LR>:$ZI) M]=Q[CTGG(&* 8^8;>OH)(1UU7@'7GG9.G/FU<_.\!KIR@4M5G@:0K/"?Y%#^ M?IY\]LPWW9GME"FV3&9Z# >7YT)_IMN0\5?(0R=B(AMU0D*YG"<13:>XW-"- M$C61W U172#-O_F)[%Y:V68"G$J)XMHF<$-27)48#36'8LJ48ID2!6!V="FB M .27^D3ZY700/W?5+,L5&^6.6KMWW;10F%9OM!'.2$ #<\IB)/00SGFG)%>^ M#Q&;B\K%YCGR4>G"[&7",".;P4@V29P3KMD6N-UD-G&8?,I1Y<+OL\G74UPF M^!J;MS339J227(C$A$=^I72 WXW.XE>J9 &EC&*.%GN5M\V#8FXC>S?Q*U:V M%JV/R,HOYG 3AUC -^Z4=TY;9RC^E^E$G" M'Y&(>$QA8&23!0BKD2[F-!<%$"6 M%S>!Q8@I !5 O1F8LQX&YJ8 0S\')'SSUHE^?&X^*6.;!,:O9.00JX*2S'7. M<*!]DTQ!YK_2VOF4W 2^_+:=4E+$&9^LV84@WN4>@9A*/=#^YH5K[]I(?.HO>=E/$) M"94_YDG\G-=OV.EE\,#E+0^<.CY'W_B.P1FOPPVJ(%6,<3SNG>6M6RBM,-%- M!A!K+4/JW:+0")" ;!($ 7!>;ST(H4NM*"%0) #B]E8"#C8['0WK?F?C!'DB M'#A\;+];&LDS>0GB[-B-TER>'S]>8722:@G/"Y-[<+1[JHY^^R7]C& ]M-)XE_4\^V8WEN[?GB)#(V MX42%637/9?L/.O5>^!M&OUU.\^>IU@2Y_4!A*1Y@8:)MT?> G_,OU:VE-11 M /]J:0*L6L<3/4E1RZ;L*T^9<<.>W[U3O>GM#'K:];J- +V5 '_LB,?+;!18 MZ!X^";[J9$53[QEGO DF!F9LBT^Z)TVM6*C#8U*"VPIT@30C7_"R5>>U4W-: M%J^%SD_?UO&!MP;:/!+>(=%\/KKNI-M0&NG+$[GI+&=S,CZ97:Q1YIEN(T5> M\TTB6^J1Y!:&78VX.EIEBG#) /TMZC,^Z)Z30-&?">S_H M!UEL\L\X6E]2,C?WVPZ./#^8MS27IPYO3H.F.W(V"]]]UT]]H180R5CC#%S> M]@B7/TZ"W%<.<4G\4URRV(Z0A[L2Y*OR82?VPS:H'"?LK"=B%;UIM+2.(\&!=/G^@0#BY^)Z(#83FSPS(3?6HZ]! MTH:9YSPCWKKPT!?^/(#&\ H'ACKBY;RP2,G?8U[6N_YV@[P/Y@!4G(GR9 TH M.'>%+,\[&A=2 XIXC ]Y#0V>*DW%)@L"=?C)2PPM/X-%T$^"WN4S>AE\+ GX M0)V;'Z7 S>T&J#EU?+^!(HERD1! H$@RT*$#)%_>'!-$MS6;(*$3$QKLI >^ M+V5^$S9;9.6Q,IF^VT7J%<.,C\@@!8+R)FB)/!**%GQTQ3A1U?QO$&H^O'^* M,,.:-H$KCC>2"5!Z' *D'HV*#B:_)1SM,><6M)5M#2L3D[/&$R(_60?'5F M-LZ8RTV58'.S AF0;&/")=V=<4SV(T^MURD"A"(".[.;0L6<)+_&V"D/Y"*J M #%"RTX^YG81VNG2DG;V*_F+XD7SYK53]QF5' @ O3AL6@#VHGOS2-F5I(DZ M+ >>-]%*IA7Q8UG% BX.&#NBFTR39>O&%?')S&]B"L<+HDSBF]E/=$D";7/^ M&.-U;5!UVBEO":BYZ_RA.6F3/\CD0&(-<.F8$UPJ&0N7*E"[$ OT-,@!M%EN MZ%R&J,I0VB\3XLCNZW_"U%1!7$4VVHKQ.@>8NV!>"PYYLTQLXYKSZCF? _]\ MUO!U];L9$ :MRS#!TTP*1D6@&D\FG0?.+7K0U,U1+5MV_+WCH(EN#E!I.BTT M>%Z6P]"\Y9$P+/K*FTT*)C%"^,+_IST1-=>+3-CY/L^?W$G*0 [2T-?Q[#(6 M\3B&T0,[9&%TG#EWM%GN&<%M"T).T-ON?9=_4(Z.1<<$-$^.YC31HVEE2W!< MUW2:-1\:I5*S+0"BPK(105.:TIK6V\"'%!K1:QTR)G%[X :W(6#2]VCIQ"]T M([&C/$VIJ,SGIYDXU%XV.4.2#+LF#DBQ=J._J55N1YF4#L^P(YD2JFFFE&J2 M(]T*A,:1*"T-(X3Q)'!Z_;J:8\VW)MY1?3D(W"9J-_V.T=!DX#044+G6K &: M+XMY*PK+9A!'Z=A6](9*'L=T=TTVFCB4LCBIA&QV?BR5N0W!I@B3L*D?O6&Z M0_MOHDIX*)"ST.?H'&VB,R.="H(B)FKSB*G:9(IF*=^2"4&B'+\&MMG3]'0J M'T&=+$0'3J #%QCY:H@.)(.;R!&6=ND5:)R*4-^<4?6.J M'Y.((E'4(1W3UVG'/(F6-2E^)$^UPYN-N1@,3'96)[&+@3F>FG91C)AQ_"(^ MUD9^=U*C*""3X-E&;$H6,@F0V4XJ &(Q,,+ (H M2AF(@3G#X-<33\IAL_MQV/ <,%LBDV4DOF]4RE@JY".7PY#Z,_PRS$F+2UM M,3.0"KT-I+2",TC*#"5>VC*CL+5AYYA3F*;\W,HA2!&<*L\%9X@S8]@EA/NE MT99MT,ON:+YP>1=JHR=2^ 2<8SYRVO"MAL>^[I1TN43\7(;S76?V3,AA M 6>#O#U J(VQN]!0HZ7Y19=J.40L**%Q7%?^U!=>_L"@)LI1HYNPPZDX\7'X MYF1T>[TXGC0SQ8@Q9-P5/?%_ID_Z19]4B ?'L\XI,WVA&%/4H/MO_TDK/'QV M[@)YSD%^Z5KO\2F04WD&EORE>3E[YO).>(B:XWDJ($>?.KRO5 =29$KX9'T^ M(YN YL&67N_N3PC>ZI/&/M6%/TY16NX3"I H163"_*Y>D8$LFV>N^7>[N]7Y MXJJ&%,=>9MS.)S"WPVK^!I-UZ7Q5/,#2S[-P)#)>F]TZG:=3K-CHN7!DR M&EV#DR $'[L4VV@N3+5']N0!,UA\A0&FIN[??^3,C% M165V/;UVYH33#)G_C Y,/UV 8$'6HWLO %H#7(/F)]-SVL^OIP)TS=D 9:L= MTV9[H%,):$=R [I)$_4Q.@M]%U"L&HNM6IH9K1GBU,"%^L+7) F4+7KC?(NF M0'5[5D#K6A3M1.J/OR'NBY9*K]*=EH==.K)@F2JR DW>]M M&71ILST5:L547A@:\H+R2P6(W W%XS(49I-&Y8;V'4N!O=!?J. M@1IK L05PN>1T#E-N8XB.T>!T%#.YO#03UG #%1Z2,^ABB(:)BY4Y>)'[8;B M.;^A*%+&)CJ2_9EY_'.^2*4; L0YS1YU0NH3[([.-2.8E32C(NO1L+;59#&. M%8]N5SF^ICWTEO;7/(:BQB2DADHD8B,UV5D$T&R"*HV*BLXFXN_TCM8C%8?B M0XV8=;\B:0]_)Q;ALK*7"S(!D1U9(R M*W6(#,DR9@\13 J1E 26^^1]SSRA6THSU):!\T5^X)AOCKR?(8@J/FHX_ R< M -U$I,5\9:A&CCD279,V_"*9GII0#4J4BJ@2513] =VH+U$E)!H@)KHG-;J! MV#!^"3[] WW-P:E. ZM]"H^2<3:D**]TWNF4-&0.!+( -3\DP,&262:CN^6M M!6=T8 00VT.5)LJO4?EA2I6B%[AL)*J],Y.%60>$+>Q/3/$CF76 X.25>?/(3S) DOU^C\]!=@ M[V8":+HG'H%/FL=C+%!^Y4 $EI^/H=3..W!O_-GP_")SHTXVFG4Q?6?[4P[0 M$\R<>\\-W\W.C>?ZK+M-&1@1&#K8)'^/,4H^O,"I"=%T2U4"GT-N=BE!DS;& M&E^A,3OD)&=R:DE,^S'J)W&CQTR*@-F4^.=7]3-2/9V>,Z#!:JWQP;A*S/Z% MZD1] ,)C70+U,4=9/4.8ZPAW,CSOUPO0NM9)2]-A^6B<6;WK%KVN;Q2'!%'& M5G=ONCKA),LF+FIBXWJ2WO*:,4+:*:.0#D#J0Q:">5!SPS79JK9N78.#H?-= M[P)II=3](:!.40@"U#]83(6>_T00X*?PO+H A0)" Y%WH;;,Y)EM-7IH8:,) ;J'M;^6GE<@.@=6C*LI3X6,;U,J:(UP;)=0/>;A M\LQ_P\05'_0N/W@[#8/V!,9S@,M=7H8/=X 9H*FU[KJ+507B (R3LA:ZC*7A M 2145J*#JC@S-UEB7=0%1V%X%L X9&B2L=IB=$^^ZQP$=R&V7=XN7G9>LNF9-NC./A!OND DK3W([N71<&7. M$EMZ40BU!GYN]Z@/5=U%&D-I8+6$0!O22;A?'4%*Y_*E?<7;78(2[:?_B4B] M +,I=L<6I>0R?/FSB1RN9$Z 1Y7;&OE4(OD=19_V'(&J0$MS(C@3/;I.M.>= M+KV9;I7CWL_&:VA>?5UV!D.)I$I*H7) !X'Q/$_Z^=R47K_3 :2HN49ZTS_\ M!01JR9RZRR95?4@:@ <*B/:IPL"7'RU2#/ H/..M&8*CP[@7X7GPH.IS7<#DAAFMQH\S=P=(%]W,0UW7H^'&RV6VT"'%<$ M)XFLRU66J\=TUE??BZB=*%N#]M(40/:SRY#SM,XM"LVK(5,#IH#NX,D8_+6* M'8*M)D3N!D]5/JH0.99V(J-VW,M&H#J50Q#E+-5@>Q""AYJ;8Q+5?"H@?6J2 M';F<9TKQ*)N2?7I./%XN* VCBQX!(N9FT,E)G=MH-NJ<%]) :IZS=+EIE+NR M-1%)51EE*!\5"<*I?&!^*GN>U,[6&R4/]S=:X=!&. M"P6(1QS.:RPUG1<@2%0N(T^D8K]&)0MSD@JI9)%6(P&=&T2F*(@JZ$,3-!Q. M7Z.BUM==96VSM(@0!5;>;KX;@1L^#4!2B_G;G"%6^VR(LL6$9+,RFPI.PRUR M4YF;Z;UOZB;N]0;-*T.FVT*1;(YB*13J+;E6/"+6;+:8W$HUJ77S66$2_;]* M42MVHX.8J%5N,7E0734RR^0)#5C]G5OPXX=__">>)8$(.\:&:H;Q@D=2K=.T M5,F;%-5#*7IS"E!+)0+<'], @;DP@'[U!60TA<"6YPH('[\$PE%2DGE2A:,M M)L&;F!%-G%*29"A/"'C&;+;2O/&P[%>R+0L(>L'](]H(L+>#W8$.FD=.6GN2NE%H:K/P= M":M&&D!,7= .-?<"8F;^URRF7\ >; LPQ_:_NPETZY29"02E:9;M8M$@H.== M[_IUUXE(9/@P#6N!6\.&)V-S%"4USR%Y25(AWW-,(9 ^*J<*$XJIPH6CT/!+B M8;N9;[KD)YSU(B5M;'Y:[]9TF%>^995P&$=HU0/V8$.F D+I*:!P:8H-K*H= M9"]J"MF](4,1%TC>FIV):5I MOU"&JM#A'RPQUBBB;+2>\T9_>M:*G4#U%_DBG;+BT=BGYD\;J ;P](DQ?,6* M!FN/&U %X*AQC0,4>D8L>IIT"0&EP%ZQ8RJ[@_'E+U6,9A_ ZI+N;GKB&S4Z M8PN?E[[=7J0NGHF M3:(/.-XKP)UK%Y4$(C:J[3!T< -W-:E:Y!4<[=L8]G- MZ32(O,A\!O+N*$C]::Z1[DBRV()-)%0HM$-=3,DR9+$%<<+)[+$3)BN3)1RU M !>JJD/5Y^S/W0A)>PPT$,)\_$%$8Y/N@6HWK=#A31%L?3NM;%%V\2,ARLUI M,&N/4L#::@50&5LME&[PYP24)\JW[%$VV[AA=,M>8\6-C$)#:QX69$@J4%'R M:X*)W]@U'1\S'+M8Y=!5V+2#X* [6I4!Z5D#=!%. 'E\;<&)[#I3%4N0[2EJ M1B8X9T>4+,5/)=L+-(F@9W6HJD(#)#'"R_#'@QU"9@V!=L/T;.1+H<@W[ 0J M.+)W^EE584PVH]H+U.*(-IZS+]F^(Q9@/ML"O+7&;?)JC=FH(;SN9(K::*O= M @7*49R%$\ L!K(&U#L/?5U"8$/)>IOZ !I9+"A9E]S9-5!@6^6^/>0U<-B MZ"*KA5,^;$CQ[/B'5<,:[(ZJFS5!X>.T-1=5B(N6!U>)=#_QK/[NSOI_<[/V M])IT+=H]*X9.&LF')4\>"8"L3Q];QJ)JK/.JF/(^N1Y.$DSZ.(8YP _C;NV4R M8G5S407AH# -"V<^!-1X#3NQ:%IPH)HV EA<,X]BZ$J7GK_(7#QV[B@-O)@0 M(_&Q@=K6V]'KW@H_+:[539NM[5,?I^2.5L?I.XO&[A:N8->&Z]W/K,>)_;9)/&4ZUF&;)0&')B7+>C8UA!IELBO'EA-^>DZ!9<> M]"BN=3%;*,:UUL-&U84*%=FSZX)##3;T/>J*& 4N:/%KBJ+66ZTV(L61S=7& M/5VG9,A>Q(\A(7C)G'0I+G4E*; MU*@.5J_.;.IQ# =)F]929'%\C[_/9K74Y,F@C-V!U:Z0"[JE+!/RWA=1>[BN M;^Z?:KL1HW#6Z4;&G--(;7RNYUH.*$!U(=N?9==>DZ0V[UJ*WX!V)ONLN'%( M%S&VQ#-G[7PVWU-^2\?::@&;N-K>9:)5^4GU=(WN\[BUMM,YH:3#T. M<;*(:+HK9%A.8%MT:T*FVTAO&CSC+&T6=*=Y?=B> 7^QUM;LH2R0?;KI*X8V MULB39-@CK'(@LI>R=-A83(V2[5IC?-4,'QF6'UH7T %J M-NAIZM2Z0 ^UIWI"9*9R7T.==1K 3=R6LM7;W B>.@F2J*3=_Q86V;L5':@H3'+I\7:6.0Y^OE]3A+7+DXFK9S)XF$'NAJ$?5HUU31_IZO-/> M9HEWMD?B;.[Q9LG*Y#T21^&,CEC$ZBWPJ5>N93RR;P^+L%(^Y2@UN :\H[FF M 1Z@NMLFG29V4=B)_%GM@EV_(O/>?7]&#:UI((F@T%YNLV@8N\;0(D M :8 30!!:K,T15JQ(XOJ/8FRKD'A; I@#H "J$%F5J6?&L!_X(+.NB-PC=,M M"E.QI4]D9BMV:\C7:M\2%ENX[=!;ZA'T[,?#;'123^NAM]59$B4NEE/'<"V2<65;]*7J+-13LH2K9.N M*\4 "E@;+",B LOF=,'R,J6 $UC2I7?S3XJ!?>WE:!\#2S:M2+2S"#M15942 M,EFEFT 2K!/ !#N?0L%RV%2PC$87[*[6!2NUA>2RV&2P1E$:['@S]MK(S9EE M*P6#A%R7*N\6O GQ \(R$G5 *M4(W$RUDIN$I8FF_$*YJ=)4K*S2WP?&E<(N M_3*BAD.H'WXS,FGU^W.,,GVK?EDV&HTS#$L:)!3697><05CE+!J6 ]=87<-B M,QNYC[N8:S\Q#LMNG,."!\>EEM'D[!G617O*P_Z-%P!_OCEF;BW68">(E<36 M;PNQ6,!#+)TV"@FNU4UV]"YWBDG$*B1V#HF?F\3Z8]V&$=!+K&MMR,: 9.B5 M:G5 _EO\7R@6H)M/(\6Z_@:R3MCY=?MZ6*Q:LA9K#CW;F>+O41B M+?MI+L)L(:P0-?>+9>B) >%I2+4"W0BT&)M*9>CR\F)VQ]AWXVRO(#2?0[ I M:5EL;8"Z31 M&^>9-L+5ZK7QS2K[K!0^(NF QLQ6J(P5LC?2Y0(6,5>MU,*VZNRN/TFUP],A M*,&6>]FV0*&@SDHM_.F&!K%\NMGY'R#/\,?QK-@Q_L:N0D*67>LOUOB+18"" M2(K6RR8B'9G 5N+<5PUN^3BM_X\FM98A.0;!W:I!2 M2X6TQLM\;;^QV@B];-U%7*>CNU8(6^\MFR@3P$^<;?^C1QST92=7&;>CS+(- M:BFXAEK@G*(6HFGK\_#%;-FY7$L.'NVU+)>G=3+N:4UW1$,VG> M&.-\1DKTH'_PGCNJW>B.2:5R*[=^K@DJK%=)6 M:X=X"+9)+@?5M$MF/:6F:I<]_-W!XL\UT'$0=!^N:U>J$-[8+;Q6/@NM9RUMUK!J+6VN$H\?;%-<>]WIO$%MXX&S7)2K@;;-A M>"NHAM"0;;R60*NTD]8:>/&U*MM&W;[V@NO-_-?F[NYHM=&)[CDUW6:P70B: MZ&I]L]F)3JKVOQCA9;=J7.<4.5XJXH7W4"/KR/+":S>VGMF&F^T&R^O W/ ^ M:]UH9CN;!N),4M3:NHR9D>W.^XJ8I.[[MPF[R-I'Z>S5VZ%03&] V:A?EW7PJ#Z45GT>D;="'#;BT/?LU;>^H:;LQK+.QDDNU#3>: M#T-]6-MVGM:V]9:Y;6DV 0FE:UN6 M$V6NOKP6'E27/_CX.Q*V8O5W=\A;8!;QJ_FZG?:6;D6O:$U"ZMEO=;MG*]K* M$[6A-1O#IN]P=DM3\5S.;\.(<\?D:J@V_EF]*AS-!JC_%5@HM+2[Z"(*&1S-<^)U^Q' I]/8:2!KZ]U=Z#+[/6=XB]Q:]I M;Z^M%-QO+8NWRUK2A=6==,6W%U8D;?E60VAR1=^J5]6W*8@NZ?<0!KC"M@/O!%" ^ MZ%2OG5<.P='GL/C C>#*;VVW;#0&:5F4^(?7T^!R<#VXB+N[KI[N^R%M;*K]W5$E+J^'R2&^X/M+2Y@YWTIG:_.'& M; B'0ESKJ[*F"NLDY54F<4^+[9OWS:RM;BN0G*;F;:U]$E&^[?KUU1GQV.+. M.KNX=-^LTV;1P9$J/)/N.H^+]M3KIIOT)DGQHY69_/:8"]@Y;@$42>>"7;8- M-)>%%-@][@46!OLN1;5N8'M_::!>X$C V+GPI?B]\E0 0+XW '=Q5RKO-.3J M1 66F\!3X!# "G""3<&N8"6YH-\7K'(@!OL8T$[2]^Z+E])-+G_Q!ELI!>6: MYG]RQYRH7@)O*34IN*K?',G&7Y.7J-\L>ADA0HHS3*A?:+?3^ M"LFWRMS4+"#68 B,\7]/ZUR25I H="VAW%6#4/6 M:,>Y_5/;G^0.G?NU)7[^.(%[C%BYK$!WMPLKA*0%6'5S>49.8[4Q=YO/%>1A M8H5IFEB9[\TWRT9?(Z?]5VV^!-TLFT$WXX::2\4BXB!I6U^DK& NR]B,//4#ZIG;)C*5 N]$J*G[[LN=MKXI'Z2 M(O^K^L;4K,?4(]N+S;*%9'5Z_KS[;#:%41,00\BB $,#_-F5+ V8S#?H>])I M$91:.^<+ \ET08"?6]]D3$%'NARZ[2+6IZ)@UK(+S=+3XVA?O_<_?F&7-V_\G[K/I-<#.+4,\F M@@-K4E/6*B68D1@ C@2C@#>@/+S4G=J5N-L)GLT)?RRQP-,X'BG81MB(_?CJ M[/)I%4=G\#46>4>QBUO,V-AVE5$^IHK6=KG;]=\" $>YOD\A7AYPH$N?]?^! M8X]T#+OOK_U6/QC$4[5!9B.9=YN/7(+V%ZP*=K,.@Q_!R]G.Z3'X'RRYB\M> MY5BF_SW?*)S.$:RCO>UQ@%&,?;MD\%%WS=?#"#9>6L%MJ3+,)W VL.D(4K13P2!CE;:[I+EVM#C[$J!-4VAA2S(&6 M)Y^>YTF#)E*0 5S+0P*SZ/B$G5HQL%_N'QB M@U.Q"MT)XI%PO7H.EALB&$6'PE)5H1/@"4 $" )0 4AJOD\W*S=S=UO'; Y- MAK.\"EG+[^7.!#PYM0(+T\*LID:QY:()N->BO:HV"_L+)KH*HALOP\7>0/X&YKB]3<3(3,-'U MAHVYR=X)L$S7#]IVY472=X7 %('*7^OPL2M519U"Y4X>VM/(Y9;V0\ST>V+V M^T!4WLK[3:% \:I']8[N=JN[OMW)J\#2">@ME9^^&1V\*;W$Z!CU RRAO=/J M,NEYSUT;,6".>:FV2:3>9TD#?1NFW]W0;?,+SJ1V1Z>[.J!-KW571FQ8"^XR M6]6CP3GO[IQW,CD"'A6N:ZBOFE3\,'LVDVHDGB=2=\F[^;3S;DSS['>I;>\N M6T^#FUJ8I[P43LL"+L[)@&_!F#M%G2[88B)+=6 B@FN\1=MLI%:8[-?;5=IJ M%=FTY%L69Z37$EGH!= ]@>^X,CII[RM$4)RU11.3#@6TD#@F+S\*(WGQ:WT+T6:J/#5\A;G7,83M?^&0I3[TRC.*AULB*E0.'T>7;-S>78>XET P96S\;KE!/9R>C'%H3C.[:9887NUT$/2;7:3QFNZ'6!J>ZFA[M7#XY]F>GL+31 MB=FK\5Z>[P&W,5B\[;P>;].U?4<3J6(SDKKGS.$Z-OV<+E+)I@(@>XKOS=3< MBTF/IL?LK2@5Z^SK]Y;\\7@1M8G)%" M#QFX1%\(KLR8@FOFTPXN".4),1V"7,NPH]IL')NB#MZL[=*Z[M8RUD@';N/- M2%JXU-O7+0S71BK#U6&V'G>I/DQZ:-H7]WJO6OO.0'VXP-(@[N.V$/CE\-MX M:8U[NV(HJ1(WRC:S0=[,;'B;TKTMYI64FFHLWMM>4\>86=SD9NLF(SI^3/Q^ M<:_%H0'I)IJ4C"N C4F61)>+4D18K0I0-LS&!7:Z<2=W<%S,KQP7DDM&W.#9 MWG3?+?1P+<2/'M]S$KRZW*FK_O8I>_7ZYH+TO+$DW#41AM?-^ M''&9"5[R,LFS).!\V@E8-^ !QL)1 M"%]_$E?Z[J=1@MS2>P&/TF[&_UQ/+%N09SR*_0NI7;&^ES=_PBIV7(Q;5>Q* MB+N,A.$U[*&1+GO172SP%>]U3N"I;8,63CC6+?55=<=SYTQ>GOLSN*K2)4/" M#9QS+MTS9 '5WM@<-)WV!,8"-UV[GMPS!;K3/0KV=!=^]E?['G<4J6;LO;L9 M=5&3.E8JLONS8W@TY1'"[9:]3MJI;E,7-ZIJ!;(V![5R_RZ M(96RN4+'+J1W^)<-WMA1=E^'?%.M[)QP-3P"QL>J@O_ ?S5'I$!QL#@(EOF4 M%:>(?^)U[2+X/M<(3MM%@#?(D^!=H2/X<+KI>Y F"6W![6!(6C,8\CG,4YR MJ%1^HF"OWS1X#?R_305C@]-SK. GWZ!MW\A&BP7;=V?!]5D,'/<-OLIB8(=L/%@8K@XG!VUQC, ?YQ.IG$YMV&0)I,<)M3^@VN-Y<)&U'OR_E1LZ5>XV%$A?<(VWGQQF_">;82'! F$.,D&XVGCI.PA[ M@A?(JT&&<*_P "?)QUC@7J-8!RDG<#+F M5MV1A+L^G<]218ODHZEA2Q.S-.':98;1D=!6YF8J_R3 -5G%28EN::<*ZAL] M?.9J+L@;@(8 .XRO&Q"F5F_"?N-10N-/LX(,.& ?OQ_4ZQ> M8EVMR3HL';B->:HL:)=V"7(4-L;Y0%!(?-#FY8@QY,OGUU%@= M=GM]XYV[K'/0?)<#VI@^_R004D#SG\;J?=Q?Q07VYO"?JE L8'.51>MH>D9( M:%.-DT;5'U93M6QP_%IJ*-'*J($UP/MXK?SC_$+&7]%T436 <+QNL_R,K1_; MA=2%8U;X,679!F!9)J4)/WF70[8FGXAVOT==]CVP8YFQ%64U7M(T-]"_.Z&]E-8Y^5#QEXC]66W6'?XM8IPA.])[_ZUS,'#WA(#;2C=,%*2 M"IJ$*341XVAW'5MI:K[EYK">_S=8HD/NXOND/1A^/6/,R\(6@ YO50)9U0Y> M_.0YMW%-[M]W 'J84/\U\-/VGWO#7> MB9G,>,>(A^2F3M,?CM7R#>.N&;OS7-[4.UN;5<]Y:+O,?+L?ZFM590QF5A^R M.>@TH %GJ']8F?S1%2$&!O:5,37L\H#OLV:5:Q07+M]U0H8W?1&TL(0X&R%P.6Q5/61"88R8P >-------------------------------Cut-Here-------------------------------< ---------------------------------------------------------------------- / e||) Lyndon J Clarke Edinburgh Parallel Computing Centre e||) \ \ c||c Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk c||c / ---------------------------------------------------------------------- From owner-mpi-comm@CS.UTK.EDU Tue Mar 2 15:16:27 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09733; Tue, 2 Mar 93 15:16:27 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24112; Tue, 2 Mar 93 15:14:04 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 2 Mar 1993 15:14:02 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24104; Tue, 2 Mar 93 15:13:57 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA26860 (5.65c/IDA-1.4.4 for ); Tue, 2 Mar 1993 14:13:54 -0600 From: William Gropp Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA08480; Tue, 2 Mar 93 14:13:52 CST Date: Tue, 2 Mar 93 14:13:52 CST Message-Id: <9303022013.AA08480@godzilla.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: A proposal for buffer descriptors The following postscript file contains a more detailed account of the buffer descriptors that we discussed at the last meeting. These allow one to send non-contiguous structures of mixed typed data between heterogeneous machines. Bill and Rusty %!PS-Adobe-2.0 %%Creator: dvips, version 5.4 (C) 1986-90 Radical Eye Software %%Title: bd.dvi %%Pages: 8 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: tex.pro /TeXDict 200 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR}B /@letter{/vsize 10 N}B /@landscape{ /isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{/vsize 15.5531 N }B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0 ]N /df{/sf 1 N /fntrx FMat N df-tail}B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0 ]N df-tail}B /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder}N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[ }B /E{pop nn dup definefont setfont}B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /ch-width{ch-data dup length 5 sub get}B /ch-height{ch-data dup length 4 sub get}B /ch-xoff{128 ch-data dup length 3 sub get sub}B /ch-yoff{ch-data dup length 2 sub get 127 sub}B /ch-dx{ch-data dup length 1 sub get}B /ctr 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image}imagemask restore}B /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}B /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}B /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put}for}B /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 8 string N /v{/ruley X /rulex X V}B /V{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}B /a{moveto}B /delta 0 N /tail{ dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail}B /c{-4 M}B /d{ -3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t {p 4 w}B /w{0 rmoveto}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 1 1 df0 D E /Fb 46 123 df<003FE3F801F03F1C03C03E3E07C07C3E0F807C3E0F807C1C0F807C000F80 7C000F807C000F807C000F807C00FFFFFFC0FFFFFFC00F807C000F807C000F807C000F807C000F 807C000F807C000F807C000F807C000F807C000F807C000F807C000F807C000F807C000F807C00 7FE1FFC07FE1FFC01F1D809C1C>11 D<78FCFCFCFC7806067D850D>46 D<00600001E0000FE000 FFE000F3E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E000 03E00003E00003E00003E00003E00003E00003E00003E00003E0007FFF807FFF80111B7D9A18> 49 D<07F8001FFE00383F80780FC0FC07C0FC07E0FC03E0FC03E07803E00007E00007C00007C0 000F80001F00001E0000380000700000E0000180600300600600600800E01FFFC03FFFC07FFFC0 FFFFC0FFFFC0131B7E9A18>I<03F8001FFE003C1F003C0F807C07C07E07C07C07C03807C0000F 80000F80001E00003C0003F800001E00000F800007C00007C00007E03007E07807E0FC07E0FC07 E0FC07C0780F80781F001FFE0007F800131B7E9A18>I<000180000380000780000F80001F8000 3F80006F8000CF80008F80018F80030F80060F800C0F80180F80300F80600F80C00F80FFFFF8FF FFF8000F80000F80000F80000F80000F80000F8001FFF801FFF8151B7F9A18>I<000380000003 80000007C0000007C0000007C000000FE000000FE000001FF000001BF000001BF0000031F80000 31F8000061FC000060FC0000E0FE0000C07E0000C07E0001803F0001FFFF0003FFFF8003001F80 03001F8006000FC006000FC00E000FE00C0007E0FFC07FFEFFC07FFE1F1C7E9B24>65 DI<001FE02000FFF8E003F80FE007C003E00F8001E01F0000E03E0000E03E0000607E000060 7C000060FC000000FC000000FC000000FC000000FC000000FC000000FC000000FC0000007C0000 607E0000603E0000603E0000C01F0000C00F80018007C0030003F80E0000FFFC00001FE0001B1C 7D9B22>IIII<000FF008007FFE3801FC07F807E001F80F8000781F0000783F000038 3E0000387E0000187C000018FC000000FC000000FC000000FC000000FC000000FC000000FC007F FFFC007FFF7C0001F87E0001F83E0001F83F0001F81F0001F80F8001F807E001F801FC07F8007F FE78000FF818201C7D9B26>I73 D76 DII<003FE00001F07C0003C01E000F800F801F0007C01E0003C03E0003E07E0003 F07C0001F07C0001F0FC0001F8FC0001F8FC0001F8FC0001F8FC0001F8FC0001F8FC0001F8FC00 01F87C0001F07E0003F07E0003F03E0003E03F0007E01F0007C00F800F8003C01E0001F07C0000 3FE0001D1C7D9B24>II82 D<07F8201FFEE03C07E07801E07000E0F000E0F00060F00060F80000FE0000FFE0007FFE 003FFF003FFF800FFFC007FFE0007FE00003F00001F00000F0C000F0C000F0C000E0E000E0F001 C0FC03C0EFFF0083FC00141C7D9B1B>I<7FFFFFE07FFFFFE0781F81E0701F80E0601F8060E01F 8070C01F8030C01F8030C01F8030C01F8030001F8000001F8000001F8000001F8000001F800000 1F8000001F8000001F8000001F8000001F8000001F8000001F8000001F8000001F8000001F8000 001F800007FFFE0007FFFE001C1C7E9B21>III89 D<0FF8001C1E003E0F803E07803E07C01C07C00007C0007FC007E7C01F07C03C07C07C07 C0F807C0F807C0F807C0780BC03E13F80FE1F815127F9117>97 DI<03FC000E0E001C1F003C1F00781F00780E00F80000F80000F80000F80000F80000F800 007800007801803C01801C03000E0E0003F80011127E9115>I<000FF0000FF00001F00001F000 01F00001F00001F00001F00001F00001F00001F001F9F00F07F01C03F03C01F07801F07801F0F8 01F0F801F0F801F0F801F0F801F0F801F07801F07801F03C01F01C03F00F0FFE03F9FE171D7E9C 1B>I<01FC000F07001C03803C01C07801C07801E0F801E0F801E0FFFFE0F80000F80000F80000 7800007C00603C00601E00C00F038001FC0013127F9116>I<03F8F00E0F381E0F381C07303C07 803C07803C07803C07801C07001E0F000E0E001BF8001000001800001800001FFF001FFFC00FFF E01FFFF07801F8F00078F00078F000787000707800F01E03C007FF00151B7F9118>103 D<1E003F003F003F003F001E00000000000000000000000000FF00FF001F001F001F001F001F00 1F001F001F001F001F001F001F001F001F00FFE0FFE00B1E7F9D0E>105 D107 DIII<01FC000F07801C01C03C01E07800F07800F0F800F8F800F8F800F8F800F8F8 00F8F800F87800F07800F03C01E01E03C00F078001FC0015127F9118>II114 D<1FD830786018E018E018F000FF807FE07F F01FF807FC007CC01CC01CE01CE018F830CFC00E127E9113>I<0300030003000300070007000F 000F003FFCFFFC1F001F001F001F001F001F001F001F001F001F0C1F0C1F0C1F0C0F08079803F0 0E1A7F9913>II119 DII<3FFF803C1F00303F00 303E00607C0060FC0060F80001F00003F00007E00007C1800F81801F81801F03803E03007E0700 7C0F00FFFF0011127F9115>I E /Fc 58 126 df<03800007E0000FE0001E70001C70001C7000 1C70001C77E01CE7E01DE7E00FC7000F8E000F0E001E0E003F1C007F1C00739C00E3F800E1F800 E0F1C0E0F1C071F9C07FFFC03F9F801E070013197F9816>38 D<00E001E0038007000E001C001C 0038003800700070007000E000E000E000E000E000E000E000E000E00070007000700038003800 1C001C000E000700038001E000E00B217A9C16>40 DI<01C00001C00001C00001C00071C700F9CF807FFF001F FC0007F00007F0001FFC007FFF00F9CF8071C70001C00001C00001C00001C00011127E9516>I< 387C7E7E3E0E1E1C78F060070B798416>44 D<70F8F8F8700505788416>46 D<03E0000FF8001FFC001E3C00380E00780F00700700700700E00380E00380E00380E00380E003 80E00380E00380E00380F00780700700700700780F003C1E001E3C001FFC000FF80003E0001119 7E9816>48 D<01800380038007800F807F80FF8073800380038003800380038003800380038003 80038003800380038003807FF87FFC7FF80E197C9816>I<07E0001FF8003FFC00783E00E00700 F00780F00380600380000380000380000700000700000E00001C0000380000700000E00001C000 0380000F00001E03803803807FFF80FFFF807FFF8011197E9816>I<07E0001FF8003FFC00781E 00780700300700000700000700000E00003E0007FC0007F00007FC00001E000007000003000003 80000380600380F00380E00700781E003FFC001FF80007E00011197E9816>I<007C0000FC0000 DC0001DC00039C00039C00071C000F1C000E1C001E1C003C1C00381C00781C00F01C00FFFFE0FF FFE0FFFFE0001C00001C00001C00001C00001C0001FFC001FFC001FFC013197F9816>I<07F000 1FFC003FFE007C1F00F00780E00380E00380E003807007007C1F001FFC0007F0001FFC003C1E00 700700F00780E00380E00380E00380F007807007007C1F003FFE001FFC0007F00011197E9816> 56 D<70F8F8F870000000000000000070F8F8F8700512789116>58 D<387C7C7C380000000000 00000038787C7C3C1C1C3870E0400618799116>I<7FFF00FFFF80FFFF80000000000000000000 000000000000FFFF80FFFF807FFF00110B7E9116>61 D<7FF800FFFE007FFF001C0F001C07801C 03801C03801C03801C07801C07001FFF001FFE001FFE001C1F001C03801C03C01C01C01C01C01C 01C01C01C01C03C01C07807FFF80FFFF007FFC0012197F9816>66 D<7FF800FFFE007FFF001C0F 001C07801C03C01C01C01C01C01C01E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00 E01C01C01C01C01C03C01C07801C0F807FFF00FFFE007FF8001319809816>68 D<7FFFC0FFFFC07FFFC01C01C01C01C01C01C01C01C01C00001C00001C1C001C1C001FFC001FFC 001FFC001C1C001C1C001C00001C00E01C00E01C00E01C00E01C00E07FFFE0FFFFE07FFFE01319 7F9816>I<7F1FC0FFBFE07F1FC01C07001C07001C07001C07001C07001C07001C07001FFF001F FF001FFF001C07001C07001C07001C07001C07001C07001C07001C07001C07007F1FC0FFBFE07F 1FC013197F9816>72 DI76 DI<7E1FC0FF3FE07F1FC01D07001D87001D87001D87001DC7001DC7001CC7001CC7001C E7001CE7001CE7001C67001C67001C77001C77001C37001C37001C37001C17007F1F00FF9F007F 0F0013197F9816>I<1FFC003FFE007FFF00780F00F00780E00380E00380E00380E00380E00380 E00380E00380E00380E00380E00380E00380E00380E00380E00380F00780F00780780F007FFF00 3FFE001FFC0011197E9816>I<7FF800FFFE007FFF001C0F801C03801C03C01C01C01C01C01C01 C01C03C01C03801C0F801FFF001FFE001FF8001C00001C00001C00001C00001C00001C00001C00 007F0000FF80007F000012197F9816>I<7FE000FFF8007FFC001C1E001C0F001C07001C07001C 07001C07001C0F001C1E001FFC001FF8001FFC001C1C001C0E001C0E001C0E001C0E001C0E201C 0E701C0E707F07E0FF87E07F03C014197F9816>82 D<07E3001FFF003FFF00781F00F00700E007 00E00700E00000F000007800003F80001FF00007FC0000FE00000F000007000003800003806003 80E00380E00700F80F00FFFE00FFFC00C7F00011197E9816>I<7FFFE0FFFFE0FFFFE0E0E0E0E0 E0E0E0E0E0E0E0E000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000 E00000E00000E00000E00000E00007FC000FFE0007FC0013197F9816>I86 D91 D93 D95 D<1FE0003FF0007FF800783C00300E00000E00000E00 03FE001FFE003E0E00700E00E00E00E00E00E00E00783E007FFFE03FE7E00F83E013127E9116> 97 D<7E0000FE00007E00000E00000E00000E00000E00000E3E000EFF000FFF800F83C00F00E0 0E00E00E00700E00700E00700E00700E00700E00700E00E00F01E00F83C00FFF800EFF00063C00 1419809816>I<03F80FFC1FFE3C1E780C7000E000E000E000E000E000F000700778073E0E1FFC 0FF803F010127D9116>I<003F00007F00003F0000070000070000070000070003C7000FF7001F FF003C1F00780F00700700E00700E00700E00700E00700E00700E00700700F00700F003C1F001F FFE00FE7F007C7E014197F9816>I<03E00FF81FFC3C1E780E7007E007FFFFFFFFFFFFE000E000 700778073C0F1FFE0FFC03F010127D9116>I<001F00007F8000FF8001E78001C30001C00001C0 007FFF00FFFF00FFFF0001C00001C00001C00001C00001C00001C00001C00001C00001C00001C0 0001C00001C0003FFE007FFF003FFE0011197F9816>I<03E3C007F7E00FFFE01C1CC0380E0038 0E00380E00380E00380E001C1C000FF8001FF0001BE0003800001800001FFC001FFF003FFF8078 03C0E000E0E000E0E000E0E000E07001C07C07C03FFF800FFE0003F800131C7F9116>I<7E0000 FE00007E00000E00000E00000E00000E00000E3C000EFE000FFF000F87800F03800E03800E0380 0E03800E03800E03800E03800E03800E03800E03800E03807FC7F0FFE7F87FC7F01519809816> I<018003C003C0018000000000000000007FC07FC07FC001C001C001C001C001C001C001C001C0 01C001C001C001C07FFFFFFF7FFF101A7D9916>I<7E0000FE00007E00000E00000E00000E0000 0E00000E7FE00E7FE00E7FE00E0F000E1E000E3C000E78000EF0000FF0000FF8000FBC000F1E00 0E0E000E07000E07807F87F0FFCFF07F87F01419809816>107 DII<7E3C00FEFE007FFF000F87800F03800E03800E03800E03800E 03800E03800E03800E03800E03800E03800E03807FC7F0FFE7F87FC7F01512809116>I<03E000 0FF8001FFC003C1E00780F00700700E00380E00380E00380E00380E00380F00780700700780F00 3C1E001FFC000FF80003E00011127E9116>I<7E3E00FEFF007FFF800F83C00F00E00E00E00E00 700E00700E00700E00700E00700E00700E00E00F01E00F83C00FFF800EFF000E3C000E00000E00 000E00000E00000E00000E00007FC000FFE0007FC000141B809116>I114 D<0FEC3FFC7FFCF03CE01CE01C70007F801FF007F8003C600EE0 0EF00EF81EFFFCFFF8C7E00F127D9116>I<0300000700000700000700000700007FFF00FFFF00 FFFF00070000070000070000070000070000070000070000070100070380070380070380078700 03FE0001FC0000F80011177F9616>I<7E1F80FE3F807E1F800E03800E03800E03800E03800E03 800E03800E03800E03800E03800E03800E03800E0F800FFFF007FBF803E3F01512809116>I<7F 1FC0FF1FE07F1FC01C07001E0F000E0E000E0E000E0E00071C00071C00071C00071C0003B80003 B80003B80001F00001F00000E00013127F9116>II<7F1FC07F3FC07F1FC00F1C00073C0003B80003F00001F00000E00001E00001F00003B8 00073C00071C000E0E007F1FC0FF3FE07F1FC013127F9116>I<7F1FC0FF9FE07F1FC01C07000E 07000E0E000E0E00070E00071C00071C00039C00039C0003980001B80001B80000F00000F00000 F00000E00000E00000E00001C00079C0007BC0007F80003F00003C0000131B7F9116>I<3FFFC0 7FFFC07FFFC0700780700F00701E00003C0000780001F00003E0000780000F00001E01C03C01C0 7801C0FFFFC0FFFFC0FFFFC012127F9116>I<001F80007F8000FF8001E00001C00001C00001C0 0001C00001C00001C00001C00001C00001C00003C0007F8000FF0000FF00007F800003C00001C0 0001C00001C00001C00001C00001C00001C00001C00001C00001E00000FF80007F80001F801120 7E9C16>I<7C0000FF0000FF800003C00001C00001C00001C00001C00001C00001C00001C00001 C00001C00001E00000FF00007F80007F8000FF0001E00001C00001C00001C00001C00001C00001 C00001C00001C00001C00003C000FF8000FF00007C000011207E9C16>125 D E /Fd 59 123 df<007E1F0001C1B1800303E3C00703C3C00E03C1800E01C0000E01C0000E01 C0000E01C0000E01C0000E01C000FFFFFC000E01C0000E01C0000E01C0000E01C0000E01C0000E 01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C000 0E01C0007F87FC001A1D809C18>11 D<007E0001C1800301800703C00E03C00E01800E00000E00 000E00000E00000E0000FFFFC00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01 C00E01C00E01C00E01C00E01C00E01C00E01C00E01C07F87F8151D809C17>I<003F07E00001C0 9C18000380F018000701F03C000E01E03C000E00E018000E00E000000E00E000000E00E000000E 00E000000E00E00000FFFFFFFC000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C00 0E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C 000E00E01C000E00E01C000E00E01C007FC7FCFF80211D809C23>14 D<6060F0F0F8F868680808 08080808101010102020404080800D0C7F9C15>34 D<004000800100020006000C000C00180018 00300030007000600060006000E000E000E000E000E000E000E000E000E000E000E000E0006000 60006000700030003000180018000C000C00060002000100008000400A2A7D9E10>40 D<800040002000100018000C000C000600060003000300038001800180018001C001C001C001C0 01C001C001C001C001C001C001C001C0018001800180038003000300060006000C000C00180010 002000400080000A2A7E9E10>I<01800180018001804182F18F399C0FF003C003C00FF0399CF1 8F4182018001800180018010127E9E15>I<60F0F0701010101020204080040C7C830C>44 DI<60F0F06004047C830C>I<00010003000600060006000C000C000C00 18001800180030003000300060006000C000C000C0018001800180030003000300060006000C00 0C000C00180018001800300030003000600060006000C000C00010297E9E15>I<03C00C301818 300C300C700E60066006E007E007E007E007E007E007E007E007E007E007E007E007E007600660 06700E300C300C18180C3007E0101D7E9B15>I<030007003F00C7000700070007000700070007 0007000700070007000700070007000700070007000700070007000700070007000F80FFF80D1C 7C9B15>I<07C01830201C400C400EF00FF80FF807F8077007000F000E000E001C001C00380070 006000C00180030006010C01180110023FFE7FFEFFFE101C7E9B15>I<60F0F060000000000000 0000000060F0F06004127C910C>58 D<60F0F0600000000000000000000060F0F0701010101020 204080041A7C910C>I<000600000006000000060000000F0000000F0000000F00000017800000 178000001780000023C0000023C0000023C0000041E0000041E0000041E0000080F0000080F000 0180F8000100780001FFF80003007C0002003C0002003C0006003E0004001E0004001E000C001F 001E001F00FF80FFF01C1D7F9C1F>65 DI<001F808000E0618001 801980070007800E0003801C0003801C00018038000180780000807800008070000080F0000000 F0000000F0000000F0000000F0000000F0000000F0000000F00000007000008078000080780000 80380000801C0001001C0001000E000200070004000180080000E03000001FC000191E7E9C1E> II70 D72 DI77 DI80 D<07E0801C1980300580700380600180E00180E00080E0 0080E00080F00000F800007C00007FC0003FF8001FFE0007FF0000FF80000F800007C00003C000 01C08001C08001C08001C0C00180C00180E00300D00200CC0C0083F800121E7E9C17>83 D<7FFFFFC0700F01C0600F00C0400F0040400F0040C00F0020800F0020800F0020800F0020000F 0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F000000 0F0000000F0000000F0000000F0000000F0000000F0000000F0000001F800003FFFC001B1C7F9B 1E>IIII91 D<08081010202040404040808080808080B0B0 F8F8787830300D0C7A9C15>II<1FC000307000783800781C00301C00001C 00001C0001FC000F1C00381C00701C00601C00E01C40E01C40E01C40603C40304E801F87001212 7E9115>97 DI<07E00C301878307870306000E000E000E0 00E000E000E00060007004300418080C3007C00E127E9112>I<003F0000070000070000070000 070000070000070000070000070000070000070003E7000C1700180F00300700700700600700E0 0700E00700E00700E00700E00700E00700600700700700300700180F000C370007C7E0131D7E9C 17>I<03E00C301818300C700E6006E006FFFEE000E000E000E00060007002300218040C1803E0 0F127F9112>I<00F8018C071E061E0E0C0E000E000E000E000E000E00FFE00E000E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E007FE00F1D809C0D>I<00038003C4C0 0C38C01C3880181800381C00381C00381C00381C001818001C38000C300013C000100000300000 1800001FF8001FFF001FFF803003806001C0C000C0C000C0C000C06001803003001C0E0007F800 121C7F9215>II<18003C003C0018000000000000000000 000000000000FC001C001C001C001C001C001C001C001C001C001C001C001C001C001C001C001C 00FF80091D7F9C0C>I107 DIII<03F0000E1C00180600300300700380600180E001C0E0 01C0E001C0E001C0E001C0E001C06001807003803003001806000E1C0003F00012127F9115>I< FC7C001D86001E03001C01801C01C01C00C01C00E01C00E01C00E01C00E01C00E01C00E01C01C0 1C01C01C01801E03001D06001CF8001C00001C00001C00001C00001C00001C00001C0000FF8000 131A7F9117>I<03C1000C3300180B00300F00700700700700E00700E00700E00700E00700E007 00E00700600700700700300F00180F000C370007C7000007000007000007000007000007000007 00000700003FE0131A7E9116>II<1F9030704030C010C010E010F8007F803FE00FF000 F880388018C018C018E010D0608FC00D127F9110>I<04000400040004000C000C001C003C00FF E01C001C001C001C001C001C001C001C001C001C101C101C101C101C100C100E2003C00C1A7F99 10>IIII<7F8FF00F03800F030007020003840001C80001D80000F000007000007800 00F800009C00010E00020E000607000403801E07C0FF0FF81512809116>II<7F FC70386038407040F040E041C003C0038007000F040E041C043C0C380870087038FFF80E127F91 12>I E /Fe 14 118 df<000FF83F00007FFDFFC001F81FE3E003E03F87E007C03F87E00F803F 07E00F803F03C00F801F00000F801F00000F801F00000F801F00000F801F00000F801F0000FFFF FFFC00FFFFFFFC000F801F00000F801F00000F801F00000F801F00000F801F00000F801F00000F 801F00000F801F00000F801F00000F801F00000F801F00000F801F00000F801F00000F801F0000 0F801F00000F801F00000F801F00000F801F00007FF0FFF0007FF0FFF00023237FA221>11 D<387CFEFEFE7C3807077C8610>46 D<00180000780001F800FFF800FFF80001F80001F80001F8 0001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F8 0001F80001F80001F80001F80001F80001F80001F80001F80001F8007FFFE07FFFE013207C9F1C >49 D66 D<00FF8007FFE00F83F01F03F03E03 F07E03F07C01E07C0000FC0000FC0000FC0000FC0000FC0000FC00007C00007E00007E00003E00 301F00600FC0E007FF8000FE0014167E9519>99 D<0001FE000001FE0000003E0000003E000000 3E0000003E0000003E0000003E0000003E0000003E0000003E0000003E0000003E0001FC3E0007 FFBE000F81FE001F007E003E003E007E003E007C003E00FC003E00FC003E00FC003E00FC003E00 FC003E00FC003E00FC003E00FC003E007C003E007C003E003E007E001E00FE000F83BE0007FF3F C001FC3FC01A237EA21F>I<00FE0007FF800F87C01E01E03E01F07C00F07C00F8FC00F8FC00F8 FFFFF8FFFFF8FC0000FC0000FC00007C00007C00007E00003E00181F00300FC07003FFC000FF00 15167E951A>I<1C003E007F007F007F003E001C000000000000000000000000000000FF00FF00 1F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F00FFE0FF E00B247EA310>105 D<00FE0007FFC00F83E01E00F03E00F87C007C7C007C7C007CFC007EFC00 7EFC007EFC007EFC007EFC007EFC007E7C007C7C007C3E00F81F01F00F83E007FFC000FE001716 7E951C>111 DI114 D<0FF3003FFF00781F00600700E00300E003 00F00300FC00007FE0007FF8003FFE000FFF0001FF00000F80C00780C00380E00380E00380F007 00FC0E00EFFC00C7F00011167E9516>I<01800001800001800001800003800003800007800007 80000F80003F8000FFFF00FFFF000F80000F80000F80000F80000F80000F80000F80000F80000F 80000F80000F80000F81800F81800F81800F81800F81800F830007C30003FE0000F80011207F9F 16>II E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin @letter /letter where {pop letter} if %%EndSetup %%Page: 1 1 bop 210 105 a Fe(1.)18 b(Bu\013er)g(descriptors)210 216 y Fd(The)h (message-passing)f(routines)h(describ)q(ed)h(ab)q(o)o(v)o(e)e(are)h (su\016cien)o(t)g(to)f(write)h(relativ)o(ely)e(e\016cien)o(t)i(pro-)210 286 y(grams.)25 b(In)17 b(man)o(y)e(applications,)h(ho)o(w)o(ev)o(er,)h(the)h (data)e(to)h(b)q(e)g(sen)o(t)h(or)e(receiv)o(ed)i(is)f(not)g(in)f(consequtiv) o(e)210 355 y(lo)q(cations)11 b(in)g(memory)m(.)k(While)c(an)o(y)g(data,)g (ho)o(w)o(ev)o(er)h(laid)f(out)h(in)f(memory)m(,)e(ma)o(y)g(b)q(e)k(sen)o(t)f (or)g(receiv)o(ed)h(this)210 425 y(w)o(a)o(y)m(,)h(there)j(are)e(a)g(n)o(um)o (b)q(er)g(of)f(sp)q(ecial)i(cases)h(for)d(whic)o(h)i(sp)q(ecial)f(arrangemen) o(ts)g(ma)o(y)e(b)q(e)j(made)e(b)o(y)h(the)210 495 y(op)q(erating)e(system)g (or)f(the)i(hardw)o(are.)k(These)c(include)f(constan)o(t,)g(non-unit)g (strides)h(\(useful)f(for)g(sending)210 565 y(a)k(ro)o(w)g(from)e(a)i(F)m (ortran)g(arra)o(y\),)g(scatter/gathers)i(\(useful)f(for)e(sparse)j(v)o (ectors\),)g(and)d(non-con)o(tiguous)210 634 y(blo)q(c)o(ks)j(\(useful)g(for) g(sending)g(structures)j(with)c(p)q(oin)o(ters\).)34 b(These)20 b(op)q(erations)g(ma)o(y)d(b)q(e)i(pro)o(vided)g(b)o(y)210 704 y(the)c(lo)o(w-lev)o(el)e(hardw)o(are)i(and)g(soft)o(w)o(are)g(b)q (ecause)h(they)f(are)g(common)d(and)i(b)q(ecause)j(imp)q(ortan)o(t)c(p)q (erfor-)210 774 y(mance)d(optimizations)e(can)j(b)q(e)g(made)e(for)h(them,)g (particularly)g(the)h(elimination)d(of)i(unnecessary)i(memory)210 844 y(motion.)22 b(F)m(or)16 b(Unix)g(programmers,)e(the)j(non-con)o(tiguous) e(blo)q(c)o(k)h(case)h(is)f(simply)e(the)j(analogue)e(of)g(the)210 913 y(routines)d Fc(readv)e Fd(and)g Fc(writev)p Fd(.)16 b(Sev)o(eral)c (message-passing)e(implemen)o(tatio)o(ns)f(already)i(pro)o(vide)f(non-unit) 210 983 y(stride)h(v)o(ector)h(op)q(erations,)f(and)f(researc)o(h)j(w)o(ork)d (with)g(scatter/gather)j(op)q(erations)d(has)h(sho)o(wn)g(that)g(these)210 1053 y(are)j(also)g(of)f(great)h(p)q(oten)o(tial)f(b)q(ene\014t.)272 1123 y(MPI)e(pro)o(vides)g(access)i(to)d(all)g(of)g(these)i(in)e(an)o(y)g (com)o(bination)e(b)o(y)j(w)o(a)o(y)f(of)g(a)g(general)h(data-arrangemen)o(t) 210 1192 y(descriptor.)210 1319 y Fb(1.1.)16 b(Organization)210 1415 y Fd(A)i(data-arrangemen)o(t)e(is)i(de\014ned)h(in)e(2)g(steps:)27 b(\(1\))18 b(create)h(a)f(data-arrangemen)o(t)e(descriptor)j(and)f(\(2\))210 1485 y(add)13 b(descriptions)i(of)e(the)h(data)f(arrangemen)o(t)g(to)g(it.)k (A)d(data-arrangemen)o(t)e(descriptor)j(is)e(created)i(with)210 1555 y Fc(MPI)p 279 1555 14 2 v 15 w(NewBD)p Fd(.)e(The)i(routines)g Fc(MPI)p 741 1555 V 15 w(BDVec)p Fd(,)e Fc(MPI)p 957 1555 V 15 w(BDBlk)p Fd(,)h(and)g Fc(MPI)p 1255 1555 V 15 w(BDIdx)f Fd(add)i(descriptions)g(for)f(non-unit)210 1624 y(stride)e(v)o(ectors,)h(blo) q(c)o(ks,)e(and)h(indexed)g(v)o(ectors)g(\(these)i(are)d(describ)q(ed)j(in)d (more)f(detail)h(on)h(the)g(man)e(pages)210 1694 y(for)17 b(these)i (routines\).)29 b(A)17 b(send)h(or)f(receiv)o(e)i(op)q(eration)e(then)h(tak)o (es)g(the)f(data-arrangemen)o(t)g(descriptor)210 1764 y(instead)d(of)f(the)i (\(bu\013er,size\))g(that)f(the)h(send)g(and)e(receiv)o(e)j(routines)e(use)h (for)e(con)o(tiguous)h(data.)210 1890 y Fb(1.2.)i(V)l(ectors)210 1987 y Fd(The)d(\\v)o(ector")g(data-arrangemen)o(t)f(allo)o(ws)f(sending)i (and)g(receiving)g(of)f(non-con)o(tiguous)g(data)g(where)i(the)210 2057 y(elemen)o(ts)g(are)g(separated)h(b)o(y)f(a)f(constan)o(t)i(distance)g (or)e(stride.)272 2126 y(T)m(o)g(add)h(a)g(v)o(ector)g(to)g(the)h(data)e (arrangemen)o(t,)g(use)210 2222 y Fc(MPI_BDVec\()20 b(da,)h(buffer,)f (elmlen,)h(stride,)f(elmcount,)g(datatype)g(\))210 2318 y Fd(The)14 b(parameters)g(are:)210 2415 y Fb(bu\013er)251 b Fd(p)q(oin)o(ter)14 b(to)g(bu\013er)210 2514 y Fb(elmlen)236 b Fd(size)15 b(of)e(eac)o(h)h (elemen)o(t,)f(in)h(b)o(ytes)210 2614 y Fb(stride)254 b Fd(distance)15 b(b)q(et)o(w)o(een)g(successiv)o(e)h(elemen)o(ts,)d(in)h(b)o(ytes)p eop %%Page: 2 2 bop 210 105 a Fb(elmcoun)o(t)182 b Fd(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts) 210 208 y Fb(datat)o(yp)q(e)190 b Fd(datat)o(yp)q(e)14 b(\(e.g.,)f Fc(MPI)p 932 208 14 2 v 15 w(INT)p Fd(\))272 311 y(F)m(or)h(example,)e(if)h (a)h(F)m(ortran)f(arra)o(y)h(is)g(declared)g(as)210 414 y Fc(double)21 b(precision)e(a\(10,20\))210 517 y Fd(and)14 b(the)g(ro)o(w)g Fc(a\(3,1:20\))e Fd(is)i(to)f(b)q(e)i(sen)o(t,)f(use)210 620 y Fc(bd)21 b(=)h(MPI_BDNew\(1\))210 690 y(MPI_BDVec\()e(bd,)h(a\(3,1\),)f(8,) i(8*10,20)e(\))210 759 y(MPI_bsend\()g(to,)h(tag,)g(bd,)g(context)f(\))210 862 y Fd(Here)15 b(w)o(e)f(assume)g(that)g(a)f(double)h(precision)g(quan)o (tit)o(y)f(tak)o(es)i(eigh)o(t)e(b)o(ytes.)210 990 y Fb(1.3.)j(Blo)q(c)o(ks) 210 1087 y Fd(Blo)q(c)o(ks)d(of)f(con)o(tiguous)g(data)g(ma)o(y)f(b)q(e)i (added)g(to)f(the)h(data)f(arrangemen)o(t)g(with)g Fc(MPI)p 1564 1087 V 15 w(BDBlk)p Fd(.)17 b(The)c(format)210 1156 y(is)210 1259 y Fc(MPI_BDBlk\()20 b(bd,)h(buffer,)f(len,)h(datatype)f(\))272 1362 y Fd(Multiple)12 b(uses)i(of)e Fc(MPI)p 640 1362 V 15 w(BDBlk)g Fd(ma)o(y)e(b)q(e)j(used)h(to)e(send)i(complicated)d(data)h (structures.)21 b(F)m(or)12 b(example,)210 1432 y(to)i(send)h(the)f (structure)210 1535 y Fc(struct)21 b({)297 1605 y(int)g(n[3];)297 1674 y(double)g(a[4];)297 1744 y(char)65 b(s;)297 1814 y(})22 b(S;)210 1917 y Fd(w)o(e)14 b(can)g(use)h(the)f(follo)o(wing)e(co)q(de:)210 2020 y Fc(bd)21 b(=)h(MPI_BDNew\()e(3)h(\);)210 2090 y(MPI_BDBlk\()f(bd,)h (&S.n[0],)f(3)i(*)f(sizeof\(int\),)85 b(MPI_INT)20 b(\);)210 2159 y(MPI_BDBlk\()g(bd,)h(&S.a,)86 b(4)22 b(*)f(sizeof\(double\),)e(MPI_DBL) h(\);)210 2229 y(MPI_BDBlk\()g(bd,)h(&S.s,)86 b(sizeof\(char\),)150 b(MPI_OTHER)20 b(\);)210 2299 y(MPI_bsend\()g(to,)h(tag,)g(bd,)g(context)f (\);)210 2402 y Fd(T)m(o)13 b(receiv)o(e)i(the)g(structure,)g(the)g(exact)f (same)f(co)q(de)i(is)f(used,)g(with)g(the)g Fc(MPI)p 1432 2402 V 15 w(bsend)f Fd(replaced)i(with)210 2505 y Fc(MPI_brecv\()20 b(source,)g(tag,)h(bd,)g(context,)f(recv_handle)g(\);)p eop %%Page: 3 3 bop 210 105 a Fb(1.4.)16 b(Indexed)210 202 y Fd(Another)e(t)o(yp)q(e)g(of)f (non-con)o(tiguous)g(comm)o(unicatio)o(n)e(pattern)j(that)f(is)h(common)c(in) j(applications)g(is)g(\\in-)210 271 y(dexed.")18 b(F)m(or)11 b(example,)f(if)g(elemen)o(ts)h(of)f(one)i(v)o(ector,)g(indexed)g(b)o(y)e (another,)i(are)g(used.)18 b(These)12 b(are)g(de\014ned)210 341 y(with)i Fc(MPI)p 374 341 14 2 v 15 w(BDIdx)p Fd(.)j(The)d(format)e(is) 210 444 y Fc(MPI_BDIdx\()20 b(bd,)h(buffer,)f(idxarray,)g(numitems,)g (datatype)g(\))p eop %%Page: 4 4 bop 210 105 a Fb(NAME)314 175 y(MPI)p 413 175 15 2 v 17 w(NewBD)117 b Fd(Create)15 b(a)f(new)g(bu\013er)h(descriptor)210 245 y Fb(SYNOPSIS)314 314 y Fd(MPI)p 397 314 13 2 v 15 w(BD)f(*MPI)p 589 314 V 15 w(NewBD\()h(n)o(umitems)c(\))314 384 y(in)o(t)i(n)o(umitems;)210 454 y Fb(INPUT)j(AR)o(GUMENTS)314 524 y Fd(n)o(umitesm)215 b(Maxim)o(um)10 b(n)o(um)o(b)q(er)j(of)h(bu\013er)h(descriptions)g(for)e(the) i(bu\013er)f(descriptor.)210 593 y Fb(DESCRIPTION)314 663 y(MPI)p 413 663 15 2 v 17 w(NewBD)h Fd(creates)h(a)e(new)g(bu\013er)i(descriptor.)k (The)15 b(v)n(alue)f(of)f Fc(numitems)g Fd(is)h(the)h(maxim)n(um)314 733 y(n)o(um)o(b)q(er)c(of)g(bu\013er)h(descriptions)h(that)e(can)h(b)q(e)g (added)g(with)g(the)g(routines)g Fc(MPI)p 1574 733 14 2 v 15 w(BDVec)p Fd(,)e Fc(MPI)p 1787 733 V 16 w(BDBlk)p Fd(,)314 802 y(and)j Fc(MPI)p 463 802 V 16 w(BDIdx)p Fd(.)210 872 y Fb(RETURN)j(V)-5 b(ALUE)314 942 y(MPI)p 413 942 15 2 v 17 w(NewBD)14 b Fd(returns)i(a)d(p)q(oin)o(ter)h(to)g(a)g(bu\013er)g(descriptor,)h(or)f(n)o (ull)f(if)g(an)h(error)g(o)q(ccurs.)p eop %%Page: 5 5 bop 210 105 a Fb(NAME)314 175 y(MPI)p 413 175 15 2 v 17 w(BD)o(V)l(ec)136 b Fd(Add)14 b(a)g(stride)g(v)o(ector)h(to)f(a)f(bu\013er)i(descriptor)210 245 y Fb(SYNOPSIS)314 314 y Fd(in)o(t)e(MPI)p 460 314 13 2 v 15 w(BD)o(V)m(ec\()i(b)q(d,)f(bu\013er,)g(elmlen,)e(stride,)i(elmcoun)o(t,) e(datat)o(yp)q(e)j(\);)314 384 y(MPI)p 397 384 V 15 w(BD)f(b)q(d;)314 454 y(c)o(har)g(*bu\013er;)314 524 y(in)o(t)f(elmlen,)f(stride,)i(elmcoun)o (t,)f(datat)o(yp)q(e;)210 593 y Fb(INPUT)j(AR)o(GUMENTS)314 663 y Fd(b)q(d)347 b(Data)13 b(bu\013er)314 733 y(bu\013er)290 b(p)q(oin)o(ter)14 b(to)g(bu\013er)314 802 y(elmlen)275 b(size)15 b(of)e(eac)o(h)h(elemen)o(t,)f(in)h(b)o(ytes)314 872 y(stride)293 b(distance)15 b(b)q(et)o(w)o(een)g(successiv)o(e)h(elemen)o(ts,)d(in)h(b)o (ytes)314 942 y(elmcoun)o(t)228 b(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)314 1012 y(datat)o(yp)q(e)234 b(datat)o(yp)q(e)14 b(\(e.g.,)f Fc(MPI)p 1055 1012 14 2 v 15 w(INT)p Fd(\))210 1081 y Fb(DESCRIPTION)314 1151 y(MPI)p 413 1151 15 2 v 17 w(BD)o(V)l(ec)d Fd(adds)g(a)g(description)h (of)f(a)g(v)o(ector)h(\(with)f(non-unit-stride\))h(to)f(a)g(previously)g (created)314 1221 y(bu\013er)15 b(descriptor.)210 1291 y Fb(RETURN)h(V)-5 b(ALUE)314 1360 y(MPI)p 413 1360 V 17 w(BD)o(V)l(ec)13 b Fd(returns)j(0,)d (or)h Fa(\000)p Fd(1)g(if)f(an)g(error)i(o)q(ccurs.)p eop %%Page: 6 6 bop 210 105 a Fb(NAME)314 175 y(MPI)p 413 175 15 2 v 17 w(BDBlk)137 b Fd(Add)14 b(a)g(con)o(tiguous)f(blo)q(c)o(k)h(to)g(a)f(bu\013er)i (descriptor)210 245 y Fb(SYNOPSIS)314 314 y Fd(in)o(t)e(MPI)p 460 314 13 2 v 15 w(BDBlk\()h(b)q(d,)g(bu\013er,)g(len,)g(datat)o(yp)q(e)g (\))314 384 y(MPI)p 397 384 V 15 w(BD)g(b)q(d;)314 454 y(c)o(har)g (*bu\013er;)314 524 y(in)o(t)f(len,)h(datat)o(yp)q(e;)210 593 y Fb(INPUT)i(AR)o(GUMENTS)314 663 y Fd(b)q(d)347 b(Data)13 b(bu\013er)314 733 y(bu\013er)290 b(p)q(oin)o(ter)14 b(to)g(bu\013er)314 802 y(len)341 b(n)o(um)o(b)q(er)13 b(of)h(b)o(ytes)314 872 y(datat)o(yp)q(e)234 b(datat)o(yp)q(e)14 b(\(e.g.,)f Fc(MPI)p 1055 872 14 2 v 15 w(INT)p Fd(\))210 942 y Fb(DESCRIPTION)314 1012 y(MPI)p 413 1012 15 2 v 17 w(BDBlk)18 b Fd(adds)h(a)f(description)i(of)e (a)h(con)o(tiguous)f(blo)q(c)o(k)h(to)f(a)h(previously)g(created)h(bu\013er) 314 1081 y(descriptor.)210 1151 y Fb(RETURN)c(V)-5 b(ALUE)314 1221 y(MPI)p 413 1221 V 17 w(BDBlk)13 b Fd(returns)i(0,)e(or)h Fa(\000)p Fd(1)g(if)f(an)h(error)h(o)q(ccurs.)p eop %%Page: 7 7 bop 210 105 a Fb(NAME)314 175 y(MPI)p 413 175 15 2 v 17 w(BDIdx)140 b Fd(Add)14 b(a)g(scatter/gather)h(to)f(a)g(bu\013er)h(descriptor)210 245 y Fb(SYNOPSIS)314 314 y Fd(in)o(t)e(MPI)p 460 314 13 2 v 15 w(BDIdx\()h(b)q(d,)g(bu\013er,)h(idxarra)o(y)m(,)d(n)o(umitems,)f(datat) o(yp)q(e)j(\))314 384 y(MPI)p 397 384 V 15 w(BD)g(b)q(d;)314 454 y(c)o(har)g(*bu\013er;)314 524 y(in)o(t)f(idxarra)o(y[],)f(n)o(umitems,)f (datat)o(yp)q(e;)210 593 y Fb(INPUT)16 b(AR)o(GUMENTS)314 663 y Fd(b)q(d)347 b(Data)13 b(bu\013er)314 733 y(bu\013er)290 b(p)q(oin)o(ter)14 b(to)g(bu\013er)314 802 y(idxarra)o(y)242 b(p)q(oin)o(ter)14 b(to)g(index)g(arra)o(y)314 872 y(n)o(umitems)215 b(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)314 942 y(datat)o(yp)q(e)234 b(datat)o(yp)q(e)14 b(\(e.g.,)f Fc(MPI)p 1055 942 14 2 v 15 w(INT)p Fd(\))210 1012 y Fb(DESCRIPTION)314 1081 y(MPI)p 413 1081 15 2 v 17 w(BDIdx)d Fd(adds)g(a)h(description)g(of)f(a)g(scatter)i(or)e (gather)h(blo)q(c)o(k)f(to)h(a)f(previously)g(created)i(bu\013er)314 1151 y(descriptor.)19 b(\(It)14 b(is)g(a)g(gather)g(when)g(used)h(in)e(a)h (send)h(and)f(a)f(scatter)j(when)e(used)h(in)e(a)h(receiv)o(e.\))210 1221 y Fb(RETURN)i(V)-5 b(ALUE)314 1291 y(MPI)p 413 1291 V 17 w(BDBlk)13 b Fd(returns)i(0,)e(or)h Fa(\000)p Fd(1)g(if)f(an)h(error)h(o)q (ccurs.)p eop %%Page: 8 8 bop 210 105 a Fb(NAME)314 175 y(MPI)p 413 175 15 2 v 17 w(F)l(reeBD)120 b Fd(F)m(ree)15 b(a)e(bu\013er)i(descriptor)210 245 y Fb(SYNOPSIS)314 314 y Fd(in)o(t)e(MPI)p 460 314 13 2 v 15 w(F)m(reeBD\()i(b)q(d)f(\))314 384 y(MPI)p 397 384 V 15 w(BD)g(b)q(d;)210 454 y Fb(INPUT)i(AR)o(GUMENTS)314 524 y Fd(b)q(d)347 b(Data)13 b(bu\013er)210 593 y Fb(DESCRIPTION)314 663 y(MPI)p 413 663 15 2 v 17 w(F)l(reeBD)g Fd(frees)i(a)f(bu\013er)g (descriptor)i(created)f(with)f Fc(MPI)p 1354 663 14 2 v 15 w(NewBD)p Fd(.)210 733 y Fb(RETURN)i(V)-5 b(ALUE)314 802 y(MPI)p 413 802 15 2 v 17 w(NewBD)14 b Fd(returns)i(0,)d(or)h Fa(\000)p Fd(1)f(if)g(an)h(error)h(o)q(ccurs.)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-comm@CS.UTK.EDU Tue Mar 2 16:03:25 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10713; Tue, 2 Mar 93 16:03:25 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26723; Tue, 2 Mar 93 16:01:55 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 2 Mar 1993 16:01:54 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26694; Tue, 2 Mar 93 16:01:49 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA28558 (5.65c/IDA-1.4.4 for ); Tue, 2 Mar 1993 15:01:47 -0600 Received: from localhost by donner.mcs.anl.gov (4.1/GCF-5.8) id AA00600; Tue, 2 Mar 93 15:01:45 CST Message-Id: <9303022101.AA00600@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: Revised Multi-level MPI Suggestion Date: Tue, 02 Mar 1993 15:01:44 -0600 From: Rusty Lusk Before the last meeting we sent out a suggestion on how the potential complexity of the evolving MPI library could be managed by dividing the routines into layers. The idea was to provide a very simple interface at the highest layer, which we call Level 4, and provide all of the flexibility necessary at lower levels, which we called 1 and 2. Since then the February meeting has taken place, and some of our thinking has changed. The basic idea still seems like a good one to us, so here is a new proposal for multiple levels in MPI. At the February meeting, Marc Snir's detailed point-to-point proposal basically covered what we called level 1 (where there are a small number of routines, including "init" routines for operations) and level 2 (where sets of the level 1 routines are bundled together for efficiency, but almost all functionality is preserved). There also seemed to be general agreement that there was no need to have separate routines for supporting communication among heterogeneous machines; rather, contiguous buffers could be specified by (addr,datatype,numitems) instead of (addr,len). The concept of buffer-descriptor (details in another message) would be used to describe noncontiguous buffers and buffers of mixed data types. Therefore it seems unnecessary to have *two* higher levels, since one of the differences between our level 3 and level 4 was the specification of data types. At the same time, it still seems useful to us to have a level of MPI routines that are simpler than the full set specified in Marc's document. We therefore suggest here a level 3 for MPI that imposes a set of restrictions in the interest of simplicity. (So, for example, it will not use buffer descriptors, but restrict users to contiguous buffers of data of the same type.) It corresponds roughly to "current practice", but adds thread safety (no "last message" semantics). It assumes the existence of only one default "all" group and one "universal" context, so that these need not be part of the calls. It does include nonblocking sends and receives, but simplified versions of wait, status, and probe. Users requiring more flexibility can use the routines of level 2 instead. Level 3 MPI routines: (syntax only specified for definiteness) numids() returns number of processes myrank() returns caller's rank in group "all" bsend(dest,tag,buf,datatype,numitems) blocking send (until buffer is empty) brecv(from,tag,buf,datatype,maxitems, blocking receive, with separate actual_from,actual_tag, output arguments to handle wild-card actual_numitems) input arguments Note that lots of programs have been written with just these four calls. handle = nsend(dest,tag,buf,datatype,numitems) non-blocking send handle = nrecv(from,tag,buf,datatype,maxitems, non-blocking receive actual_from,actual_tag, output arguments to handle wild-card actual_numitems) input arguments status(handle) check status of non-blocking operation wait(handle) wait for a non-blocking operation probe(from,tag,actual_from,actual_tag,actual_length) look in queue for matching message, get length so can allocate buffer This is a larger set than the "four-function" interface we proposed as level 4 earlier, but not much, and has the advantage that one can write correct programs even on machines with no buffering. It corresponds roughly to current practice in that most existing programs can be easily converted to this level. It gains simplicity and ease of use by not offering access to MPI's advanced features like noncontiguous buffers of various and mixed types, explicit handle management, high-performance protocols, and elaborate status and wait calls. It has no explicit mention of groups or contexts. The collective operations seem to be getting complicated too, and would benefit from a restricted, simple level. One way to do this would be to add a (blocking) barrier and collective operations in their simplest forms, without explicit mention of group-id's. These Level 3 operations add nothing to MPI that cannot be done at lower levels. However, it seems to us that adopting these simplified versions into the standard will make it easier for many people to use and hence pave the way for its acceptance. Any comments? Bill Gropp & Rusty Lusk From owner-mpi-comm@CS.UTK.EDU Tue Mar 2 16:17:36 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11504; Tue, 2 Mar 93 16:17:36 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27414; Tue, 2 Mar 93 16:16:32 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 2 Mar 1993 16:16:31 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27402; Tue, 2 Mar 93 16:16:25 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA29060 (5.65c/IDA-1.4.4 for ); Tue, 2 Mar 1993 15:16:23 -0600 Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA00668; Tue, 2 Mar 93 15:16:21 CST Message-Id: <9303022116.AA00668@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: Task identifiers Date: Tue, 02 Mar 93 15:16:20 CST From: Rusty Lusk One issue that has been postponed several times is that of what "process id's" will look like and what they will mean. We would like to offer a suggestion. In this short note we discuss some of the issues and make a concrete proposal. Since the word "process" is so overloaded, we use the word "task" here (possibly only a marginal improvement). We define a task as a program unit that receives a message. For most systems, this is a single process. In a multi-threaded environment, this may be a process or a thread. To allow a process to decide whether to use threads to handle some operations, we believe that a threads within a process should have the same task id; different threads can be identifed by different tags, groups, or contexts. For example, an obvious way to provide a nonblocking user-defined operation is to execute that operation in a separate thread. This should not require a separate task id. The most important issue is whether the number of tasks is fixed or not. If the number is fixed, a sequential numbering of the tasks is possible and simple. However, the trend seems to be toward allowing for the dynamic modificaiton of the number of tasks, and this is impossible if a sequential numbering is used. To avoid restricting MPI to a fixed collection of tasks, we advocate using an opaque task id. This raises the issue of how the task ids of other processes are determined. We suggest that the rank of the task in a specified group be used as a cannonical representation. This rank is required to be in the range 0 to (size of group)-1. The usual arithmetic operations can be performed on the rank. For example, you can define your right neighbor in a ring as ((myrank + 1) mod p), where p is the number of tasks in the ring. Proposal In general, task ids should be an opaque type. A routine gets its own task id with MPI_MyTid. To get the task ids of other tasks, an application may use MPI_TidFromRank(group, rank). A routine gets its own rank with MPI_MyRank(group). It is the task id that is used as a source or destination in the point-to-point message-passing routines. In Fortran it might as well be an integer. Thus the new MPI routines being proposed are: MPI_MyTid() Get caller's task id MPI_MyRank(group) Get caller's rank in group MPI_TidFromRank(group,rank) Get task id of a task from its rank in a group MPI_RankFromTid(tid,group) Get rank in a group from a task id An exception is made for the level 3 routines described in the recent earlier message; these only have one group and we propose that task id and rank in the "all" group be the same for them. This is current common practice. Discussion Note that since a task id is an opaque type, it cannot be sent to another task and used as a task id on that other task. Only the rank is meaningful across tasks. Making the task id an opaque type allows an implementation to optimize the task ids. For example, an implementation may choose to make the task ids the physical node ids used by the hardware. In addition, it encourages users to use library routines to determine neighbors rather than arithmetic on the task id (note that they can still get a neighbor by adding one to their rank). For example, the "best" neighbor in a virtual ring depends on details of the underlying hardware and software; calling a routine such as MPI_NbrInRing to get the neighbor is a more portable way of getting a good neighbor than MPI_RankToTask(MPI_MyRank(group)+1, group ). Bill Gropp and Rusty Lusk From owner-mpi-comm@CS.UTK.EDU Tue Mar 2 16:33:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12253; Tue, 2 Mar 93 16:33:15 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28299; Tue, 2 Mar 93 16:32:14 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 2 Mar 1993 16:32:13 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28287; Tue, 2 Mar 93 16:32:04 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA29509 (5.65c/IDA-1.4.4 for ); Tue, 2 Mar 1993 15:32:01 -0600 Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA00744; Tue, 2 Mar 93 15:31:59 CST Message-Id: <9303022131.AA00744@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: MPI Job startup Date: Tue, 02 Mar 93 15:31:58 CST From: Rusty Lusk We feel that the MPI standard should provide enough information to specify an entire, source-code portable program. To do this, it is necessary to specify how parallel programs start and end. This is the analogue of PROGRAM MAIN ... END in Fortran and main(argc,argv) {...exit(rc); }. This proposal does not deal with methods for spawning tasks during the execution of an MPI program, or for handling explicitly MIMD models (beyond the usual way of starting a Turing machine on all nodes and sending the appropriate tape). This is not meant to suggest that MPI not have such interfaces, just that we are only proposing an SPMD model at this time. Issues It should be possible to write a program that is source-language portable to any system, particularly if it requires only a simple interface to the operating system. An example is an environment that starts up a program with a fixed number of processors. Without something like this, it is not possible to write a portable MPI program. However, no standard should restrict programs to a particular interface. For example, some applications may want to dynamically added and remove tasks during the execution of an application. We leave it to others on the committee to propse a mechanism for this case. Below, we discuss three different approaches; we note that all of these have been tried in some existing system or systems. The seemingly simplest approach is to use an explicit initialization routine, MPI_Init. There are several problems with this approach. The major one is that the parallel part of the program is the code that "follows" the MPI_Init. This is potentially error prone; further, it can be awkward to implement on some systems. This is the way p4 does it. Another approach is to insist that the entire program be run in parallel; a simple way to do this is to replace the "main" program with a routine. In C, this could be MPI_Main( argc, argv ). The linking step then provides a "main" that handles all appropriate startup (including the processing of command line arguments where appropriate for information on the number of tasks etc.) and then calls the user's MPI_main routine. Yet another approach is to provide a routine that runs a user-defined routine in parallel. For example, MPI_Call( argc, argv, routine ) can be used to run routine( argc, argv ) in parallel. A generalization of this would allow the running of a particular routine on a defined group: MPI_Call( groupid, argc, argv, routine ). This is more flexible than the MPI_Main approach but retains the simplicity of running a well-defined program unit (the routine) in parallel. All three of these approaches are currently in use. These are not the most general methods. For example, these provide no mechanism for dynamically creating (remote spawn) tasks on other processors. Also, there is no direct support for MIMD programming. Finally, there is no support for adding an already-running process to a group of running processes. Proposal We propose the MPI_Call form. A slight variation would allow the use of a general argument descriptor instead of argc/argv. This would allow the more convenient passing arguments to the routine. We have used this mechanism (in Bill's Chameleon system) to write programs for nX, the CM-5, the nCube, and networks of workstations using both p4 and PVM, and it has worked well for us. The point is that all source code is portable. The only thing that is different is the shell command by which you start your program. Note that in this case, program termination is handled by having (a) the routine exit and (b) the calling routine exit. Thus an MPI program would look something like this: main(argc,argv) int argc; char **argv; { ret_code = MPI_Call( argc, argv, worker ); exit( irc ); } int worker( argc, argv ) int argc; char **argv; { ... if (error condition) return 1; ... return 0; } In Fortran it would like something like: program main external worker irc = MPI_call(worker) stop end integer function worker ... worker = 0 return end It may be useful to have a more drastic exit. We suggest MPI_exitall(code); the code here is an exit code that may be ignored by the invoking environment (but UNIX-like environments should treat it like exit(code)). In the case of heterogeneous environments, it is necessary to specify the set of workstations to be used. Our only requirement is that this be done, in the C/POSIX case, through command-line arguments (thus maintaining source-code portability). We could specify an extensible, common format for specifying the machines, executables, and resource limits, but, to avoid additional debate, we suggest only that the command line option -mpi be reserved for future extensions. Bill & Rusty From owner-mpi-comm@CS.UTK.EDU Tue Mar 2 18:09:29 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA15496; Tue, 2 Mar 93 18:09:29 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02859; Tue, 2 Mar 93 18:07:43 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 2 Mar 1993 18:07:41 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02851; Tue, 2 Mar 93 18:07:38 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA08222; Tue, 2 Mar 93 23:07:34 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA17367; Tue, 2 Mar 93 16:06:22 MST Date: Tue, 2 Mar 93 16:06:22 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303022306.AA17367@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: Revised Multi-level MPI Suggestion A Suggestion for probe() in the New Rusty and Bill Scheme I'm a bit confused about probe() in Rusty and Bill's latest proposal: ... > probe(from,tag,actual_from,actual_tag,actual_length) > look in queue for matching message, > get length so can allocate buffer > ... > Any comments? > > Bill Gropp & Rusty Lusk > I believe that there was general agreement that probe() should provide the caller with a "handle" to avoid "hidden state" (the thread-safety issue). (The probe() in the current point-to-point proposal does this.) Should the probe() proposed above return "handle"?: handle = probe(from,tag,actual_from,actual_tag,actual_length) If so, we need to add a "recv()" that has "handle" as an argument at Level 3 (basically precv() as I understand it). (This "handle" was also occaisionally referred to as "lock" in the last meeting.) Also, the current proposals all require that a receiving process know the data type(s) in an incoming message before that message is received. This means that datatype can be included in probe() and "numitems" can be used instead of "length". Syntax is then consistent with other routines in the new "Level 3". Here's a counter-proposal (in the spirit of Rusty and Bill's syntax): handle = probe(from, tag, datatype, actual_from, actual_tag, actual_numitems) precv(handle, buf) The use of "handle" guarantees that input parameters to probe() do not need to be repeated in precv(). "maxitems" (as in brecv()) doesn't seem to be necessary since the caller will know how much memory is required before buf is specified. It might be a good idea to add "unlock(handle)" to Level 3. (OR remove probe() from Level 3.) Of course my preference is to dump the whole mess, including buffer descriptor, into an opaque data structure (as in Leslie's and my "message capsule" proposal on Jan 15 in mpi-pt2pt :-). Tom Henderson NOAA Forecast Systems Lab hender@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Wed Mar 3 16:35:49 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11369; Wed, 3 Mar 93 16:35:49 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07168; Wed, 3 Mar 93 16:34:27 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Mar 1993 16:34:25 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07160; Wed, 3 Mar 93 16:34:22 -0500 Date: Wed, 3 Mar 93 21:34:17 GMT Message-Id: <6940.9303032134@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Task identifiers To: Rusty Lusk In-Reply-To: Rusty Lusk's message of Tue, 02 Mar 93 15:16:20 CST Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-comm@cs.utk.edu Question: How do Rusty and Bill write so many proposals? Answer: Surely they never sleep:-) (Oh well, perhaps I won't get featured on the jokes newsgroups after all.) But seriously, fellow committee members, I recall that the contexts subcommittee is about to prepare one or more proposals which will have considerable bearing on this subject, so perhaps we should again defer "task identifiers" until this has happened. One crucial point raised (again), which I do not see evidence of having really been addressed, is whether MPI will provide a reasonable interface for a static process model, or for a dynamic process model. We have to decide what we want to do here, and I think we ought to clearly state this as potential future users whom I have spoken to are asking just this kind of question. What are the arguments for and against a dynamic process model? To get started: a) The MPP programming world has a whole lot more experience with the static model, and as such it is better aligned with common practice. b) We anticipate people moving toward use of a dynamic process model. c) There are a whole load of issues in the dynamic model which complicate implementations and imply lesser efficiency. d) In order for programs to be portable in the MPI + dynamic process model, the interface to dynamic process control has to be uniform as well as the interface to message passing. Would MPI really consider addressing the question of a uniform interface to process control at this point? At least in the static case the process creation is not part of the program and we can "ignore" it. /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Wed Mar 3 17:06:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12272; Wed, 3 Mar 93 17:06:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08663; Wed, 3 Mar 93 17:04:22 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Mar 1993 17:04:20 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08641; Wed, 3 Mar 93 17:04:18 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA05267; Wed, 3 Mar 93 16:01:25 CST Date: Wed, 3 Mar 93 16:01:25 CST From: Tony Skjellum Message-Id: <9303032201.AA05267@Aurora.CS.MsState.Edu> To: lusk@mcs.anl.gov, lyndon@epcc.ed.ac.uk Subject: Re: Task identifiers Cc: mpi-comm@cs.utk.edu I think that we agreed to defer dynamic issues to a future MPI standard level, but I do agree that this is never guaranteed to happen, as Rusty has said. The lack of control for processes, and the lack of dynamic group syntax/semantics worries me. They will almost certainly appear as non-portable extensions. Perhaps the original N-month limit was too short to permit the best possible standard. - Tony From owner-mpi-comm@CS.UTK.EDU Thu Mar 4 08:24:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27412; Thu, 4 Mar 93 08:24:07 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22467; Thu, 4 Mar 93 08:22:22 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Mar 1993 08:22:15 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from gmdzi.gmd.de by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22379; Thu, 4 Mar 93 08:22:07 -0500 Received: from f1neuman.gmd.de (f1neuman) by gmdzi.gmd.de with SMTP id AA02215 (5.65c/IDA-1.4.4 for ); Thu, 4 Mar 1993 14:20:32 +0100 Received: by f1neuman.gmd.de id AA16175; Thu, 4 Mar 1993 14:21:56 GMT Date: Thu, 4 Mar 1993 14:21:56 GMT From: Rolf.Hempel@gmd.de Message-Id: <9303041421.AA16175@f1neuman.gmd.de> To: mpi-comm@cs.utk.edu Subject: Process ids Cc: gmap10@f1neuman.gmd.de recently there has been a discussion on process identifiers, groups, and dynamic process creation. I would like to add a few comments. 1. It is not so much the dynamic creation of new processes which calls for something like opaque process ids, but rather the dynamic removal of processes. In the former case process ids can be added at the end (n, n+1, ...) without changing the 0,..,n-1 numbering of the existing processes. If a process terminates, however, there is a gap in the numbering scheme. On the other hand, the same gap occurs in the group ranking which also Bill and Rusty want to keep. So, I don't see how the opaque process ids resolve the problem. 2. As Bill and Rusty say at the end of their note, inquiries like "give me my right neighbor" should be done by using the process topology functions, and not on the ranks themselves. Only this way an optimization of the process ordering is possible. The ranks mainly serve to define the buffer ordering in collective communication. 3. At our last meeting we already saw that the MPI time schedule is becoming tight. I don't think it would be wise to deal with dynamic process creation in MPI-1, and as far as I remember there was already some consensus on that. Of course, in the design of the static interface we have to keep in mind that dynamic extensions will be added later. It should not be seen as a problem if on the basis of MPI-1 non-standard extensions will emerge. After all, only this kind of experimenting can make a later standardization possible. Rolf Hempel From owner-mpi-comm@CS.UTK.EDU Sat Mar 6 15:38:10 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03415; Sat, 6 Mar 93 15:38:10 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12502; Sat, 6 Mar 93 15:37:25 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sat, 6 Mar 1993 15:37:24 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12470; Sat, 6 Mar 93 15:36:41 -0500 Date: Sat, 6 Mar 93 20:36:37 GMT Message-Id: <390.9303062036@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: dynamic processes and dynamic groups To: mpi-pt2pt@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-comm@cs.utk.edu Hi again Well, we certainly are having some fun now in this discussion of static vs dynamic processes in MPI. This is as much about dynamic groups as dynamic processes, and the latter part is really about MPI as a standards committee. I have a lot of sympathy with John Flower's concerns. There appears to be reasonable consensus among those contributing to the discussion that in a world process model where processes dynamically appear and dissapear in an asynchronous fashion an enumeration of processes is not a useful way of identifying processes - what sense could we make of the enumeration? An opaque type process (or task) identifier is better. The same considerations surely lead us to the observation that in a world group model where processes dynamically join and leave groups in an asynchronous fashion an enumeration of members of a group is not a useful way of identifying members - what sense could we make of the enumeration? Rolf previously pointed this out on mpi-comm - March 4, subject "Process ids". What are asynchronously created (terminated) processes to do? I guess they either join (leave) an existing group, or create (destroy) a group for their own purposes. The primitives for these two are really process group resizing (or destruction/recreation) and process group creation (termination). I guess their must be heavyweight PVM users who have other experiences of usage of dynamic process creation/termination - Al?. These thoughts/questions, however interesting and discussion provoking they may or may not be, do not take us any further on answering some of the BIG questions about what MPI will be, regarding for example: a) Process model; static v dynamic; ... b) Group model; static v dynamic; ... I've previously stated that I'd vote for static process model in (a). This "seemed to be" the intent of MPI (although I couldn't find it written down and recorded as agreed), would be useful, and could be agreed upon within the MPI time schedule. This is not to say that I think limitation to a static process model would be the most useful thing upon which to agree to describe a 'de facto' standard. I have pretty well the same opinions regarding process groups. I'd like us all agree on how we're going to move forward and get the details of the standard described, as I'm concerned that at the current rate we may not. I'll broadly support John Flower's suggestions, primarily because I don't see any others, and I'll do my best to implement whatever strategy we can agree on. Final thought, perhaps we should move the discussion of MPI strategy to mpi-comm. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Mon Mar 8 08:01:03 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04198; Mon, 8 Mar 93 08:01:03 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19241; Mon, 8 Mar 93 07:59:25 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Mar 1993 07:59:24 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19232; Mon, 8 Mar 93 07:59:19 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA03238 (5.65c/IDA-1.4.4 for ); Mon, 8 Mar 1993 07:59:13 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA10283; Mon, 8 Mar 93 12:59:09 GMT Date: Mon, 8 Mar 93 12:59:09 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9303081259.AA10283@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02380; Mon, 8 Mar 93 12:56:19 GMT To: lusk@mcs.anl.gov Cc: mpi-comm@cs.utk.edu In-Reply-To: Rusty Lusk's message of Tue, 02 Mar 93 15:31:58 CST <9303022131.AA00744@donner.mcs.anl.gov> Subject: MPI Job startup Content-Length: 2990 > We feel that the MPI standard should provide enough information to > specify an entire, source-code portable program. I agree. MPI_INIT style ============== > The seemingly simplest approach is to use an explicit initialization > routine, MPI_Init. There are several problems with this approach. > The major one is that the parallel part of the program is the code > that "follows" the MPI_Init. This is potentially error prone; > further, it can be awkward to implement on some systems. This is > the way p4 does it. I agree that this is simplest. I also prefer it to the others. I don't see the "major problem" outlined above. The whole of the code is parallel even before the MPI_INIT, it's just that it can't communicate yet. I'd expect code like this (given an I/O everywhere model) to execute fine int main(int argc,char ** argv) { printf("Hello world\n"); } and print as many "Hello world" messages as the number of nodes on which I ran it. It also seems to me to be quite an easy way to allow the dynamicness everyone seems to want. The newly created process starts up, and then attaches to the existing MPI environment at the time it calls MPI_INIT. In my view, then, what happens is 1) By some unspecified means a number of processes start to execute 2) They decide to collaborate by executing MPI_INIT [If dynamic process creation is permitted 2a) New processes are forked/execed 2b) The new processes join the existing collaboration by executing MPI_INIT ] 3) They each exit (by calling exit), or you can have a drastic exit if you like. MPI_CALL style ============== I don't think I understand what this is intended to mean, which may be why I don't like it ! Therefore please can you explain it to me. 1) Is the idea that there is one user process which then executes MPI_CALL at which point all of the other processes are created,and magically arrive in a user subroutine, or do all the processes execute main ? 2) If all the processes execute main, what can they do before calling MPI_CALL ? 3) What global state is accessible to the MPI_CALLed routine ? e.g. (your example with minor change...) extern int foo; /* Of course this could be an arbitrarily complex data structure ! */ main(argc,argv) int argc; char **argv; { foo = time(); /* Something which may differ on each node ! */ ret_code = MPI_Call( argc, argv, worker ); exit( irc ); } int worker( argc, argv ) int argc; char **argv; { printf("%d: %d\n", MPI_MYTID(), foo); ... if (error condition) return 1; ... return 0; } At the moment I definitely prefer MPI_INIT ! -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Mon Mar 8 09:19:50 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05513; Mon, 8 Mar 93 09:19:50 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22555; Mon, 8 Mar 93 09:18:33 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Mar 1993 09:18:28 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22546; Mon, 8 Mar 93 09:18:01 -0500 Date: Mon, 8 Mar 93 14:17:41 GMT Message-Id: <1439.9303081417@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: MPI Job startup To: Rusty Lusk , mpi-comm@cs.utk.edu In-Reply-To: Rusty Lusk's message of Tue, 02 Mar 93 15:31:58 CST Reply-To: lyndon@epcc.ed.ac.uk Hi there Spring has arrived in Scotland :-) For the thrid say this year we have beautiful sunshine :-) The snow on the Caringorms is meling fast :-( Now it's MPI job startup. Thanks to Bill and Rusty for starting this one off. The MPI_INIT style is familiar. It makes it explicit that the bit of the program before MPI_INIT is executing in parallel, and no MPI services can be used before MPI_INIT. As Jim correctly points out. It documents to the user what s/he is using. The trouble is, that the user will almost always end up calling MPI_INIT right after entry into main, because what can the user usefully do before MPI_INIT? We should consider providing this approach. I really don't like the MPI_CALL approach. It serves to confuse the user. It looks like there is one main running, and the function reference in MPI_CALL magically runs in parallel. What of use can the user do before MPI_CALL? It looks to the user like the static (global) data is all accessible everywhere. I don't think this is intended, and I don't think this is what should be provided. What about the other approach, in which the user provides a "virtual main" and the MPI library supplies a "actual main", which calls the virtual main. I guess the virtual main calls MPI_INIT and then calls the actual main. This removes the necessity for the user to call MPI_INIT, and means that s/he cannot make the mistake of attempting to use MPI services before they are available. We *might* be able to get this past C programmers. I believe that we would have enormous problems trying to get this past crusty fortran programmers --- "What, you mean I have to write a subroutine instead of a program?". Wait up though, this is what is happening anyway. Your process never starts in your own main. Your "actual main" is a "virtual main", and the compiler/stdlib/runtime executes a preamble to main. FOURTH suggestion ----------------- There is no MPI initialisation procedure. The user program is "attatched to MPI" and can use MPI services as soon as the user program starts to execute. This means that we require system vendors to embed any MPI initialisation his/her implementation requires in the preamble before main. Dead simple. Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Mon Mar 8 09:25:38 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05746; Mon, 8 Mar 93 09:25:38 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22903; Mon, 8 Mar 93 09:24:39 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Mar 1993 09:24:38 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22895; Mon, 8 Mar 93 09:24:31 -0500 Date: Mon, 8 Mar 93 14:24:19 GMT Message-Id: <1449.9303081424@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: MPI Job startup To: lyndon@epcc.ed.ac.uk, Rusty Lusk , mpi-comm@cs.utk.edu In-Reply-To: L J Clarke's message of Mon, 8 Mar 93 14:17:41 GMT Reply-To: lyndon@epcc.ed.ac.uk Ooops - small oversight. > This means that we require system vendors to embed any MPI > initialisation his/her implementation requires in the preamble before > main. > Not quite true, Lyndon, you missed something :-) On the other hand all of the MPI service routines could do one check of a static boolean to see if MPI has been initialised, and if not then call an initialisation routine which inverts the static boolean. I just prefer if it happened for sure in the preamble. Best Wishes Lyndon ps replying to my own mail, hmmm :-) /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Mon Mar 8 10:51:20 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09115; Mon, 8 Mar 93 10:51:20 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28354; Mon, 8 Mar 93 10:50:07 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Mar 1993 10:50:06 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28346; Mon, 8 Mar 93 10:50:04 -0500 Received: from id.wmich.edu (id.cs.wmich.edu) by cs.wmich.edu (4.1/SMI-4.1) id AA00863; Mon, 8 Mar 93 10:49:41 EST Date: Mon, 8 Mar 93 10:49:41 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9303081549.AA00863@cs.wmich.edu> To: mpi-comm@cs.utk.edu hi; To add to the "thread" on MPI init. I think there should be some form of MPI init call before any other MPI call. This allows MPI implementations over other systems, without a small but built in overhead. I don't feel religous about this though. Assuming an MPI init call of some form is added, the only questions I would have would be - 1) Does each thread need to make the call - or is one call per process OK? and 2) In the case of only requiring on call per process, Is it OK to make more than one call to the init routine - as in the case of each thread in a process making a call before starting. If there is any state information kept on a per thread basis - than one call per thread may be reasonable. I know we tried to keep all state per thread information out (which I view as very reasonable), but unless it is a 100% requirement, one MPI init call per process may not be enough. john From owner-mpi-comm@CS.UTK.EDU Mon Mar 8 15:11:52 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17797; Mon, 8 Mar 93 15:11:52 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13204; Mon, 8 Mar 93 15:08:17 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Mar 1993 15:08:15 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13156; Mon, 8 Mar 93 15:08:08 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Mon, 8 Mar 93 12:04 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA16386; Mon, 8 Mar 93 12:02:56 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA14497; Mon, 8 Mar 93 12:02:53 PST Date: Mon, 8 Mar 93 12:02:53 PST From: d39135@sodium.pnl.gov Subject: RE: MPI Job startup To: lusk@mcs.anl.gov, lyndon@epcc.ed.ac.uk, mpi-comm@cs.utk.edu Cc: d39135@sodium.pnl.gov Message-Id: <9303082002.AA14497@sodium.pnl.gov> X-Envelope-To: mpi-comm@cs.utk.edu Here are some comments I haven't seen yet about MPI job startup suggestions. Lyndon Clarke's FOURTH suggestion: > There is no MPI initialisation procedure. The user program is > "attatched to MPI" and can use MPI services as soon as the user program > starts to execute. > > This means that we require system vendors to embed any MPI > initialisation his/her implementation requires in the preamble before > main. This approach has the disadvantage that it requires intimate collaboration with the vendor's runtime language-support library. The interfaces to those libraries are usually undocumented and subject to change without notice. This means that one needs vendor cooperation to write and maintain an MPI implementation using this approach. From the MPI user's standpoint, "FOURTH suggestion" seems indistinguishable from the earlier proposal that there be no explicit initialization call, and that initialization be done by having each MPI routine check a static boolean. A minor problem with the static boolean suggestion is that then the initialization routine does not have access to those parts of the environment passed as (argc,argv). Such access is sometimes useful. (For example... TCGMSG's static process initializer uses command-line arguments to inform each process of its place in the grand scheme of things. In C, the command-line arguments are accessible to TCGMSG because the application has to pass them to the TCGMSG initialization procedure. (In Fortran, TCGMSG's initialization procedure is called without arguments, and it has to hunt up the argc and argv that presumably stored by Fortran initialization in global variables. See above comments about undocumented interfaces.) The MPI_Main approach (MPI does the initialization, user provides a subroutine) seems to have the disadvantage that it works for only one package. If MPI takes this approach, then I don't see how any other package FOO can too, if an application wants to use both MPI and FOO. MPI_Call has the slight problem that if another package FOO adopts the MPI_Call approach, then we end up with main (argc,argv) { MPI_main (argc,argv,myMPIinitializer); } myMPIinitializer (argc,argv) { FOO_main (argc,argv,myFOOinitializer); } myFOOinitializer (argc,argv) { myapplication (argc,argv); } compared to main (argc,argv) { MPI_Init (argc,argv); FOO_Init (argc,argv); myapplication (argc,argv); } Certainly this is not an enormous problem, but I don't see where the extra complexity buys anything. I like MPI_Init (augmented if necessary for dynamic processes). --Rik Littlefield From owner-mpi-comm@CS.UTK.EDU Mon Mar 8 15:52:41 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19445; Mon, 8 Mar 93 15:52:41 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18884; Mon, 8 Mar 93 15:51:31 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Mar 1993 15:51:30 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18876; Mon, 8 Mar 93 15:51:28 -0500 Received: from elephant by sampson.ccsf.caltech.edu with SMTP id AA06355 (5.65c/IDA-1.4.4 for mpi-comm@cs.utk.edu); Mon, 8 Mar 1993 12:51:24 -0800 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA18286; Mon, 8 Mar 93 12:44:12 PST Received: by lion.parasoft (4.1/SMI-4.1) id AA15256; Mon, 8 Mar 93 12:44:15 PST Date: Mon, 8 Mar 93 12:44:15 PST From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9303082044.AA15256@lion.parasoft> To: mpi-comm@cs.utk.edu Subject: MPI_INIT To throw in my two cents worth..... The business with argc and argv (and, of course environ) has been a thorn in the side of Express for many years. We've adopted a standard two-faced approach in which C programmers get their "main" routine renamed (silently) to exp_main which we then call from our own main. In this case argc and argv get dealt with by Express and passed to the main routine. This approach has the disease mentioned by Rik, that two or more different software suppliers may want to pull the same trick causing confusion. In fortran we went the way of MPI_INIT with no arguments. This seems to work but has the difficulty that things beyond the ken of the message passing system can be affected by it. For example, the Express program program foo write(6,*) 'I have started' call kxinit -- i.e., MPI_INIT write(6,*) 'Done with kxinit' stop end is actually wrong because the I/O system needs to know parameters that don't get set till the call to kxinit. We could, of course, enforce the rule in MPI that only MPI routines are affected by the MPI_INIT call and that, therefore, you can put anything you like before the call to MPI_INIT as long as its name doesn't start with MPI_!!!!! I dont think this is going to work in the long run since it actively discourages library vendors from building their codes with MPI routines!!! Also, in Rik's example, I think the code has to look sort of like main(argc, argv) int argc; char **argv; { MPI_INIT(&argc, &argv); FOO_INIT(&argc, &argv); /* etc... */ if we want to allow for the chance that it's actually MPI_INIT that passes down the arguments....... this is pretty ugly...... My personal vote would be for making the user rename their top level entry point: MPI_main(int argc, char **argv, char **environ) in C and MPI_MAIN in fortran Perhaps some compiler level support for potentially changing the name of the entry point so that third party software vendors could build their own "front-ends" that puts their own code first?? I appreciate that this is definitately NOT current practice and has all sorts of diseases of its own with different software suppliers providing layer upon layer of "fake" compiler front-ends. Another option that I would probably support would be bagging argc/argv altogether at the MPI level. In this case a function call with no arguments that had to be the first thing you did would be OK -- if you do it wrong you probably get a core dump! Express' attempts to deal with argc and argv date back to the days when hardware vendors didn't provide these things and we wanted to ourselves. Now all the systems I know of start up C programs apropriately and the need for Express (or some other system) to get in behind them is not so crucial...?.... A slight counter-example to my own argument, however, is that some systems give you mangled argc/argv and particularly environ. This can lead to bizarre results. (Systems which permit a remote host capability sometimes give you the environment that you would have had on the local host rather than the one you actually have on the remote host.) Jon From owner-mpi-comm@CS.UTK.EDU Mon Mar 8 20:07:09 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23824; Mon, 8 Mar 93 20:07:09 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02443; Mon, 8 Mar 93 20:06:01 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Mar 1993 20:06:00 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02433; Mon, 8 Mar 93 20:05:58 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Mon, 8 Mar 93 17:01 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA17034; Mon, 8 Mar 93 17:00:40 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA15033; Mon, 8 Mar 93 17:00:38 PST Date: Mon, 8 Mar 93 17:00:38 PST From: d39135@sodium.pnl.gov Subject: Re: MPI_INIT To: jwf@lion.Parasoft.COM, mpi-comm@cs.utk.edu Cc: d39135@sodium.pnl.gov Message-Id: <9303090100.AA15033@sodium.pnl.gov> X-Envelope-To: mpi-comm@cs.utk.edu Jon Flower says: > Also, in Rik's example, I think the code has to look sort of > like > > main(argc, argv) > int argc; > char **argv; > { > MPI_INIT(&argc, &argv); > FOO_INIT(&argc, &argv); > /* etc... */ Yeah, sort of. The example I showed was based on one version of current practice (TCGMSG). It doesn't let the INIT's alter the argument list. In general, this implies far worse ugliness than Jon's example, because it requires that applications know enough to ignore arguments intended for the INIT's. A better version of current practice is the X window system, one of whose initializers looks like XtInitialize ( ... &argc, argv) This allows deleting stuff from the argument list, but not adding. Jon's syntax goes farther and allows both. But maybe this is working the wrong issue. I said: > A minor problem with the static boolean suggestion is that then the > initialization routine does not have access to those parts of the > environment passed as (argc,argv). Such access is sometimes useful. It could also be said that the problem is with the (argc,argv) approach. I defended the (argc,argv) approach, sort of, but I confess that I don't like it very much precisely because it DOES prevent calling the MPI initializer from inside some other routine. I ran afoul of this just a few days ago, when I tried to emulate another package using TCGMSG, and discovered that my application called the other package's initializer from several procedures down. What I would *really* like (putting on my user's hat), is an explicit MPI_INIT with no arguments, like Jon suggests. I guess this means that implementors have to find some way to pass environment info other than through the argv list. Can anyone come up with a convincing reason why this is a big problem? --Rik Littlefield From owner-mpi-comm@CS.UTK.EDU Tue Mar 9 00:01:42 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27466; Tue, 9 Mar 93 00:01:42 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10468; Tue, 9 Mar 93 00:00:25 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 9 Mar 1993 00:00:24 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10455; Tue, 9 Mar 93 00:00:22 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA08899; Mon, 8 Mar 93 22:56:45 CST Date: Mon, 8 Mar 93 22:56:45 CST From: Tony Skjellum Message-Id: <9303090456.AA08899@Aurora.CS.MsState.Edu> To: jwf@lion.Parasoft.COM, mpi-comm@cs.utk.edu, d39135@sodium.pnl.gov Subject: Re: MPI_INIT; environment information Guys, What about using envp? That would provide the Unix-compatible, open-ended approach to passing information on the environment? Info could be added etc, as was suggested as a need. This is a layerable-friendly approach. ie, main(argc,argv,envp) int argc; char **argv; char **envp; 1) For Fortran, the getenv_() (or equivalent) call provides the interface. C could use the direct interface to envp, or the getenv() call. We could have mpi_getenv() to avoid conflicts with general system support, if absolutely needed to avoid POSIX interactions. 2) Handling of argc/argv A) argc, argv is made accessible to C through standard access. Made accessible to Fortran through accessor functions (eg, mpi_argc(), mpi_argv(i,string,maxlen)). B) argc, argv could be obliterated for both models (ie, left empty for C, inaccessible to Fortran). All info through mpi_getenv(). May I throw in a comment about main precursors (niam() in the BBN parlance)? This is great if supported by the linker, and a pain if not supported by the linker. If niam() is absent, then the default niam() is added, which just calls main(). If niam() is present, then the user model can choose. However, what if different programs presume the abilility to use niam(). They would not stack up... Without niam(), has to assume preprocessor substitution, or macro substitution. This requires that user uses an MPI-compatible make procedure, or compiler pseudonym, other than naked cc/f**. This turns out to be a thorny problem to handle transparently. Also, the execution model of the CM-5, for instance, does not [did not] require main to exist at all. Execution of arbitrary functions (not only main) is supported by CMMD 2.0, specifically. I don't know what CMMD 3.0 chooses to do, but you get the idea. main/program main access is an assumption true most everywhere, but not everywhere. - Tony From owner-mpi-comm@CS.UTK.EDU Tue Mar 9 15:24:37 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13894; Tue, 9 Mar 93 15:24:37 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24607; Tue, 9 Mar 93 15:14:24 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 9 Mar 1993 15:14:22 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24595; Tue, 9 Mar 93 15:14:18 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA24881; Tue, 9 Mar 93 20:13:49 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA06208; Tue, 9 Mar 93 13:12:34 MST Date: Tue, 9 Mar 93 13:12:34 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303092012.AA06208@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: MPI Goals and Scope: More Details Please? As a potential user of MPI, the discussion of I/O below disturbs me a bit and leads to a whole bunch of questions... > > We feel that the MPI standard should provide enough information to > > specify an entire, source-code portable program. > > Bill & Rusty > I agree. > > I'd expect code like this (given an I/O everywhere model) to execute > fine > > int main(int argc,char ** argv) > { > printf("Hello world\n"); > } > > and print as many "Hello world" messages as the number of nodes on > which I ran it. > > -- Jim I also agree with Bill and Rusty's statement about portability. Jim's example brings up a whole bunch of interesting problems. First, should the printf() really print one "Hello world" message per node (process)? I can think of several places where I wouldn't expect that behavior (Express "single" I/O mode for example). The next problem is "ordering". Suppose Jim's example is modified slightly: int main (int argc, char **argv) { int me; MPI_INIT(???); /* Yes, I'm casting my vote for MPI_INIT(). */ me = MPI_MYRANK(); /* another guess at syntax */ printf("Hello world from process %d\n", me); } Now what happens? Do we see first-come first-served? Do we see the "intuitive" ordering? Do we get an error message? At the last meeting a straw vote lead to the adoption of "one processor can do standard I/O". If we do this, then users will use vendor-supplied "parallel I/O" routines to get decent performance. We cannot claim that the MPI standard provides enough information to specify an entire, source-code portable program unless we include I/O. I think that there is general agreement that "parallel I/O" will not be included in MPI-1. Maybe we could re-state the portability goal: The ultimate goal of the MPI standardization effort is to provide enough information to specify an entire, source-code portable program. I certainly won't use MPI unless I believe that this will happen with some version of MPI. I think we should re-examine the scope of the MPI effort and write down what we want to accomplish with MPI-1, MPI-2, etc. in much more detail than is currently in the draft introduction. John Flower has proposed one prioritization. Here's another (from my perspective as a user): 1) Static process creation and identification (and associated environmental inquiry routines). 2) Contexts. 3) Heterogeneous communication semantics (I think we already have this). All communication routines should support heterogeneous communication. 4) Blocking send and receive, contiguous data. 5) Non-blocking send and receive, contiguous data (status(), wait(), cancel()?). 6) Creation/destruction of "static" groups (by "static" I mean processes cannot join/leave an existing group). Grid/toroidal process topology would go here if we decide to include it. 7) Blocking collective communication, contiguous data (broadcast, multiple-broadcast, concatenate, barrier, etc.). 8) "Simple" commonly used blocking collective operations (sum, max, min, etc.). 9) Blocking send and receive, strided data. 10) Non-blocking send and receive, strided data. 11) Blocking collective communication, strided data. 12) Blocking send and receive with buffer-descriptor. 13) Non-blocking send and receive with buffer-descriptor. 14) Blocking collective communication with buffer-descriptor. 15) "User-defined" blocking collective operations (Express "excombine()" style). 16) Probe (and associated "precv()" routines, if necessary). 17) Non-blocking collective communication, contiguous data, strided data, and buffer-descriptor (and "multi-cancel()"?). 18) Definition of parallel I/O. 19) Blocking parallel I/O. 20) Non-blocking parallel I/O. 21) Additional environmental inquiry and "setting" routines for performance optimization (suggestions for system buffering, relative node/interconnect performance, etc.). 22) Dynamic process creation and identification. 23) Dynamic groups (processes can join/leave an existing group). 24) "Safe" send and receive. 25) Ready-send and ready-receive. 26) General process topologies. 27) Even more neat stuff. Once a detailed list like this is agreed on, we need to decide where to draw the line between MPI-1 and MPI-2 (and maybe MPI-3, etc.). Various straw polls about I/O indicate that 18-20 will probably be in MPI-2. I imagine an absolute minimum might be 1-5 for MPI-1. Here's three proposals for the list above: MPI-1 MPI-2 MPI-3, etc. "Ambitious" 1-17 18-26 27 "Less Ambitious" 1-14 15-23 24-27 "Conservative" 1-11 12-19 20-27 I share John Flower's "EXTREME concern" about the complexity of the current design. I believe that the MPIF must agree on SOME prioritization list and explicitly decide what is going to be in MPI-1 as soon as possible. I also feel strongly that we should decide what will be in future MPI versions to give potential users some confidence that MPI will eventually be useful for developing source-code portable programs. Tom Henderson (a potential MPI user) NOAA Forecast Systems Laboratory hender@fsl.noaa.gov (Thanks to Leslie Hart for some helpful suggestions.) From owner-mpi-comm@CS.UTK.EDU Tue Mar 9 20:46:50 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17983; Tue, 9 Mar 93 20:46:50 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10433; Tue, 9 Mar 93 20:45:22 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 9 Mar 1993 20:45:19 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10412; Tue, 9 Mar 93 20:45:16 -0500 Date: Wed, 10 Mar 93 01:45:05 GMT Message-Id: <2554.9303100145@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: MPI Goals and Scope: More Details Please? To: hender@macaw.fsl.noaa.gov (Tom Henderson), mpi-comm@cs.utk.edu In-Reply-To: Tom Henderson's message of Tue, 9 Mar 93 13:12:34 MST Reply-To: lyndon@epcc.ed.ac.uk Hi all There seems to be something to be discussed in what Tom is saying. Here is my 2 pennies worth. Please understand my multicomputer world bias, or flame me for it, whichever your prefer. > > > We feel that the MPI standard should provide enough information to > > > specify an entire, source-code portable program. > > > Bill & Rusty > > > I agree. This is going to sound awful picky, but I'm afraid I gotta say that I disagree with this. It's a grand ambition, and its also a grand challenge, in fact I conjecture that MPI is implicitly incapable of completing the stated mission even given arbitrary time, and that it is not a valid goal for MPI. The goal requires MPI to describe a whole raft of issues which are outwith MPI. I reckon John Flowers could spend the rest of his MPI time telling us about his experiences in Parasoft and Express with the issues that MPI cannot address! So MPI needs a more restricted mission. I'm too sleepy to try right now to find a suitable modification of the above sentence which might serve as a better mission for MPI. > > At the last meeting a straw vote lead to the adoption of "one processor can do > standard I/O". If we do this, then users will use vendor-supplied > "parallel I/O" routines to get decent performance. We cannot claim that the > MPI standard provides enough information to specify an entire, source-code > portable program unless we include I/O. I think that there is general > agreement that "parallel I/O" will not be included in MPI-1. Don't you think it will be some time before the world is ready to even think about standardisation of "parallel I/O". I can't imagine trying to do such a thing at the moment. > Maybe we could > re-state the portability goal: > The ultimate goal of the MPI standardization effort is to provide enough > information to specify an entire, source-code portable program. I have the same problem with this mission as that above. > I certainly won't use MPI unless I believe that this will happen with some > version of MPI. I reckon you will, actually, because although it won't give you the portability asked for above, it will be your best way of getting some portability. > I think we should re-examine the scope of the MPI effort and write down what > we want to accomplish with MPI-1, MPI-2, etc. in much more detail than is > currently in the draft introduction. I concur, in an ambivalent fashion. The introduction is vague, I'll agree with that. I think we are currently working out what we should be doing, in a rather ad hoc fashion. > John Flower has proposed one > prioritization. Here's another (from my perspective as a user): I'm going to walk down this list and give marks out of ten, with ten as highest priority and 0 as lowest priority, from my point of view, and some justification, from my point of view. > 1) Static process creation and identification (and associated environmental > inquiry routines). 10 - bare minimum MPI can acheive without being complete waste of time. > 2) Contexts. 9 - can get away with just tags but would impede library development. > 3) Heterogeneous communication semantics (I think we already have this). > All communication routines should support heterogeneous communication. 6 - static SPMD programs are most sensible in homogeneous machines. > 4) Blocking send and receive, contiguous data. 10 - bare minimum MPI can acheive without being complete waste of time. > 5) Non-blocking send and receive, contiguous data (status(), wait(), > cancel()?). 10 - bare minimum MPI can acheive without being complete waste of time. > 6) Creation/destruction of "static" groups (by "static" I mean processes > cannot join/leave an existing group). Grid/toroidal process topology > would go here if we decide to include it. 8 - can get away with just tags but would impede library development. > 7) Blocking collective communication, contiguous data (broadcast, > multiple-broadcast, concatenate, barrier, etc.). 10 - bare minimum MPI can acheive without being complete waste of time. > 8) "Simple" commonly used blocking collective operations (sum, max, min, > etc.). 10 - bare minimum MPI can acheive without being complete waste of time. > 9) Blocking send and receive, strided data. 8 - useful, but not essential. > 10) Non-blocking send and receive, strided data. 8 - useful, bit not essential. > 11) Blocking collective communication, strided data. 5 - makes collective communications too big. > 12) Blocking send and receive with buffer-descriptor. 8 - useful, but not essential. > 13) Non-blocking send and receive with buffer-descriptor. 8 - useful, but not essential. > 14) Blocking collective communication with buffer-descriptor. 5 - makes collective communications too big. > 15) "User-defined" blocking collective operations (Express "excombine()" > style). 10 - bare minimum MPI can acheive without being complete waste of time. > 16) Probe (and associated "precv()" routines, if necessary). 0 (boy, am I biased! :-) > 17) Non-blocking collective communication, contiguous data, strided data, and > buffer-descriptor (and "multi-cancel()"?). 5 - makes collective communication to big, and has technical complications. > 18) Definition of parallel I/O. 2 - too early to think about standardisation. > 19) Blocking parallel I/O. 2 - too early to think about standardisation. > 20) Non-blocking parallel I/O. 2 - too early to think about standardisation. > 21) Additional environmental inquiry and "setting" routines for performance > optimization (suggestions for system buffering, relative > node/interconnect performance, etc.). 2 - useless without better knowledge of how to use them. > 22) Dynamic process creation and identification. 6 - process group creation and resize is tractable model > 23) Dynamic groups (processes can join/leave an existing group). 2 - asynchronous join/leave very experimental 6 - process group creation and resize is tractable model > 24) "Safe" send and receive. 10 - bare minimum MPI can acheive without being complete waste of time. > 25) Ready-send and ready-receive. 0 - what benefit except for some old machines? > 26) General process topologies. 8 - if we have topologies at all, then should do this. > 27) Even more neat stuff. ? - ??? [stuff deleted] I think you can see what my staged schedule would be from the above. > > I share John Flower's "EXTREME concern" about the complexity of the current > design. I believe that the MPIF must agree on SOME prioritization list and > explicitly decide what is going to be in MPI-1 as soon as possible. I concur, ambivalently, as I think that it will yet take a little time for a strong consensus to emerge. I applaude efforts to generate such a consensus. > I also > feel strongly that we should decide what will be in future MPI versions to > give potential users some confidence that MPI will eventually be useful for > developing source-code portable programs. I am again ambivalent. I think it important to identify what items should be suitable for future MPI, but I'm not at all confident that it will be practical to commit ourselves to a schedule for trying to standardise such items. It will be harder to agree on standards for the items which are deferred. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 08:14:38 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA28523; Wed, 10 Mar 93 08:14:38 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11106; Wed, 10 Mar 93 08:13:23 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 08:13:22 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from super.super.org by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11098; Wed, 10 Mar 93 08:13:21 -0500 Received: from b125.super.org by super.super.org (4.1/SMI-4.1) id AA20589; Wed, 10 Mar 93 08:13:19 EST Received: by b125.super.org (4.1/SMI-4.1) id AA01744; Wed, 10 Mar 93 08:13:18 EST Date: Wed, 10 Mar 93 08:13:18 EST From: lederman@b125.super.org (Steve Huss-Lederman) Message-Id: <9303101313.AA01744@b125.super.org> To: mpi-comm@cs.utk.edu Subject: MPI directions & priorities I was in the process of composing a note to the committee about what MPI-1 should compose and was letting it sit overnight when Tom sent out his message. Since he has done a nice job I will shorten my comments. At the last meeting there were several group and private discussions about the growing complexity of MPI. Al has posted his feeling that point-to-point and collective communications get complex and untested ideas added but other areas such as dynamic processes are not considered. Jon posted about the complexity and now Tom has started a list of priorities. I add my voice to these concerns. I have been hesitant to post because I don't want it to seem that I think MPI is hopeless. However, I think it is clear that we cannot adequately complete all the current proposals in 3 more meetings. I would propose that we continue the discussion concerning priorities via e-mail and begin the next meeting with a whole group meeting to set our course. At the higher level we need to decide: 1) How long will the MPI-1 process be allowed to take? Are we going to stick to the 6 month deadline or try and extend it? 2) What is going to be in MPI-1? 3) Of the things we leave out of MPI-1 are they going to be proposed for a later version of MPI? Should we make sure that MPI-1 is compatible with the future goals so that major changes in the base routines is not needed later? 4) If we want to have MPI-#, what mechanism and time scale will be used to get it done. I think it is past time to find the forest for the trees. Global planning to focus our efforts now could significantly reduce our discussions and efforts in the short term as deadlines approach. Furthermore, with a global blueprint we might see a more even handed approach to different aspects of the standard. These are just some thoughts. I hope the discussion will continue. Steve From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 10:15:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02371; Wed, 10 Mar 93 10:15:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16993; Wed, 10 Mar 93 10:13:23 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 10:13:21 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16985; Wed, 10 Mar 93 10:13:18 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA03655 (5.65c/IDA-1.4.4 for ); Wed, 10 Mar 1993 09:13:16 -0600 From: William Gropp Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA05373; Wed, 10 Mar 93 09:13:13 CST Date: Wed, 10 Mar 93 09:13:13 CST Message-Id: <9303101513.AA05373@godzilla.mcs.anl.gov> To: lederman@b125.super.org Cc: mpi-comm@cs.utk.edu In-Reply-To: Steve Huss-Lederman's message of Wed, 10 Mar 93 08:13:18 EST <9303101313.AA01744@b125.super.org> Subject: MPI directions & priorities One approach that we could take is to describe a set of MPI that is extensible to the more elaborate facilities but does not yet specify them. For example, we could keep the goal of being "thread-safe" and resolve (at least in an extensible way) the group/context/process-id issues, and then settle on, for example, somethine like what Rusty and I have called level 3, plus some collective communication routines. This would not have everything that I would want or even consider important, but would be a usable set for casual use. This is a very different process than just defining a simple set of operations; it MUST be possible to extend, in a consistent way, MPI to meet new needs. I'd like to point out that many of the so-called new features are ones that are in use in some (usually only one) system; in most cases they fulfill a perceived need. For MPI to be successful, it must not be incompatible with these needs. I realize that the other side of this is that, by specifying only a subset of desired functionality, there is the danger of mutually incompatible extensions. We need to weigh this against the possibility that MPI will become too complex to be accepted. Bill From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 10:24:04 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02512; Wed, 10 Mar 93 10:24:04 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17488; Wed, 10 Mar 93 10:22:34 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 10:22:33 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17476; Wed, 10 Mar 93 10:22:29 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA03990 (5.65c/IDA-1.4.4 for ); Wed, 10 Mar 1993 09:22:24 -0600 From: William Gropp Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA05385; Wed, 10 Mar 93 09:22:22 CST Date: Wed, 10 Mar 93 09:22:22 CST Message-Id: <9303101522.AA05385@godzilla.mcs.anl.gov> To: lyndon@epcc.ed.ac.uk Cc: hender@macaw.fsl.noaa.gov, mpi-comm@cs.utk.edu In-Reply-To: L J Clarke's message of Wed, 10 Mar 93 01:45:05 GMT <2554.9303100145@subnode.epcc.ed.ac.uk> Subject: MPI Goals and Scope: More Details Please? I'd like to clarify what we mean by > We feel that the MPI standard should provide enough information to > specify an entire, source-code portable program. This is NOT intended to mean that MPI must provide a way to specify ANY (or even most) source-code portable parallel program. Rather, given restrictions strong enough to satisfy a 2/3rds majority ( :) ), a way to specify at least one source-code portable program. In the spirit of extensibility, whatever mechanism is proposed for this simple interface should not preclude non-MPI solutions for more demanding situations. Bill From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 11:33:20 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03903; Wed, 10 Mar 93 11:33:20 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21975; Wed, 10 Mar 93 11:31:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 11:31:41 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21960; Wed, 10 Mar 93 11:31:39 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA10050; Wed, 10 Mar 93 10:27:36 CST Date: Wed, 10 Mar 93 10:27:36 CST From: Tony Skjellum Message-Id: <9303101627.AA10050@Aurora.CS.MsState.Edu> To: lederman@b125.super.org, gropp@mcs.anl.gov Subject: Re: MPI directions & priorities Cc: mpi-comm@cs.utk.edu We must agree to reconvene about extensions in future. I also agree that it would be preferable to specify a whole portable program, but I think the I/O subject is hard to address right now... - Tony ----- Begin Included Message ----- From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 09:42:01 1993 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 10:13:21 EST From: William Gropp Date: Wed, 10 Mar 93 09:13:13 CST To: lederman@b125.super.org Cc: mpi-comm@cs.utk.edu Subject: MPI directions & priorities Content-Length: 1193 One approach that we could take is to describe a set of MPI that is extensible to the more elaborate facilities but does not yet specify them. For example, we could keep the goal of being "thread-safe" and resolve (at least in an extensible way) the group/context/process-id issues, and then settle on, for example, somethine like what Rusty and I have called level 3, plus some collective communication routines. This would not have everything that I would want or even consider important, but would be a usable set for casual use. This is a very different process than just defining a simple set of operations; it MUST be possible to extend, in a consistent way, MPI to meet new needs. I'd like to point out that many of the so-called new features are ones that are in use in some (usually only one) system; in most cases they fulfill a perceived need. For MPI to be successful, it must not be incompatible with these needs. I realize that the other side of this is that, by specifying only a subset of desired functionality, there is the danger of mutually incompatible extensions. We need to weigh this against the possibility that MPI will become too complex to be accepted. Bill ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 12:23:22 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05096; Wed, 10 Mar 93 12:23:22 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24792; Wed, 10 Mar 93 12:22:05 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 12:22:04 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24784; Wed, 10 Mar 93 12:22:01 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA26984; Wed, 10 Mar 93 17:21:17 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA07764; Wed, 10 Mar 93 10:20:02 MST Date: Wed, 10 Mar 93 10:20:02 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303101720.AA07764@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: MPI Goals and Scope: More Details Please? Cc: lyndon@epcc.edinburgh.ac.uk Lyndon writes, > Don't you think it will be some time before the world is ready to even > think about standardisation of "parallel I/O". I can't imagine trying > to do such a thing at the moment. I agree. I'm not proposing we do this now. I'm proposing that we do it when it is as well understood as message passing is now. I think "eventual standardization of parallel I/O" should be a stated goal to give potential MPI users confidence that MPI won't be just another message-passing library. > [stuff deleted...] > > > I certainly won't use MPI unless I believe that this will happen with some > > version of MPI. > > I reckon you will, actually, because although it won't give you the > portability asked for above, it will be your best way of getting some > portability. I think Lyndon has a good point here. I will use MPI if: A) It gives me as much "portability" as existing solutions OR I believe that this will eventually happen. B) It gives me equal or better "performance". I will borrow/build/buy whatever additional stuff I need to make my programs more "portable", if necessary (required for my current project). I am hopeful that A will happen with some version of MPI. Vendor participation gives me some confidence that B will happen. > > John Flower has proposed one > > prioritization. Here's another (from my perspective as a user): > > I'm going to walk down this list and give marks out of ten, with ten as > highest priority and 0 as lowest priority, from my point of view, and > some justification, from my point of view. > > [list deleted...] Thanks Lyndon, this is exactly what I was looking for. At this point I feel it is extremely important that the MPIF come to some agreement about a list like this. It is a lot more important than my actually liking what's on the list! :) > [stuff deleted...] > > I am again ambivalent. I think it important to identify what items > should be suitable for future MPI, but I'm not at all confident that it > will be practical to commit ourselves to a schedule for trying to > standardise such items. It will be harder to agree on standards for the > items which are deferred. > > Best Wishes > Lyndon > I didn't mean to imply an absolute time-frame for the introduction of future MPI features. (It doesn't make sense for the more "researchy" stuff.) Prioritization is more important to me than a strict schedule. Tom Henderson NOAA Forecast Systems Laboratory hender@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 12:32:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05213; Wed, 10 Mar 93 12:32:07 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25472; Wed, 10 Mar 93 12:30:52 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 12:30:51 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25460; Wed, 10 Mar 93 12:30:47 -0500 Date: Wed, 10 Mar 93 17:30:30 GMT Message-Id: <3289.9303101730@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: MPI Goals and Scope: More Details Please? To: hender@macaw.fsl.noaa.gov (Tom Henderson), mpi-comm@cs.utk.edu In-Reply-To: Tom Henderson's message of Wed, 10 Mar 93 10:20:02 MST Reply-To: lyndon@epcc.ed.ac.uk Cc: lyndon@epcc.edinburgh.ac.uk > > I'm going to walk down this list and give marks out of ten, with ten as > > highest priority and 0 as lowest priority, from my point of view, and > > some justification, from my point of view. > > > > [list deleted...] > > Thanks Lyndon, this is exactly what I was looking for. Oh, good, glad I did that then! > At this point I feel > it is extremely important that the MPIF come to some agreement about a list > like this. AGREED!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! > > I didn't mean to imply an absolute time-frame for the introduction of future > MPI features. (It doesn't make sense for the more "researchy" stuff.) > Prioritization is more important to me than a strict schedule. Sorry, Tom, I misread your email. It's over and out for a week from me, colleagues, as I'm visiting my folks for the weekend and a couple of work meetings - surprise surprise, one of them to talk about MPI to a potential user, urgle urgle urgle! May your mail boxes be temporarily lighter ;-) Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 12:43:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05464; Wed, 10 Mar 93 12:43:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26174; Wed, 10 Mar 93 12:41:59 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 12:41:58 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26164; Wed, 10 Mar 93 12:41:55 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA27092; Wed, 10 Mar 93 17:41:49 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA07798; Wed, 10 Mar 93 10:40:35 MST Date: Wed, 10 Mar 93 10:40:35 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303101740.AA07798@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: MPI Goals and Scope: More Details Please? Cc: gropp@mcs.anl.gov Bill writes, > I'd like to clarify what we mean by > > > We feel that the MPI standard should provide enough information to > > specify an entire, source-code portable program. > > This is NOT intended to mean that MPI must provide a way to specify ANY (or > even most) source-code portable parallel program. Rather, given restrictions > strong enough to satisfy a 2/3rds majority ( :) ), a way to specify at least > one source-code portable program. In the spirit of extensibility, whatever > mechanism is proposed for this simple interface should not preclude non-MPI > solutions for more demanding situations. > Bill > Could you give an example of the one source-code portable program (or class of programs) you expect MPI to support in current and future versions? I think that might help me understand your statement a bit better. Also, what do you mean by "non-MPI solution" and "demanding situations"? Thanks, Tom Henderson NOAA Forecast Systems Laboratory hender@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 13:19:43 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06396; Wed, 10 Mar 93 13:19:43 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28116; Wed, 10 Mar 93 13:18:26 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 13:18:25 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28057; Wed, 10 Mar 93 13:18:02 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 10 Mar 93 10:10 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA22798; Wed, 10 Mar 93 10:08:19 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA20772; Wed, 10 Mar 93 10:08:13 PST Date: Wed, 10 Mar 93 10:08:13 PST From: d39135@sodium.pnl.gov Subject: implementation assumptions To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.edinburgh.ac.uk, mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Message-Id: <9303101808.AA20772@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu, mpi-comm@cs.utk.edu In a discussion we are having within the contexts subcommittee, Tony Skjellum commented that: > ... I want to make sure that we do > not provide illusory improvements to the standard, while letting other > parts of it exceed ours in the unscalability department, or let > implementations qualify as compliant, whose unscalability disables the > implied scalability of our syntax/semantics... I agree. This is why (I think) proposals should explicitly address performance. If a proposal assumes that the cost of a particular function is irrelevant, it should say that. If a proposal assumes that a particular function is going to be cheap, it should justify that assumption by naming a time/memory price and outlining an implementation that can achieve it. Sometimes the suggested implementation will reveal further assumptions that have wide implications. Ideally, each proposal will eventually evolve a set of bottom-line assumptions that are consistent and believable. These comments are particularly addressed to those of us beating on contexts and groups, where there seem to be lots of subtle interactions. But I am cross-posting them to the whole committee as food for thought... --Rik ("The Skeptic") Littlefield From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 13:22:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06459; Wed, 10 Mar 93 13:22:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28196; Wed, 10 Mar 93 13:20:20 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 13:20:18 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28188; Wed, 10 Mar 93 13:20:16 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA27261; Wed, 10 Mar 93 18:20:08 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA07839; Wed, 10 Mar 93 11:18:54 MST Date: Wed, 10 Mar 93 11:18:54 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303101818.AA07839@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: MPI directions & priorities Cc: lederman@b125.super.org Steve writes, > 1) How long will the MPI-1 process be allowed to take? Are we going > to stick to the 6 month deadline or try and extend it? I'd like to stick to 6 months for MPI-1. I'm willing to have less functionality in MPI-1 as long as there is a plan for further upgrade. > 2) What is going to be in MPI-1? Good question. :) > 3) Of the things we leave out of MPI-1 are they going to be proposed > for a later version of MPI? Should we make sure that MPI-1 is > compatible with the future goals so that major changes in the base > routines is not needed later? Yes and Yes. > 4) If we want to have MPI-#, what mechanism and time scale will be > used to get it done. Maybe next year we meet again for MPI-2? We should have some experience with MPI-1 by then. (...wait a minute, I only signed on for a six-month tour...) > I think it is past time to find the forest for the trees. Global > planning to focus our efforts now could significantly reduce our > discussions and efforts in the short term as deadlines approach. > Furthermore, with a global blueprint we might see a more even handed > approach to different aspects of the standard. These are just some > thoughts. I hope the discussion will continue. Me too. > Steve Tom Henderson NOAA Forecast Systems Laboratory hender@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 13:22:55 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06475; Wed, 10 Mar 93 13:22:55 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28313; Wed, 10 Mar 93 13:21:38 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 13:21:37 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28305; Wed, 10 Mar 93 13:21:34 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA27273; Wed, 10 Mar 93 18:21:30 GMT Received: by nipmuc.fsl.noaa.gov (4.1/SMI-4.1) id AA10163; Wed, 10 Mar 93 11:24:00 MST Date: Wed, 10 Mar 93 11:24:00 MST From: hart@nipmuc.fsl.noaa.gov (Leslie Hart) Message-Id: <9303101824.AA10163@nipmuc.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: MPI directions & priorities Bill Gropp sez: > One approach that we could take is to describe a set of MPI that is extensible > to the more elaborate facilities but does not yet specify them. For example, > we could keep the goal of being "thread-safe" and resolve (at least in an > extensible way) the group/context/process-id issues, and then settle on, for > example, somethine like what Rusty and I have called level 3, plus some > collective communication routines. This would not have everything that I would > want or even consider important, but would be a usable set for casual use. > > This is a very different process than just defining a simple set of operations; > it MUST be possible to extend, in a consistent way, MPI to meet new needs. > I'd like to point out that many of the so-called new features are ones that > are in use in some (usually only one) system; in most cases they fulfill > a perceived need. For MPI to be successful, it must not be incompatible with > these needs. > > I realize that the other side of this is that, by specifying only a subset of > desired functionality, there is the danger of mutually incompatible extensions. > We need to weigh this against the possibility that MPI will become too complex > to be accepted. > > Bill I agree with Bill, I think whatever we produce for MPI-1 should be extensible. I think we should consider carefully our semantics and syntax to not prohibit extensibility. An approach that has been proposed by us (Tom Henderson and myself) as well as by other people is opaque messages. This allows an initial implementation to support a relatively simple to use subset while allowing vendor extensions and future upgrades to MPI-1. We could perhaps specify some core of MPI-1 (not a new idea) as well as provide semantic and syntatical guidance to vendors for their extensions (e.g. a clue about MPI-2 if you will). The opaque message object allows these extensions to be put in place without an explosion of syntax. Leslie Hart (hart@fsl.noaa.gov) P.S This may fit into Rusty & Bill's Level 1 or Level 2 proposal neatly. From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 13:26:58 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06583; Wed, 10 Mar 93 13:26:58 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28684; Wed, 10 Mar 93 13:25:59 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 13:25:58 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28669; Wed, 10 Mar 93 13:25:55 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA10195 (5.65c/IDA-1.4.4 for ); Wed, 10 Mar 1993 12:25:52 -0600 From: William Gropp Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA22431; Wed, 10 Mar 93 12:25:51 CST Date: Wed, 10 Mar 93 12:25:51 CST Message-Id: <9303101825.AA22431@godzilla.mcs.anl.gov> To: hender@macaw.fsl.noaa.gov Cc: mpi-comm@cs.utk.edu In-Reply-To: Tom Henderson's message of Wed, 10 Mar 93 10:40:35 MST <9303101740.AA07798@macaw.fsl.noaa.gov> Subject: MPI Goals and Scope: More Details Please? X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 12:41:58 EST Date: Wed, 10 Mar 93 10:40:35 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Could you give an example of the one source-code portable program (or class of programs) you expect MPI to support in current and future versions? I think that might help me understand your statement a bit better. Basically, SPMD programs that have "simple" I/O requirements. For example, we could ask the following: IF fprintf (C) or write (Fortran) are supported, THEN each use writes to the selected file, with NO guarentees on the ordering of output. Something similiar for fscanf/read . Thus, we can now write the following two programs (using the MPI_main startup approach): MPI_main() { printf( "Hello world\n" ); } and MPI_main() { int i; for (i=0; i To: mpi-comm@cs.utk.edu Subject: Re: MPI directions & priorities > From: Tony Skjellum > To: lederman@b125.super.org, gropp@mcs.anl.gov > Subject: Re: MPI directions & priorities > Cc: mpi-comm@cs.utk.edu > > We must agree to reconvene about extensions in future. I also agree > that it would be preferable to specify a whole portable program, but I > think the I/O subject is hard to address right now... > > - Tony I also understood that the process would be ongoing and hope that it will continue to track existing practice. I think some parts of the I/O subject are hard to address right now, but there is some existing common practice. Parasoft, Intel, nCUBE and TMC all have some sort of parallel I/O specification. There are ways to force the effect of a printf or of an fseek/fread/fwrite to be deterministic. I believe that the common practice could be standardized, maybe not in the time frame for MPI-1 but probably for MPI-2 (which is where it appeared in Tom's posting). Regards, Leslie From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 14:03:55 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07595; Wed, 10 Mar 93 14:03:55 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01003; Wed, 10 Mar 93 14:02:53 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 14:02:52 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA00994; Wed, 10 Mar 93 14:02:47 -0500 Received: from elephant by sampson.ccsf.caltech.edu with SMTP id AA08764 (5.65c/IDA-1.4.4 for mpi-comm@cs.utk.edu); Wed, 10 Mar 1993 11:02:44 -0800 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA26008; Wed, 10 Mar 93 10:55:24 PST Received: by lion.parasoft (4.1/SMI-4.1) id AA26815; Wed, 10 Mar 93 10:55:34 PST Date: Wed, 10 Mar 93 10:55:34 PST From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9303101855.AA26815@lion.parasoft> To: mpi-comm@cs.utk.edu To: mpi-comm@cs.utk.edu Re: A way of prioritizing? Something Bill said seemed to strike a chord in my mind: > This is a very different process than just defining a simple > set of operations; it MUST be possible to extend, in a > consistent way, MPI to meet new needs. I'd like to point > out that many of the so-called new features are ones that > are in use in some (usually only one) system; in most cases > they fulfill a perceived need. For MPI to be successful, > it must not be incompatible with these needs. I also have the feeling that a lot of the things that we're trying to put in MPI are really only based on experience (or even need) on a single machine. Sometimes that machine may even be "pre-history". My question is: Is incorporating into MPI a feature that is only based on a single machine any better than allowing a vendor-specific extension on that machine? Take "ready-receive", for example. Adding this to MPI means that users can justifiably put it in their portable codes. If they do, however, what should they assume about it? That it's faster or slower than the non-ready receive? I assume that there will be systems in which it's slower than the non-ready recv. because of the extra protocol that the application might need to ensure that the ready-recv.'s don't crash. In this case, are we expecting users to put code in their applications that can switch between ready-recv, and non-ready-recv semantics based on the hardware they're running on? Although I think there are some users who will do this (and I would encourage library developers to do so) I think this is far beyond the lengths that most users will go to. So what am I suggesting? --------------------------------------------------------- My suggestion (which I expect to be roundly rejected) is that the MPI document ONLY contain the stuff that is current practice. This includes EVERYTHING required to run at least a common class of programs - i.e., MPI_INIT or some such. These things should be designed to maximize ease of use and learning and encourage portability. By this I am including performance level portability. I would like to not have things in the spec. that run well on some machines and poorly on others. I would include a requirement that a conforming implementation include either documents or (preferably) programs from which a user could assess the performance of the various routines. All the other things that are off-shoots of requirements for odd-machines or otherwise more advanced should go in Appendices or "Annexes" (Ada notation(!)) ***AND WOULD NOT BE REQUIRED BY A CONFORMING IMPLEMENTATION OF MPI ***. --------------------------------------------------------- The effect I am after is that the core MPI standard is simple, effective and encourages users to write code. The annexes essentially say to vendors; "If you want to support an advanced feature that will really help on your machine, please do it this way so that my programs that use it will be portable." This is similar to Bill & Rusty's multi-level scenario except that I wouldn't make the lower levels compulsory. My hope is that two things will happen if we adopt this strategy: a) Details of things that go in Annexes can be postponed or left vague, or even left to specially interested groups to define, but we will definitely have something that could be used by an application developer at the end of the six months. b) The MPI core would be smaller which will encourage users/vendors to make implementations, and to optimize the #$%$% out of them. If this concept is acceptable I would suggest that someone prepare a list of topics, in quite great detail, and we all give them grades out of ten (via e-mail), just as Lyndon did. Someone with experience with statistics could analyze the results and give us weighted rankings at the next meeting. We could then prioritize according to this list and also make guided decisions as to what should go in the Annexes. I would then suggest that we go back to the small sub-committee approach to putting the Annexes together rather than our current "committee-of-the-whole" style which seems to encourage hour-long "arguments" followed by 35-0 votes! Jon Flower p.s. Having suggested a pile of work I should say who's going to do it. In the absence of any volunteers I might be persuaded to come up with a list of issues on which we should score. I'm not sure my statistical skills amount to anything much more that simple averages, so someone else could certainly do better than I. I would be very interested in working with a group/sub-committee trying to get up a set of MPI-compliant performance evaluation standards. From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 15:32:12 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09642; Wed, 10 Mar 93 15:32:12 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07600; Wed, 10 Mar 93 15:30:40 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 15:30:38 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07590; Wed, 10 Mar 93 15:30:35 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 10 Mar 93 12:29 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA24307; Wed, 10 Mar 93 12:27:24 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA21996; Wed, 10 Mar 93 12:27:21 PST Date: Wed, 10 Mar 93 12:27:21 PST From: d39135@sodium.pnl.gov Subject: RE: MPI directions & priorities To: gropp@mcs.anl.gov, lederman@b125.super.org, tony@Aurora.CS.MsState.Edu Cc: d39135@sodium.pnl.gov, mpi-comm@cs.utk.edu Message-Id: <9303102027.AA21996@sodium.pnl.gov> X-Envelope-To: mpi-comm@cs.utk.edu > We must agree to reconvene about extensions in future. I also agree > that it would be preferable to specify a whole portable program, but I > think the I/O subject is hard to address right now... > > - Tony I presume that Tony speaks of parallel I/O. I agree that parallel I/O is too hard for now. But I think that we have to specify at least some minimum serial I/O capability. I would buy off on something as simple as . there exists one process that can do serial I/O in the standard fashion for the language (e.g., printf() or WRITE(*,*) ) . the tid of that process can be obtained from MPI (in any process) Example: if (mytid == MPI_IOROOT()) { printf ("I'm the designated serial I/O process\n"); } --Rik Littlefield From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 15:37:36 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09805; Wed, 10 Mar 93 15:37:36 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08036; Wed, 10 Mar 93 15:36:28 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 15:36:27 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08022; Wed, 10 Mar 93 15:36:25 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA27653; Wed, 10 Mar 93 20:36:21 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA08016; Wed, 10 Mar 93 13:35:06 MST Date: Wed, 10 Mar 93 13:35:06 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303102035.AA08016@macaw.fsl.noaa.gov> To: jwf@lion.parasoft.com Subject: Annexes, etc. Cc: mpi-comm@cs.utk.edu Jon, > My suggestion (which I expect to be roundly rejected) is > that the MPI document ONLY contain the stuff that is current > practice. This includes EVERYTHING required to run at least > a common class of programs - i.e., MPI_INIT or some such. > > These things should be designed to maximize ease of use and > learning and encourage portability. By this I am including > performance level portability. I would like to not have > things in the spec. that run well on some machines and > poorly on others. I would include a requirement that a > conforming implementation include either documents or > (preferably) programs from which a user could assess > the performance of the various routines. > > All the other things that are off-shoots of requirements > for odd-machines or otherwise more advanced should go in > Appendices or "Annexes" (Ada notation(!)) ***AND WOULD NOT > BE REQUIRED BY A CONFORMING IMPLEMENTATION OF MPI ***. Would the MPIF make any commitment to including "Annex" features in future MPI versions? Would there be any future MPI versions? Would you mind providing a bit more detail on what you feel are a "common class of programs"? (Gee, I don't ask for much... :-) > If this concept is acceptable I would suggest that someone > prepare a list of topics, in quite great detail, and we > all give them grades out of ten (via e-mail), just as > Lyndon did. Someone with experience with statistics > could analyze the results and give us weighted rankings > at the next meeting. What level of detail do you have in mind? Could you give an example? I like this idea. Hashing this out over e-mail could save us a lot of pointless wrangling at the next meeting. Tom P.S. Jon, my apologies for mis-spellings in previous messages! :) From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 05:10:18 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25347; Thu, 11 Mar 93 05:10:18 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11372; Thu, 11 Mar 93 05:08:22 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 05:08:21 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11362; Thu, 11 Mar 93 05:08:16 -0500 Date: Thu, 11 Mar 93 10:07:58 GMT Message-Id: <3792.9303111007@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: minutes? To: mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk I never saw any minutes for the February meeting yet, perhaps I've been expecting them too early. Please, can someone let me know when they are expected to be circulated. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 06:02:48 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07862; Thu, 11 Mar 93 06:02:48 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18258; Thu, 11 Mar 93 06:01:26 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 06:01:25 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18250; Thu, 11 Mar 93 06:01:20 -0500 Date: Thu, 11 Mar 93 11:01:15 GMT Message-Id: <3852.9303111101@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: regarding discussion of directions and priorities To: mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Dear MPI Colleagues I am shortly to give a presentation regarding MPI to a potential end user of some imprtance within the UK. In preparation for this, I have spent the last hour of this morning looking back over the discussions on mpi-comm and various subcommittees. The single thing which most struck me was that since March 9 the content on mpi-comm has been dominated by discussion of directions and priorities for MPI. I recorded 20 mail messages in total from 8 individuals. There were preceeding discussions of this nature on mpi-pt2pt dating from March 4. The discussions indicate to me that there are some rather concerned committee members. It would appear, to my mind at least, that different committee members have different views of the extant directions and priorities of MPI, and of desired directions and priorities for MPI. I am sure that I have a different view from some other members of MPI. In order to make satisfactory progress in the process of interface definition, which is apparently not occuring at the moment, I believe that there would indeed be advantage in describing and formally agreeing the directions of MPI and a set of priorities within those directions. I applaude efforts to create consensus regarding directions and priorities. I also believe that, from the point of view of the perception of MPI of individuals and organisations not directly involved in the committee, there is advantage in publicising the agreed directions of MPI. In order to make satisfactory progress in the process of determining directions and priorities, I believe that there would be advantage in describing and formally agreeing a framework within which we can agree on such directions and priorities. Sadly, I do not observe such a framework in operation, and I am skeptical whether MPI can succeed without such a framework. I would therefore suggest that the immediate priorities for MPI as as follows: 1. Discuss and establish framework within which directions for MPI are assessed. 2. Discuss and establish directions for MPI within agreed framework. 3. Discuss and establish priorities for MPI within agreed directions. 4. Review proposals in all subcommittees within agreed priorities. 5. Extend and complete proposals as necessary within terms of 3 and 4. As a suggestion, can we all describe what we mean by "common practice", a starting point for framework discussion? Some of the 8 mentioned above have already done this, to some extent, which I also applaud. It seems to me that time demands 1,2 and 3 must occur primarily by email. I would suggest that these should be formally accepted at the beginning of the next meeting. From a practical point of view, I further suggest that the next meeting should call a meeting of the whole committee for this purpose, and we should be prepared to hold said meeting on the Wednesday morning, on order to allow the subcommittees to make progress with 4 and 5. Now my taxi is waiting, and I must go off-line for a week. I apologise to you all that I will not be able to follow up this somewhat despondent message for 6 days. Good luck! Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 11:20:08 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29874; Thu, 11 Mar 93 11:20:08 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01139; Thu, 11 Mar 93 11:17:49 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 11:17:48 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01131; Thu, 11 Mar 93 11:17:47 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA13509; Thu, 11 Mar 93 10:13:56 CST Date: Thu, 11 Mar 93 10:13:56 CST From: Tony Skjellum Message-Id: <9303111613.AA13509@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, lyndon@epcc.ed.ac.uk Subject: Re: minutes? Yes, having the minutes is becoming essential. - Tony ----- Begin Included Message ----- I never saw any minutes for the February meeting yet, perhaps I've been expecting them too early. Please, can someone let me know when they are expected to be circulated. Best Wishes Lyndon ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 11:23:41 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00145; Thu, 11 Mar 93 11:23:41 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01349; Thu, 11 Mar 93 11:21:30 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 11:21:28 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01340; Thu, 11 Mar 93 11:21:27 -0500 Received: by msr.EPM.ORNL.GOV (5.67/1.34) id AA00895; Thu, 11 Mar 93 11:21:24 -0500 Date: Thu, 11 Mar 93 11:21:24 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9303111621.AA00895@msr.EPM.ORNL.GOV> To: mpi-comm@cs.utk.edu Subject: Re: Jon's suggestion Jon writes: > My suggestion (which I expect to be roundly rejected) is > that the MPI document ONLY contain the stuff that is current > practice. This includes EVERYTHING required to run at least > a common class of programs - i.e., MPI_INIT or some such. > > These things should be designed to maximize ease of use and > learning and encourage portability. By this I am including > performance level portability. I would like to not have > things in the spec. that run well on some machines and > poorly on others. I would include a requirement that a > conforming implementation include either documents or > (preferably) programs from which a user could assess > the performance of the various routines. > > All the other things that are off-shoots of requirements > for odd-machines or otherwise more advanced should go in I do not reject this in fact I applaud your suggestion. This may be the only way we can get MPI-1 off the ground. I am a big advocate of maximizing ease of use this is only way to get the large user base necessary for MPI to become a standard. Al Geist From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 12:23:03 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01995; Thu, 11 Mar 93 12:23:03 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04568; Thu, 11 Mar 93 12:19:59 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 12:19:57 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from NA-GW.CS.YALE.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04559; Thu, 11 Mar 93 12:19:56 -0500 Received: from YOGI.NA.CS.YALE.EDU by CASPER.NA.CS.YALE.EDU via SMTP; Thu, 11 Mar 1993 12:19:53 -0500 Received: by YOGI.NA.CS.YALE.EDU (Sendmail-5.65c/res.client.cf-3.5) id AA21933; Thu, 11 Mar 1993 12:19:51 -0500 Date: Thu, 11 Mar 1993 12:19:51 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Message-Id: <199303111719.AA21933@YOGI.NA.CS.YALE.EDU> To: mpi-comm@cs.utk.edu Subject: Message Passing not for the masses Al say's I do not reject this in fact I applaud your suggestion. This may be the only way we can get MPI-1 off the ground. I am a big advocate of maximizing ease of use this is only way to get the large user base necessary for MPI to become a standard. \begin{flame} I generally agree with Al, but not this time. While I tend to think that there should be some way to figure out how to use MPI in one sitting, I certainly do not think that we should be "maximizing ease of use" as the principle design goal. If I wanted things to be simple and slow, I'd be a Linda user. Message passing is nasty. Message passing sucks. Message passing is the way to get machines to go fast, at least for the time being. \end{flame} That being said, even if MPI turns into a huge, ugly mass of routines, it should still be possible to isolate a few of these routines and present them as the "simple subset" for people who don't need to drive their communication costs down. I don't see "limiting the size of" being the same as "making easy to use". -scott berryman From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 12:32:20 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02217; Thu, 11 Mar 93 12:32:20 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA05147; Thu, 11 Mar 93 12:30:50 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 12:30:48 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA05132; Thu, 11 Mar 93 12:30:46 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA29711; Thu, 11 Mar 93 17:30:24 GMT Received: by nipmuc.fsl.noaa.gov (4.1/SMI-4.1) id AA05324; Thu, 11 Mar 93 10:32:53 MST Date: Thu, 11 Mar 93 10:32:53 MST From: hart@nipmuc.fsl.noaa.gov (Leslie Hart) Message-Id: <9303111732.AA05324@nipmuc.fsl.noaa.gov> To: mpi-comm@cs.utk.edu, geist@msr.epm.ornl.gov Subject: Re: Jon's suggestion > Jon writes: > > My suggestion (which I expect to be roundly rejected) is > > that the MPI document ONLY contain the stuff that is current > > practice. This includes EVERYTHING required to run at least > > a common class of programs - i.e., MPI_INIT or some such. > > > > These things should be designed to maximize ease of use and > > learning and encourage portability. By this I am including > > performance level portability. I would like to not have > > things in the spec. that run well on some machines and > > poorly on others. I would include a requirement that a > > conforming implementation include either documents or > > (preferably) programs from which a user could assess > > the performance of the various routines. > > > > All the other things that are off-shoots of requirements > > for odd-machines or otherwise more advanced should go in > > I do not reject this in fact I applaud your suggestion. > This may be the only way we can get MPI-1 off the ground. > I am a big advocate of maximizing ease of use > this is only way to get the large user base necessary > for MPI to become a standard. > > Al Geist I also agree. I think that there is significant concern regarding the near term and long term success of MPI. I think some of the concern can be aleviated if we settle on a goal that can be reached by this summer. It may be wise to set up some sort of electronic straw poll procedure to help get us past some of the issues regarding what should be in the first MPI, what are our goals for MPI-1, is there going to be MPI-N (for N>1), what features are most important for MPI-1, etc. Perhaps we can use similar rules to those agreed upon in the first meeting. You must have been in Dallas for 2 of the last three meetings (1 of the first 2, I guess). Calls for vote should be made through the committee chair (or someone appointed by the committee chair). The votes will be counted over a three day period by the comittee chair (or the designee) and reported. (Only your "last" vote counts :-). These will, of course, be straw polls and will be reported again at the beginning of the next Dallas meeting. Regards, Leslie Hart (hart@fsl.noaa.gov) P.S. We may wish to use a separate mail list for voting (mpi-vote??) which includes those eligible to vote via the voter elibibilty rules. Of course, votes for a particular topic should be clearly identified and the sense of the vote should be clear (i.e yes or no). The suggestion of electronic votes is important to me, but the mechanics is not that important. > From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 13:04:39 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03015; Thu, 11 Mar 93 13:04:39 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08235; Thu, 11 Mar 93 13:00:11 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 13:00:08 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08214; Thu, 11 Mar 93 13:00:05 -0500 Received: from jadoube.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA19857 (5.65c/IDA-1.4.4 for ); Thu, 11 Mar 1993 12:00:02 -0600 From: David Levine Received: by jadoube.mcs.anl.gov (4.1/GeneV4) id AA15971; Thu, 11 Mar 93 12:00:00 CST Date: Thu, 11 Mar 93 12:00:00 CST Message-Id: <9303111800.AA15971@jadoube.mcs.anl.gov> To: berryman-harry@cs.yale.edu Cc: mpi-comm@cs.utk.edu In-Reply-To: Harry Berryman's message of Thu, 11 Mar 1993 12:19:51 -0500 <199303111719.AA21933@YOGI.NA.CS.YALE.EDU> Subject: Message Passing not for the masses X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 12:19:57 EST Date: Thu, 11 Mar 1993 12:19:51 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Al say's I do not reject this in fact I applaud your suggestion. This may be the only way we can get MPI-1 off the ground. I am a big advocate of maximizing ease of use this is only way to get the large user base necessary for MPI to become a standard. \begin{flame} : Message passing is nasty. Message passing sucks. Message passing is the way to get machines to go fast, at least for the time being. \end{flame} I'd like to add my two cents in agreement with this comment. Message passing is NOT for the masses. Its just too hard and grungy. My opinion is that message passing is the assembly language of parallel programming and it will evolve that way. I think the "large user base necessary for MPI to become a standard." will be *expert* users, vendors, and HPF compiler writers. If MPI doesn`t support what they need it will not be useful (it may still, however, become yet-another-standard.) That said, I am in agreement that there should be an easy-to-use MPI layer and a nice tutorial document for novice users, but don't handicap MPI to accomodate novice users who will migrate away from message-passing as higher-level software tools (HPF, MIMDizer, BlockComm, ....) become available. --dave levine From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 13:27:29 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04067; Thu, 11 Mar 93 13:27:29 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09727; Thu, 11 Mar 93 13:26:20 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 13:26:19 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09719; Thu, 11 Mar 93 13:26:17 -0500 Received: by msr.EPM.ORNL.GOV (5.67/1.34) id AA01747; Thu, 11 Mar 93 13:26:15 -0500 Date: Thu, 11 Mar 93 13:26:15 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9303111826.AA01747@msr.EPM.ORNL.GOV> To: mpi-comm@cs.utk.edu Subject: Re: Message Passing not for the masses scott berryman flames... >even if MPI turns into a huge, ugly mass of routines, There are two problems with this argument. a. at the rate we are going we will never be able to agree on (or document) MPI so there wont be a subset. b. existing packages like NX were not designed to spare performance yet it doesn't have 4 layers and 34 different send variants. Jon and my comment reflect the concern that we will not be able to define MPI-1 in any reasonable time frame if it is a huge, ugly mass of routines, Al From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 14:23:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05158; Thu, 11 Mar 93 14:23:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12286; Thu, 11 Mar 93 14:21:53 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 14:21:52 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from NA-GW.CS.YALE.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12277; Thu, 11 Mar 93 14:21:50 -0500 Received: from YOGI.NA.CS.YALE.EDU by CASPER.NA.CS.YALE.EDU via SMTP; Thu, 11 Mar 1993 14:21:48 -0500 Received: by YOGI.NA.CS.YALE.EDU (Sendmail-5.65c/res.client.cf-3.5) id AA22157; Thu, 11 Mar 1993 14:21:47 -0500 Date: Thu, 11 Mar 1993 14:21:47 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Message-Id: <199303111921.AA22157@YOGI.NA.CS.YALE.EDU> To: mpi-comm@cs.utk.edu Subject: Huge ugly mass of routines Al Giest simmers... Jon and my comment reflect the concern that we will not be able to define MPI-1 in any reasonable time frame if it is a huge, ugly mass of routines, At the risk of agreeing with you, there is a real danger at this point of the huge ugly mass of routines preventing us from do the job in the time alloted. This is a different concern than ease of use. What would you suggest eliminating to simplify the standard? -scott From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 14:37:30 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05455; Thu, 11 Mar 93 14:37:30 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12897; Thu, 11 Mar 93 14:35:33 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 14:35:31 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from chert.CS.ORST.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12882; Thu, 11 Mar 93 14:35:29 -0500 Received: from sycamore.CS.ORST.EDU by research.CS.ORST.EDU (4.1/1.30) id AA16681; Thu, 11 Mar 93 11:35:26 PST From: pancake@chert.CS.ORST.EDU (Cherri Pancake) Received: by sycamore.CS.ORST.EDU (4.1/CS-Client) id AA06719; Thu, 11 Mar 93 11:35:25 PST Date: Thu, 11 Mar 93 11:35:25 PST Message-Id: <9303111935.AA06719@sycamore.CS.ORST.EDU> To: mpi-comm@cs.utk.edu Subject: Response to Berryman's comments Scott says (and I love his "\begin{flame}") -- While I tend to think that there should be some way to figure out how to use MPI in one sitting, I certainly do not think that we should be "maximizing ease of use" as the principle design goal. and then even if MPI turns into a huge, ugly mass of routines, it should still be possible to isolate a few of these routines and present them as the "simple subset" for people who don't need to drive their communication costs down. I don't see "limiting the size of" being the same as "making easy to use". It's certainly try that ease-of-use does not necessarily mean a small number of routines. Ease-of-use is more dependent on: a) If there are subsets, there must be subsets that can function as a reasonal mini-language, in the sense that applications can be developed using just those routines - and getting reasonable performance. b) There must be syntactic/semantic consistency both among routines in a subset and across subset boundaries. Otherwise, users will not be able to learn subsets incrementally with any ease. c) Perhaps most importantly, the names of routines, order and type of arguments, and user-level semantics (by which I mean what can go wrong with respect to the user program) must be consistent and as simple as possible. Cherri From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 15:42:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07612; Thu, 11 Mar 93 15:42:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15959; Thu, 11 Mar 93 15:40:41 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 15:40:39 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15920; Thu, 11 Mar 93 15:40:35 -0500 Received: from elephant by sampson.ccsf.caltech.edu with SMTP id AA16701 (5.65c/IDA-1.4.4 for mpi-comm@cs.utk.edu); Thu, 11 Mar 1993 12:40:32 -0800 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA19196; Thu, 11 Mar 93 12:33:06 PST Received: by lion.parasoft (4.1/SMI-4.1) id AA00810; Thu, 11 Mar 93 12:33:21 PST Date: Thu, 11 Mar 93 12:33:21 PST From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9303112033.AA00810@lion.parasoft> To: mpi-comm@cs.utk.edu Subject: To fan the flames.... To add another layer of hypothetical justification to my argument .......... I think that looking at what we can achieve in the next few meetings is actually only a part of the issue. Probably a larger part of my concern is how long after we produce a draft some reasonably large percentage of the vendors will actually be supporting whatever it is we come up with. Having commitments from vendors to support MPI1 is a long way from actually having an implementation that "kicks #$%$%^$%" in our hot little hands. (Cf. the vendor "acceptance" of OSF for a good example.) My biggest fear is that if MPI-1 is a massive system then the first implementations might be a year after we finish and the first "good" ones (i.e., that actually use some of the really advanced and "neat" stuff that we put into MPI-1 just for that purposes) may take two or more years!! Is that worth it? I don't think so. Of course, I'm probably slandering the vendors and if so I apologise. Perhaps they are better off than I think they are. My impression, however, is that they are struggling to get other stuff solid which they probably would be totally justified in thinking is more important than implementing MPI. Of course this argument doesn't include third parties, either commercial or not, from implementing MPI on top of other systems and giving them to the world. This could clearly happen on a shorter time-scale. However, I expect MPI to fail if this is the best we can offer because people will still find it "convenient" to use vendor specific alternatives that have higher performance. Jon p.s., Just for the sake of it someone asked for a definition of "common practice" - a term that I've bandied about without justification. Without too much thought I think a viable definition might be to take the intersection of all the systems that are currently in use to develop application codes. This is a conceptual intersection, of course, not a strict one since all systems differ in details. This differs in my mind from what we have in the current MPI spec. which might be termed the transitive closure of all the ideas in current use!!! From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 18:15:01 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10139; Thu, 11 Mar 93 18:15:01 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23334; Thu, 11 Mar 93 18:13:26 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 18:13:25 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sun2.nsfnet-relay.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23325; Thu, 11 Mar 93 18:13:18 -0500 Via: uk.ac.southampton.ecs; Thu, 11 Mar 1993 19:07:28 +0000 Via: brewery.ecs.soton.ac.uk; Thu, 11 Mar 93 18:08:41 GMT From: Ian Glendinning Received: from holt.ecs.soton.ac.uk by brewery.ecs.soton.ac.uk; Thu, 11 Mar 93 18:16:56 GMT Date: Thu, 11 Mar 93 18:16:58 GMT Message-Id: <19579.9303111816@holt.ecs.soton.ac.uk> To: mpi-comm@cs.utk.edu Subject: Re: Jon's suggestion > From: hart@gov.noaa.fsl.nipmuc (Leslie Hart) > I also agree. I think that there is significant concern regarding > the near term and long term success of MPI. I think some of the > concern can be aleviated if we settle on a goal that can be reached > by this summer. It may be wise to set up some sort of electronic > straw poll procedure to help get us past some of the issues regarding > what should be in the first MPI, what are our goals for MPI-1, is there > going to be MPI-N (for N>1), what features are most important for MPI-1, etc. I agree. Just when I thought things were converging there's been a flurry of alternative proposals. For example, can we please agree *not* to address concurrent I/O in MPI1, else we'll never get finished within the allotted timescale. On the other hand, I do think that we should decide to have an MPI2, and that that would be the proper way to deal with such issues. I don't think concurrent I/O is an intractible problem, its just that to do a reasonable job on will require more time than we've got right now. Another case in point is dynamic vs static process creation. My vote is for static now, dynamic later. If we can agree that there will definitely be an MPI2, then maybe we can keep everyone happy in the sense that their favourite features *will* all be included, but in a staged fashion. Ian -- I.Glendinning@ecs.soton.ac.uk Ian Glendinning Tel: +44 703 593368 Dept of Electronics and Computer Science Fax: +44 703 593045 University of Southampton SO9 5NH England From owner-mpi-comm@CS.UTK.EDU Fri Mar 12 15:14:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29560; Fri, 12 Mar 93 15:14:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20854; Fri, 12 Mar 93 15:12:27 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 12 Mar 1993 15:12:25 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ocfmail.ocf.llnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20838; Fri, 12 Mar 93 15:12:23 -0500 Received: from [134.9.250.226] (nessett.ocf.llnl.gov) by ocfmail.ocf.llnl.gov (4.1/SMI-4.0) id AA00567; Fri, 12 Mar 93 12:12:15 PST Message-Id: <9303122012.AA00567@ocfmail.ocf.llnl.gov> Date: Fri, 12 Mar 1993 12:20:03 -0800 To: mpi-comm@cs.utk.edu From: nessett@ocfmail.ocf.llnl.gov (Dan Nessett) X-Sender: nessett@ocfmail.llnl.gov Subject: Buffer descriptors and the simple program Cc: rathkopf@llnl.gov, mirin@llnl.gov, jlowens@llnl.gov, geltgroth@llnl.gov, tsw@llnl.gov, seager@llnl.gov, trj@llnl.gov, lstanberry@llnl.gov, bolstad@llnl.gov, zosel@llnl.gov, jbrown@nersc.gov, frobose@llnl.gov, hendrickson1@llnl.gov, batkinson@llnl.gov I met with a number of application and system programmers here at the Lab to get some feedback on the decisions made at the last MPI meeting. One aspect that they felt uncomfortable with was being forced to use the buffer descriptor routines (i.e., bd_create, bd_contiguous, bd_stride and bd_free) in order to just send a simple message. The application programmers want both simple send and receive primitives as well as more general primitives based on buffer descriptors. For example, the Parallel Monte Carlo and Atmospheric Chemistry codes they are working on would benefit from the more general buffer descriptor based primitives, while the Atmospheric Dynamics code uses only simple send and receive. The programmers that use only a single contiguous buffer for their codes would be upset if MPI forced them to make two bd calls to set up the descriptor and then another bd call to free it. (BTW the last call could be eliminated if there were some way to "clear" a buffer descriptor so it could be reused). Dan Nessett From owner-mpi-comm@CS.UTK.EDU Fri Mar 12 15:52:31 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00970; Fri, 12 Mar 93 15:52:31 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22518; Fri, 12 Mar 93 15:51:15 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 12 Mar 1993 15:51:14 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22510; Fri, 12 Mar 93 15:51:09 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA07519 (5.65c/IDA-1.4.4 for ); Fri, 12 Mar 1993 14:51:06 -0600 From: William Gropp Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA21843; Fri, 12 Mar 93 14:51:04 CST Date: Fri, 12 Mar 93 14:51:04 CST Message-Id: <9303122051.AA21843@godzilla.mcs.anl.gov> To: nessett@ocfmail.ocf.llnl.gov Cc: mpi-comm@cs.utk.edu, rathkopf@llnl.gov, mirin@llnl.gov, jlowens@llnl.gov, geltgroth@llnl.gov, tsw@llnl.gov, seager@llnl.gov, trj@llnl.gov, lstanberry@llnl.gov, bolstad@llnl.gov, zosel@llnl.gov, jbrown@nersc.gov, frobose@llnl.gov, hendrickson1@llnl.gov, batkinson@llnl.gov In-Reply-To: Dan Nessett's message of Fri, 12 Mar 1993 12:20:03 -0800 <9303122012.AA00567@ocfmail.ocf.llnl.gov> Subject: Buffer descriptors and the simple program [ message about having to use the "bd_xxx" routines ] This is what the level 2 (or 3, depending on where you draw the line) routines are for. Level 3 contains the "simple snd and receive" with contiguous buffers, while level 1 has everything at the cost of greater complexity. Bill From owner-mpi-comm@CS.UTK.EDU Tue Mar 16 15:54:42 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29714; Tue, 16 Mar 93 15:54:42 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19013; Tue, 16 Mar 93 15:51:37 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 16 Mar 1993 15:51:36 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19005; Tue, 16 Mar 93 15:51:34 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA19519; Tue, 16 Mar 1993 15:51:33 -0500 Date: Tue, 16 Mar 1993 15:51:33 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9303162051.AA19519@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: MPI test implementation An Argonne report by William Gropp and Ewing Lusk dated December 1992 ("A Test Implementation of the MPI Draft Message-Passing Standard", ANL-92/47) has recently been distributed. Please note that the MPI1 routines referred to in this document are those in a draft paper that was circulated as a discussion document. Many of these routines differ from those in the final version of MPI1, or are omitted. The final version of the MPI1 draft standard may be obtained by sending the email message "send mpi1.ps from mpi" to netlib@ornl.gov. For those of you interested in correctly referencing this the bibtex item is as follows: @techreport{MPI1, author = "J. J. Dongarra and R. Hempel and A. J. G. Hey and D. W. Walker", title = "A Proposal for a User-Level, Message Passing Interface in a Distributed Memory Environment", institution = "Oak Ridge National Laboratory", number = "{TM}-12231", month = "March", year = 1993} Regards, David Walker. From owner-mpi-comm@CS.UTK.EDU Tue Mar 16 16:43:35 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01882; Tue, 16 Mar 93 16:43:35 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22479; Tue, 16 Mar 93 16:41:45 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 16 Mar 1993 16:41:44 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22461; Tue, 16 Mar 93 16:41:41 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA13171 (5.65c/IDA-1.4.4 for ); Tue, 16 Mar 1993 15:41:34 -0600 Message-Id: <199303162141.AA13171@antares.mcs.anl.gov> To: walker@rios2.epm.ornl.gov (David Walker) Cc: mpi-comm@cs.utk.edu Subject: Re: MPI test implementation In-Reply-To: Your message of "Tue, 16 Mar 1993 15:51:33 EST." <9303162051.AA19519@rios2.epm.ornl.gov> Date: Tue, 16 Mar 1993 15:41:32 -0600 From: Rusty Lusk Thanks for setting the record straight. That Tech. Report went into the Argonne mill in early December, and we had no idea that it would take it this long to be distributed and therefore how out-of-date it would be when it finally did get mailed. - Rusty From owner-mpi-comm@CS.UTK.EDU Tue Mar 16 20:54:58 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06431; Tue, 16 Mar 93 20:54:58 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09400; Tue, 16 Mar 93 20:53:03 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 16 Mar 1993 20:53:01 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from iliamna.cse.ogi.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09390; Tue, 16 Mar 93 20:53:00 -0500 Received: by iliamna.cse.ogi.edu (/\==/\ Smail3.1.25.1 #25.15) id ; Tue, 16 Mar 93 17:52 PST Message-Id: Date: Tue, 16 Mar 93 17:52 PST From: otto@iliamna.cse.ogi.edu (Steve Otto) To: mpi-comm@cs.utk.edu Subject: PostScript version of the current MPI Draft Document A (compressed) PostScript version of the current MPI Draft Document is available via anonyomous ftp at cse.ogi.edu: ftp cse.ogi.edu cd pub/otto binary get draft.ps.Z quit This should get you a copy. You then need to uncompress it. All the source is there also if you are interested. --Steve Otto From owner-mpi-comm@CS.UTK.EDU Thu Mar 18 08:39:18 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10600; Thu, 18 Mar 93 08:39:18 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04620; Thu, 18 Mar 93 08:36:29 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 18 Mar 1993 08:36:28 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04612; Thu, 18 Mar 93 08:36:21 -0500 Date: Thu, 18 Mar 93 13:36:16 GMT Message-Id: <9166.9303181336@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: MINUTES! To: mpi-comm@cs.utk.edu In-Reply-To: David Walker's message of Wed, 17 Mar 1993 09:57:50 -0500 Reply-To: lyndon@epcc.ed.ac.uk Hi all On Thursday of last week I enquired about the minutes of the last meeting. Dave Walker writes: > The minutes of the last meeting in February are not yet available > electronically, but should be soon. I observe that subcommittee chairs have already circulated revised "working documents" for subcommittees. The minutes of previous meeting are the primary reference for results of subcommittee straw polls which must surely be the primary determinants of working document revisions. It is now past time when the minutes were essential. Please, can these be distributed. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Thu Mar 18 11:18:58 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14049; Thu, 18 Mar 93 11:18:58 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11559; Thu, 18 Mar 93 11:17:00 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 18 Mar 1993 11:16:59 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11550; Thu, 18 Mar 93 11:16:51 -0500 Date: Thu, 18 Mar 93 16:16:47 GMT Message-Id: <9298.9303181616@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: AGENDA! To: walker@rios2.epm.ornl.gov (David Walker) In-Reply-To: David Walker's message of Thu, 18 Mar 1993 10:48:16 -0500 Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-comm@cs.utk.edu David writes: > > I don't think Lyndon's suggestion to make the timetable session the first of > the meeting is a good idea. How can a timetable be set if we don't know > what's to be included, > and if we don't know how much of the current proposals > are acceptable. > I see your point on the matter of establishing a firm timetable. It seems to me, and there has been a fair bit of traffic on it, that we need to establish boundaries for what will be in MPI. It seems to me that this needed to be the first thing to thrash out at the next meeting. I agree with you that timetables should be sorted out later. > We could move the session on contexts, groups etc. up front I guess. This seems to mean that the context subcommittee has to come to the "report and proposal to whole committee" stage without the opportunity to meet. My understanding was that we had intended to reach that stage at the next meeting, after a subcommittee meeting. Personally I can handle doing this by email over the next 10 days, but I'm not sure that the whole of the subcommittee can. Thanks for the comments, and pointing out my error in supposing that the timetable could be determined at the beginning of the meeting. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Thu Mar 18 12:54:40 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA15727; Thu, 18 Mar 93 12:54:40 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16896; Thu, 18 Mar 93 12:50:45 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 18 Mar 1993 12:50:43 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16880; Thu, 18 Mar 93 12:50:38 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA14802; Thu, 18 Mar 93 17:44:46 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA20644; Thu, 18 Mar 93 10:43:30 MST Date: Thu, 18 Mar 93 10:43:30 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303181743.AA20644@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: MPI Priorities and Scope: Opinion Survey Cc: jwf@lion.parasoft.com, lederman@b125.super.org, igl@ecs.soton.ac.uk, lyndon@epcc.ed.ac.uk Hi all, Steve Huss-Lederman writes: > I would like to propose that we have a meeting of all participants at > the start of the next meeting to settle the issue of what MPI-1 will > cover. Several mail messages have expressed concern that we cannot > conclude by June if we try to incorporate everything that everyone > wants. Since the scope and time frame will effect most of the other > discussions I think it would be best to begin with this. My only > concern is that it will take too long to decide the global picture. > I would hope it could be done in 1-2 hours and help to refocus the > group. I fully support this idea. Here's a proposal for an opinion survey that could shorten and focus the discussion. This proposal originated with Jon Flower and has had input from Ian Glendinning, Steve Huss-Lederman, Leslie Hart, myself, and others. If you think this is a good idea, please complete the survey and return to me at: hender@fsl.noaa.gov If you think this is a bad idea, please say so! We will collect responses and distribute a summary via email before the next meeting if there is sufficient interest. Tom Henderson ----------------------------------------------------------------------------- MPI PRIORITIES OPINION SURVEY Several MPI committee members have expressed concern about the current and future directions of MPI. The MPI committee began discussion using an existing document (Dongarra et al) as a starting point. This document contains background rationale and a statement of direction for the MPI standardization effort. Since these issues were consistent with the original MPI proposal, they have not recently been discussed in detail by the MPIF. The current MPI proposals are very different from the original MPI document. As a result, we feel that the original rationale and direction need to be re-examined. In particular we need to prioritize and limit the issues that MPI will try to resolve so that o the initial MPI standard document ("MPI-1") is delivered in a timely fashion and o the functions that it defines can (and will) be implemented on major platforms within a reasonable time frame. To get discussion of these issues started, we propose an informal, email "opinion survey". The objective of the survey is to find out which MPI features are most important. High-priority features would be included in MPI-1. Lower-priority features might be deferred to a later version of MPI. Some features might never be included. The compiled results of this survey could be used as a basis for discussion at the next MPI meeting. *** This survey is in no way binding and is a completely unofficial "straw poll" arranged by its authors for their own personal information. It is not intended to distract attention from the current proposals. Any data collected will be made available to the group at large if requested. *** Following is a ballot containing possible MPI features and goals. Please rate each item (or sub-item) using the following rating scheme: 0 This feature should not appear in any version of MPI ever. 1-2 MPIF should consider including this feature in some future version of MPI (after MPI-1). 3-4 This feature should be included as an Annex to MPI-1 for possible inclusion in a future version of MPI. 5-6 This feature should be included as an Annex to MPI-1 for definite inclusion in a future version of MPI. 7-10 This feature should be included in MPI-1. In this context an "Annex" is a part of the standard document that will not be required by a conforming implementation of MPI. Rather it should be taken as a recommendation by the MPIF to vendors for the way in which particular features should be implemented, if they are implemented at all. This way features that are not usefully implementable on particular hardware can be eliminated while machines that do support these features do so in a portable manner. When replying to the survey, please use the response form attached to the end of this message. (This will make compilation of the results a lot less painful!) :-) +-------------------------------------------------------+ | POSSIBLE FEATURES FOR INITIAL AND FUTURE MPI VERSIONS | +-------------------------------------------------------+ Ordering is for convenience only. This list is not meant to suggest any priority. GENERAL FEATURES 1) Routines for static process creation: A) Same program in every node. B) Host-node programming model in which there is one "special node". C) Static model but with potentially different programs in different groups of nodes. D) Static model with a way to create more than one process per processor. (Heavy duty processes, not threads). 2) Routines for identification of statically created processes. 3) Associated environmental inquiry routines. 4) Some way of avoiding message tag conflicts between independently developed libraries and user code (contexts). 5) Heterogeneous communication. A) All communication routines should support heterogeneous communication. B) A special set of communication routines should support heterogeneous communication. 6) Creation/destruction of "static" groups (processes cannot join/leave an existing group). Groups are used to limit the scope of collective operations. A) No logical process topology is associated with a group. B) Grid/toroidal process topology is associated with a group at the time the group is created. (The default "ALL" group has a default topology.) C) General graph topology is also supported. D) Provide access to all relevant information (via topological environmental inquiry routines) to allow users to define custom topologies. 7) Support for communication between processes in different groups without reference to the "ALL" group. 8) Dynamic groups (processes can join/leave an existing group while the total number of processes remains static.) 9) Dynamic process creation and identification. 10) Additional environmental inquiry and "setting" routines for performance optimization. A) Inquire/suggest system buffering resource allocation. B) Inquire relative node performance. C) Inquire relative interconnect performance. 11) Termination of programs. A) Kill all others nodes. B) Wait for all the other nodes to complete. C) Automatically de-allocate resources. 12) Standard organization. A) "Multi-level" organization of routines (users may use simple features without having to understand all of MPI). B) Simple "core" standard with (optional) Annexes. C) Simple "core" standard without Annexes. D) Should an MPI "subset" be defined that can be quickly implemented as a layer on top of existing systems? (This question was inspired by section 1.5 of Bill and Rusty's "A Test Implementation of the MPI Draft Message-Passing Standard", ANL-92/47.) 13) Should there be future version(s) of MPI (MPI-2, etc.) (yes/no)? POINT-TO-POINT COMMUNICATION 14) Blocking send and receive. A) Contiguous data. B) Strided data. C) Buffer-descriptor (multiple data locations, multiple data types). 15) Non-blocking receive. A) Contiguous data. B) Strided data. C) Buffer-descriptor (multiple data locations, multiple data types). D) Check for completion (status()). E) Wait for completion (wait()). F) Cancel operation (cancel()). 16) Non-blocking send. A) Contiguous data. B) Strided data. C) Buffer-descriptor (multiple data locations, multiple data types). D) Check for completion (status()). E) Wait for completion (wait()). F) Cancel operation (cancel()). 17) Low-level functionality (from Marc Snir's proposal) visible to users. A) Initialize. B) Start. C) Complete. D) Free. 18) "SECURE" versions of send and receive as proposed by Lyndon Clarke on 3/3/93. (Correct programs using SECURE routines are guaranteed to work regardless of system buffering resources.) 19) Ready-send. 20) Ready-receive. 21) Message handlers (a la iNTEL hrecv, TMC active messages, Express exhandle). A) Simple definition sufficient to support anticipated MPI requirements and make a self-consistent system. B) More advanced routines for remote procedure calls, etc... 22) Additional point-to-point primitives. A) Exchange between two processors. B) "Shift-exchange": read from one, send to another (superset of "A"). C) Contiguous data. D) Strided data. E) Buffer-descriptor (multiple data locations, multiple data types). F) Non-blocking versions? G) Check for completion (status()). H) Wait for completion (wait()). I) Cancel operation (cancel()). COLLECTIVE OPERATIONS 23) Blocking collective communication. A) Broadcast. B) Concatenate. C) Barrier. D) Sum. E) Extrema (max, min). F) Average. G) Sum of squared differences (error). H) Should strided data be allowed for these operations? I) Should buffer-descriptor be allowed for these operations? 24) "User-defined" blocking collective operations (Express "excombine()" style). 25) Non-blocking collective communication, collective operations, and user-defined operations (if this is desired, detailed ranking can be done later). 26) If any non-blocking collective operations are supported, what should completion/cancel options be? A) Check for completion (status()). B) Wait for completion (wait()). C) "Multiple cancel" operation (cancel()). 27) Should status()/wait()/cancel() syntax be consistent across all types of non-blocking operations? OTHER COMMUNICATION OPERATIONS 28) Probe (and associated "precv()" routines, if necessary). I/O 29) Standard I/O is supported in the standard fashion for the language on one process (no file I/O). This process can be identified by an environmental inquiry routine. (This is the current proposal.) 30) Serial I/O is supported in the standard fashion for the language on one process. This process can be identified by an environmental inquiry routine. (Rik Littlefield's proposal.) 31) Multiplexed I/O similar to that currently provided in systems such as CMMD (TMC), VERTEX (nCUBE) and Express (ParaSoft). Basically a way of defining the behavior of multi-node I/O statements which interact with a sequential device. No special considerations for parallel I/O channels. 32) Blocking parallel I/O. Intended to take advantage of parallel I/O channels, where available. 33) Non-blocking parallel I/O. Intended to take advantage of parallel I/O channels, where available. LANGUAGE BINDINGS 34) Ada 35) C 36) C++ 37) Fortran 77 38) Fortran 90 39) Other IMPLEMENTATION PERIOD There will be a delay between the generation of the standard and its implementation on various platforms. If this delay is too long users will continue to rely more and more heavily on vendor routines and specialized calls and MPI will become less and less attractive, particularly if new technology such as Fortran 90/HPF becomes effective. 40) What is the maximum delay that you think MPI can stand and still succeed (please feel free to comment if you wish)? ---------------------------------------------------------------------------- SURVEY RESPONSE FORM 1) A) B) C) D) 2) 3) 4) 5) A) B) 6) A) B) C) D) 7) 8) 9) 10) A) B) C) 11) A) B) C) 12) A) B) C) D) 13) (yes/no) 14) A) B) C) 15) A) B) C) D) E) F) 16) A) B) C) D) E) F) 17) A) B) C) D) 18) 19) 20) 21) A) B) 22) A) B) C) D) E) F) G) H) I) 23) A) B) C) D) E) F) G) H) I) 24) 25) 26) A) B) C) 27) 28) 29) 30) 31) 32) 33) 34) 35) 36) 37) 38) 39) 40) (free form): Comments? From owner-mpi-comm@CS.UTK.EDU Thu Mar 18 14:28:27 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17761; Thu, 18 Mar 93 14:28:27 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22028; Thu, 18 Mar 93 14:22:46 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 18 Mar 1993 14:22:45 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ssd.intel.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22020; Thu, 18 Mar 93 14:22:43 -0500 Received: from tualatin.SSD.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA23272; Thu, 18 Mar 93 11:22:16 PST Date: Thu, 18 Mar 93 11:22:16 PST Message-Id: <9303181922.AA23272@SSD.intel.com> Received: by tualatin.SSD.intel.com (4.1/SMI-4.0) id AA10070; Thu, 18 Mar 93 11:22:13 PST From: Bob Knighten Sender: knighten@SSD.intel.com To: lyndon@epcc.ed.ac.uk Cc: mpi-comm@cs.utk.edu Subject: Re: MINUTES! In-Reply-To: <9166.9303181336@subnode.epcc.ed.ac.uk> References: <9166.9303181336@subnode.epcc.ed.ac.uk> Reply-To: knighten@SSD.intel.com (Bob Knighten) Yes, the minutes are late and yes, they will be available soon. Rusty has been traveling extensively since the meeting and now that I am supposed to finish them I have had some personal emergencies and my company has decided there are more important ways to spend my time. That is just an explanation of the delay, I am well aware of the importance of the minutes and will have them out shortly. -- Bob Robert L. Knighten | knighten@ssd.intel.com Intel Supercomputer Systems Division | 15201 N.W. Greenbrier Pkwy. | (503) 629-4315 Beaverton, Oregon 97006 | (503) 629-9147 [FAX] From owner-mpi-comm@CS.UTK.EDU Thu Mar 25 10:48:01 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17196; Thu, 25 Mar 93 10:48:01 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21805; Thu, 25 Mar 93 10:46:00 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Mar 1993 10:45:59 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21766; Thu, 25 Mar 93 10:45:43 -0500 Date: Thu, 25 Mar 93 15:44:59 GMT Message-Id: <18119.9303251544@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Helpful Summary of Contexts Proposals To: mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: tony@aurora.cs.msstate.edu, nbm@castle.ed.ac.uk, bobf@epcc.ed.ac.uk Dear MPI Colleagues I imagine that many of you have started or are about to start reading the three contexts subcommittee proposals. We have prepared a comparative, and non judgmental, summary of the three proposals which may be of some assistance to MPI. The three proposals in the context subcommittee share common features, and have differences both in concept and detail. Two of these proposals contain features which are "separable" and could equally appear as components of one or more other proposals. Hopefully the summary will: (a) help us to discuss the important differnces between the proposals and make agreements on how we should proceed with respect to those issues; (b) help us to isolate the separable points and make separate agreements on those issues. I hope that the summary is both accurate and complete. Please make corrections and additions if you discover such. I apologise in advance for my errors, which are surely inevitable. Best Wishes Lyndon ---------------------------------------------------------------------- Summary of context subcommittee proposals ***************************************** The three proposals in the context subcommittee share common features, and have differences both in concept and detail. Two of these proposals contain features which are "separable" and could equally appear as components of one or more other proposals. This summary identifies feature of proposals as: Common Features; Separable Features; Concept Differences; Detail Differences. Common Features =============== 1. Process group management --------------------------- In each proposal groups are created dynamically and have static membership. In each proposal a group can be created as a partition of an existing group and as a permutation of an existing group. Each proposal allows (or suggests) that a group can be created as an explicit list of processes. 2. Provision for point-to-point communication within group. ----------------------------------------------------------- In each proposal point-to-point communcation of scope closed within a group can be expressed in terms of a reference to a group coupled with a process rank within the group. 3. Provision for collective communication within group. ------------------------------------------------------ In each proposal collective communication of scope closed within a group can be expressed in terms of a reference to a group. 4. Opacity of group and process description. -------------------------------------------- In each proposal the description of groups and processes is opaque. Groups and processes are referred to by a handle like object. Separable Features ================== 1. Tag usage in point-to-point communication. --------------------------------------------- Proposal III describes tag selection for Receive in a two-integer form. Proposals I and VII say nothing about tag usage. This feature can be placed in all Proposals I, III and VII. [Historical note: Tony did say to methat this would appear as an appendix with our mutual recognition that it can equally appear as a feature of any of the proposals.] 2. Tag usage in collective communication. ----------------------------------------- Proposal III suggests that tag should be used as an argument to collective communication where this will assist debugging. Proposals I and VII say nothing about usage. This feature can be placed in all Proposals I, III and VII. 3. Context or Group cache ------------------------- Proposal VII decribes a cache facility associated with contexts and groups. Proposal III describes a similar cache facility associated with groups. This feature can be placed in all Proposals I, III and VII. 4. Opaque object (descriptor) transmission. ------------------------------------------- Proposal VII suggests that opaque object transmission can be provided by integration with transmission of typed data. Proposal III suggests that opaque transmission is provided by a mechanism for flattening a descriptor into a memory buffer. These are just details of different ways of providing the feature. This feature can be placed in Proposals III and VII. This feature cannot be placed in Proposal I. 5. Context registry. -------------------- Proposal III describes a context name registry service. Proposal VII indicates that such a service would be useful. This feature can be placed in Proposals III and VII. This feature cannot be placed in Proposal I. Concept Differences. ==================== 1. Concept of CONTEXT and GROUP ------------------------------- In Proposal I CONTEXT and GROUP are identical concepts and are not distinguished. In Proposal III CONTEXT is a lower degree concept that GROUP. The GROUP concept inherits aspects of the CONTEXT concept. In Proposal VII CONTEXT is a higher concept than GROUP. The CONTEXT concept inherits aspects of the GROUP concept. 2. Scope of point-to-point communication. ----------------------------------------- In Proposal I the scope of point-to-point communication is limited to the CONTEXT. Processes which are members of distinct groups can only communicate through a common ancestor group. In Proposals III and VII the scope of point-to-point communciation is not limited. Processes which are members of distinct groups can communicate without reference to a common ancestor group. 3. Transmission of group or context. ------------------------------------ In Proposal I the CONTEXT cannot be transmitted from one process to another. In Proposals VII and III both CONTEXT and GROUP can be transmitted from one process to another. In Proposal VII PROCESS can alo be transmitted (Proposal III suggests such but makes no specific provision, presumably a small oversight?) Detail differences. =================== 1. Manifestation of context --------------------------- In Proposals I and VII context is an opaque object. In Proposal III context is an integer(?). 2. Deletion of group. --------------------- In Proposals VII and III groups can be deleted. In Proposal I there is no provision for group deletion (possibly a small oversight?). 3. Duplication of group. ------------------------ In Proposals I and III a group there is explicit provision for duplication of an existing group to form a new (distinct, homomorphic) group. In Proposal VII there is no such provision as similar funtionality is provided by the context (although the provision for group partition, permutation and definition can be used to create a snapshot copy of a group). 4. Global shared variables. --------------------------- Proposals I and VII do not require global shared variables. Proposal III requires a global shared variable (which can be implemented as such or of course in the traditional approach as a global service process.) 3. Process identifier addressed communication. ---------------------------------------------- Proposal I does not make provision for process identifier addressed communication. Proposal III makes provision for process identifier addressed communication within multiple distinct tag spaces. Proposal VII makes provision for process identifier addressed communication within a single distinct tag space. 5. Inter-group communication. ----------------------------- Proposal I does not provide inter-group communication as it limits the scope of point-to-point communication to be closed within a group. Proposal VII provides inter-group communication in an addressing form specified by sender (receiver) group, receiver (sender) group and sender (receiver) rank. Proposal III provides inter-group communication as process identifier addressed communication. ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Thu Mar 25 12:41:24 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21413; Thu, 25 Mar 93 12:41:24 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26938; Thu, 25 Mar 93 12:40:11 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Mar 1993 12:40:10 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26850; Thu, 25 Mar 93 12:39:35 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA07566; Thu, 25 Mar 93 11:33:39 CST Date: Thu, 25 Mar 93 11:33:39 CST From: Tony Skjellum Message-Id: <9303251733.AA07566@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu, lyndon@epcc.ed.ac.uk Subject: Re: Helpful Summary of Contexts Proposals Cc: tony@aurora.cs.msstate.edu, nbm@castle.ed.ac.uk, bobf@epcc.ed.ac.uk Thank you for this additional work, Lyndon. Why don't we include this as the preface to our proposal? It looks good, but I will have to read it in detail, before rendering my complete view on this matter. Who are these other, new cc people... - Tony ----- Begin Included Message ----- From owner-mpi-comm@CS.UTK.EDU Thu Mar 25 10:07:51 1993 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Mar 1993 10:45:59 EST Date: Thu, 25 Mar 93 15:44:59 GMT From: L J Clarke Subject: Helpful Summary of Contexts Proposals To: mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: tony@aurora.cs.msstate.edu, nbm@castle.ed.ac.uk, bobf@epcc.ed.ac.uk Content-Length: 8050 Dear MPI Colleagues I imagine that many of you have started or are about to start reading the three contexts subcommittee proposals. We have prepared a comparative, and non judgmental, summary of the three proposals which may be of some assistance to MPI. The three proposals in the context subcommittee share common features, and have differences both in concept and detail. Two of these proposals contain features which are "separable" and could equally appear as components of one or more other proposals. Hopefully the summary will: (a) help us to discuss the important differnces between the proposals and make agreements on how we should proceed with respect to those issues; (b) help us to isolate the separable points and make separate agreements on those issues. I hope that the summary is both accurate and complete. Please make corrections and additions if you discover such. I apologise in advance for my errors, which are surely inevitable. Best Wishes Lyndon ---------------------------------------------------------------------- Summary of context subcommittee proposals ***************************************** The three proposals in the context subcommittee share common features, and have differences both in concept and detail. Two of these proposals contain features which are "separable" and could equally appear as components of one or more other proposals. This summary identifies feature of proposals as: Common Features; Separable Features; Concept Differences; Detail Differences. Common Features =============== 1. Process group management --------------------------- In each proposal groups are created dynamically and have static membership. In each proposal a group can be created as a partition of an existing group and as a permutation of an existing group. Each proposal allows (or suggests) that a group can be created as an explicit list of processes. 2. Provision for point-to-point communication within group. ----------------------------------------------------------- In each proposal point-to-point communcation of scope closed within a group can be expressed in terms of a reference to a group coupled with a process rank within the group. 3. Provision for collective communication within group. ------------------------------------------------------ In each proposal collective communication of scope closed within a group can be expressed in terms of a reference to a group. 4. Opacity of group and process description. -------------------------------------------- In each proposal the description of groups and processes is opaque. Groups and processes are referred to by a handle like object. Separable Features ================== 1. Tag usage in point-to-point communication. --------------------------------------------- Proposal III describes tag selection for Receive in a two-integer form. Proposals I and VII say nothing about tag usage. This feature can be placed in all Proposals I, III and VII. [Historical note: Tony did say to methat this would appear as an appendix with our mutual recognition that it can equally appear as a feature of any of the proposals.] 2. Tag usage in collective communication. ----------------------------------------- Proposal III suggests that tag should be used as an argument to collective communication where this will assist debugging. Proposals I and VII say nothing about usage. This feature can be placed in all Proposals I, III and VII. 3. Context or Group cache ------------------------- Proposal VII decribes a cache facility associated with contexts and groups. Proposal III describes a similar cache facility associated with groups. This feature can be placed in all Proposals I, III and VII. 4. Opaque object (descriptor) transmission. ------------------------------------------- Proposal VII suggests that opaque object transmission can be provided by integration with transmission of typed data. Proposal III suggests that opaque transmission is provided by a mechanism for flattening a descriptor into a memory buffer. These are just details of different ways of providing the feature. This feature can be placed in Proposals III and VII. This feature cannot be placed in Proposal I. 5. Context registry. -------------------- Proposal III describes a context name registry service. Proposal VII indicates that such a service would be useful. This feature can be placed in Proposals III and VII. This feature cannot be placed in Proposal I. Concept Differences. ==================== 1. Concept of CONTEXT and GROUP ------------------------------- In Proposal I CONTEXT and GROUP are identical concepts and are not distinguished. In Proposal III CONTEXT is a lower degree concept that GROUP. The GROUP concept inherits aspects of the CONTEXT concept. In Proposal VII CONTEXT is a higher concept than GROUP. The CONTEXT concept inherits aspects of the GROUP concept. 2. Scope of point-to-point communication. ----------------------------------------- In Proposal I the scope of point-to-point communication is limited to the CONTEXT. Processes which are members of distinct groups can only communicate through a common ancestor group. In Proposals III and VII the scope of point-to-point communciation is not limited. Processes which are members of distinct groups can communicate without reference to a common ancestor group. 3. Transmission of group or context. ------------------------------------ In Proposal I the CONTEXT cannot be transmitted from one process to another. In Proposals VII and III both CONTEXT and GROUP can be transmitted from one process to another. In Proposal VII PROCESS can alo be transmitted (Proposal III suggests such but makes no specific provision, presumably a small oversight?) Detail differences. =================== 1. Manifestation of context --------------------------- In Proposals I and VII context is an opaque object. In Proposal III context is an integer(?). 2. Deletion of group. --------------------- In Proposals VII and III groups can be deleted. In Proposal I there is no provision for group deletion (possibly a small oversight?). 3. Duplication of group. ------------------------ In Proposals I and III a group there is explicit provision for duplication of an existing group to form a new (distinct, homomorphic) group. In Proposal VII there is no such provision as similar funtionality is provided by the context (although the provision for group partition, permutation and definition can be used to create a snapshot copy of a group). 4. Global shared variables. --------------------------- Proposals I and VII do not require global shared variables. Proposal III requires a global shared variable (which can be implemented as such or of course in the traditional approach as a global service process.) 3. Process identifier addressed communication. ---------------------------------------------- Proposal I does not make provision for process identifier addressed communication. Proposal III makes provision for process identifier addressed communication within multiple distinct tag spaces. Proposal VII makes provision for process identifier addressed communication within a single distinct tag space. 5. Inter-group communication. ----------------------------- Proposal I does not provide inter-group communication as it limits the scope of point-to-point communication to be closed within a group. Proposal VII provides inter-group communication in an addressing form specified by sender (receiver) group, receiver (sender) group and sender (receiver) rank. Proposal III provides inter-group communication as process identifier addressed communication. ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Thu Mar 25 14:01:25 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23412; Thu, 25 Mar 93 14:01:25 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00787; Thu, 25 Mar 93 13:59:45 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Mar 1993 13:59:44 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00747; Thu, 25 Mar 93 13:58:34 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA07654; Thu, 25 Mar 93 12:52:38 CST Date: Thu, 25 Mar 93 12:52:38 CST From: Tony Skjellum Message-Id: <9303251852.AA07654@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu, lyndon@epcc.ed.ac.uk Subject: Re: Helpful Summary of Contexts Proposals Cc: tony@aurora.cs.msstate.edu, nbm@castle.ed.ac.uk, bobf@epcc.ed.ac.uk Lyndon, as you know I was immensely under the gun yesterday, and my 9-hour trip was quite an ordeal, but I did get the sub-committee's proposal in on time, and it is basically quite good. I do feel that it is appropriate to include comparisons and even criticisms in the conclusions, but it was not done evenly, since mine was last. To this extent, I see why you did not like that. In same light, I can see why we should drop your name and Rik's name from proposal III, as this might unfairly indicate your support for proposal III (which is admittedly, an extended version of the programming model I support most, and is what Zipcode does, plus some additions). Please restrain yourself from completely dominating this subcommittee, because of the remarkable amount of time you are able to devote to it. To be fair, I cannot devote more than 2hr/day, and you are able to devote something like 12hr/day to it. People at the SIAM meeting were overwhelmed by the volume of mail you generate on different topics. You are a star performer, but you are also very demanding, and I must say that I have had some aggravations in this last weeks, but not only from you. I am trying not to push my personal viewpoint too hard. Please work from a viewpoint of cooperation with me, as we move towards the meeting next week. In this vein, see next paragraph. I will read your summary, but I already agree (note: agree) to drop all conclusion comments from all three subproposals that are "judgemental" (a loaded word). Since we are making a report out of this, the comparison should go in as latex. I will give this to Otto today (I will give it to him). Do you want me to latex your contribution, or will you (it is not too late there). Tell me. Ala Frankie & Johnny, I have to hang up now, since there is more mail from you to read :-)... I will not respond to that which is outdated by this letter. Please work on a latex version of your excellent new contribution, which also makes our joint report much better. I thank you. PS For rest of committee, the new contribution by Lyndon does not invalidate yesterday's tripartite proposal, it will merely be included in it. ----- Begin Included Message ----- From owner-mpi-comm@CS.UTK.EDU Thu Mar 25 10:07:51 1993 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Mar 1993 10:45:59 EST Date: Thu, 25 Mar 93 15:44:59 GMT From: L J Clarke Subject: Helpful Summary of Contexts Proposals To: mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: tony@aurora.cs.msstate.edu, nbm@castle.ed.ac.uk, bobf@epcc.ed.ac.uk Content-Length: 8050 Dear MPI Colleagues I imagine that many of you have started or are about to start reading the three contexts subcommittee proposals. We have prepared a comparative, and non judgmental, summary of the three proposals which may be of some assistance to MPI. The three proposals in the context subcommittee share common features, and have differences both in concept and detail. Two of these proposals contain features which are "separable" and could equally appear as components of one or more other proposals. Hopefully the summary will: (a) help us to discuss the important differnces between the proposals and make agreements on how we should proceed with respect to those issues; (b) help us to isolate the separable points and make separate agreements on those issues. I hope that the summary is both accurate and complete. Please make corrections and additions if you discover such. I apologise in advance for my errors, which are surely inevitable. Best Wishes Lyndon ---------------------------------------------------------------------- Summary of context subcommittee proposals ***************************************** The three proposals in the context subcommittee share common features, and have differences both in concept and detail. Two of these proposals contain features which are "separable" and could equally appear as components of one or more other proposals. This summary identifies feature of proposals as: Common Features; Separable Features; Concept Differences; Detail Differences. Common Features =============== 1. Process group management --------------------------- In each proposal groups are created dynamically and have static membership. In each proposal a group can be created as a partition of an existing group and as a permutation of an existing group. Each proposal allows (or suggests) that a group can be created as an explicit list of processes. 2. Provision for point-to-point communication within group. ----------------------------------------------------------- In each proposal point-to-point communcation of scope closed within a group can be expressed in terms of a reference to a group coupled with a process rank within the group. 3. Provision for collective communication within group. ------------------------------------------------------ In each proposal collective communication of scope closed within a group can be expressed in terms of a reference to a group. 4. Opacity of group and process description. -------------------------------------------- In each proposal the description of groups and processes is opaque. Groups and processes are referred to by a handle like object. Separable Features ================== 1. Tag usage in point-to-point communication. --------------------------------------------- Proposal III describes tag selection for Receive in a two-integer form. Proposals I and VII say nothing about tag usage. This feature can be placed in all Proposals I, III and VII. [Historical note: Tony did say to methat this would appear as an appendix with our mutual recognition that it can equally appear as a feature of any of the proposals.] 2. Tag usage in collective communication. ----------------------------------------- Proposal III suggests that tag should be used as an argument to collective communication where this will assist debugging. Proposals I and VII say nothing about usage. This feature can be placed in all Proposals I, III and VII. 3. Context or Group cache ------------------------- Proposal VII decribes a cache facility associated with contexts and groups. Proposal III describes a similar cache facility associated with groups. This feature can be placed in all Proposals I, III and VII. 4. Opaque object (descriptor) transmission. ------------------------------------------- Proposal VII suggests that opaque object transmission can be provided by integration with transmission of typed data. Proposal III suggests that opaque transmission is provided by a mechanism for flattening a descriptor into a memory buffer. These are just details of different ways of providing the feature. This feature can be placed in Proposals III and VII. This feature cannot be placed in Proposal I. 5. Context registry. -------------------- Proposal III describes a context name registry service. Proposal VII indicates that such a service would be useful. This feature can be placed in Proposals III and VII. This feature cannot be placed in Proposal I. Concept Differences. ==================== 1. Concept of CONTEXT and GROUP ------------------------------- In Proposal I CONTEXT and GROUP are identical concepts and are not distinguished. In Proposal III CONTEXT is a lower degree concept that GROUP. The GROUP concept inherits aspects of the CONTEXT concept. In Proposal VII CONTEXT is a higher concept than GROUP. The CONTEXT concept inherits aspects of the GROUP concept. 2. Scope of point-to-point communication. ----------------------------------------- In Proposal I the scope of point-to-point communication is limited to the CONTEXT. Processes which are members of distinct groups can only communicate through a common ancestor group. In Proposals III and VII the scope of point-to-point communciation is not limited. Processes which are members of distinct groups can communicate without reference to a common ancestor group. 3. Transmission of group or context. ------------------------------------ In Proposal I the CONTEXT cannot be transmitted from one process to another. In Proposals VII and III both CONTEXT and GROUP can be transmitted from one process to another. In Proposal VII PROCESS can alo be transmitted (Proposal III suggests such but makes no specific provision, presumably a small oversight?) Detail differences. =================== 1. Manifestation of context --------------------------- In Proposals I and VII context is an opaque object. In Proposal III context is an integer(?). 2. Deletion of group. --------------------- In Proposals VII and III groups can be deleted. In Proposal I there is no provision for group deletion (possibly a small oversight?). 3. Duplication of group. ------------------------ In Proposals I and III a group there is explicit provision for duplication of an existing group to form a new (distinct, homomorphic) group. In Proposal VII there is no such provision as similar funtionality is provided by the context (although the provision for group partition, permutation and definition can be used to create a snapshot copy of a group). 4. Global shared variables. --------------------------- Proposals I and VII do not require global shared variables. Proposal III requires a global shared variable (which can be implemented as such or of course in the traditional approach as a global service process.) 3. Process identifier addressed communication. ---------------------------------------------- Proposal I does not make provision for process identifier addressed communication. Proposal III makes provision for process identifier addressed communication within multiple distinct tag spaces. Proposal VII makes provision for process identifier addressed communication within a single distinct tag space. 5. Inter-group communication. ----------------------------- Proposal I does not provide inter-group communication as it limits the scope of point-to-point communication to be closed within a group. Proposal VII provides inter-group communication in an addressing form specified by sender (receiver) group, receiver (sender) group and sender (receiver) rank. Proposal III provides inter-group communication as process identifier addressed communication. ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Fri Mar 26 18:57:02 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA28001; Fri, 26 Mar 93 18:57:02 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17683; Fri, 26 Mar 93 18:54:46 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 26 Mar 1993 18:54:44 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from iliamna.cse.ogi.edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17674; Fri, 26 Mar 93 18:54:43 -0500 Received: by iliamna.cse.ogi.edu (/\==/\ Smail3.1.25.1 #25.15) id ; Fri, 26 Mar 93 15:54 PST Message-Id: Date: Fri, 26 Mar 93 15:54 PST From: otto@iliamna.cse.ogi.edu (Steve Otto) To: mpi-comm@cs.utk.edu Subject: Draft available on netlib A PostScript version of the current MPI Draft Document is available on netlib. It is called draft326.ps.z.uu and is a postscript, compressed, uuencoded file. The size is 1830521 bytes. send draft326.ps.z.uu from mpi will get the file, or you may want to pull it over using xnetlib. --Steve Otto From owner-mpi-comm@CS.UTK.EDU Fri Mar 26 19:42:13 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA28702; Fri, 26 Mar 93 19:42:13 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19291; Fri, 26 Mar 93 19:40:36 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 26 Mar 1993 19:40:33 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from wk46.nas.nasa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19262; Fri, 26 Mar 93 19:40:26 -0500 Received: by wk46.nas.nasa.gov (5.61/1.34) id AA15348; Fri, 26 Mar 93 16:40:19 -0800 Date: Fri, 26 Mar 93 16:40:19 -0800 From: barszcz@nas.nasa.gov (Eric Barszcz) Message-Id: <9303270040.AA15348@wk46.nas.nasa.gov> To: mpi-comm@cs.utk.edu Subject: Multidisciplinary Position Paper %!PS-Adobe-2.0 %%Creator: dvips, version 5.4 (C) 1986-90 Radical Eye Software %%Pages: 10 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: tex.pro /TeXDict 200 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR}B /@letter{/vsize 10 N}B /@landscape{ /isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{/vsize 15.5531 N }B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0 ]N /df{/sf 1 N /fntrx FMat N df-tail}B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0 ]N df-tail}B /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder}N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[ }B /E{pop nn dup definefont setfont}B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /ch-width{ch-data dup length 5 sub get}B /ch-height{ch-data dup length 4 sub get}B /ch-xoff{128 ch-data dup length 3 sub get sub}B /ch-yoff{ch-data dup length 2 sub get 127 sub}B /ch-dx{ch-data dup length 1 sub get}B /ctr 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image}imagemask restore}B /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}B /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}B /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put}for}B /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 8 string N /v{/ruley X /rulex X V}B /V{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}B /a{moveto}B /delta 0 N /tail{ dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail}B /c{-4 M}B /d{ -3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t {p 4 w}B /w{0 rmoveto}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 34 122 df45 D<000FC000106000603800801800801C01001C02201E02101E04101E04101E04101E08203C0820 3C0840380840780880F00700E00001C000030000060000180000200000C0000100000200000400 100800301000202000605F80C063FFC040FF80807F00801E0017227CA019>50 D<000FC000386000703000E03001C0380380380780380700380F00380F00380F00381E00781E00 781E00781E00F81E00F01C00F00E01F00E02F00605E00309E001F1E00003C00003C00003800007 00000700600E00F00C00F01800E0300080600041C0003F000015227BA019>57 D<0000030000000300000007000000070000000F0000000F0000001F0000002F0000002F000000 4F0000004F80000087800000878000010780000207800002078000040780000407800008078000 080780001007800030078000200780007FFF80004007C0008007C0008003C0010003C0030003C0 020003C0040003C0040003C00C0003C03C0007C0FF003FFC1E237DA224>65 D<00007F00800003808100000E00630000380027000070001F0000E0000E0001C0000E00038000 0E000700000E000F000004000E000004001E000004003C000004003C0000080078000000007800 0000007800000000F000000000F000000000F000000000F000000000F000000000E000000000E0 00002000E000002000E000004000E000004000F000008000700000800070000100003800020000 18000400001C0008000006003000000381C0000000FE000000212479A223>67 D<00FFFFFF000F000F000F0003000F0003001E0003001E0003001E0002001E0002003C0002003C 0002003C0102003C010000780200007802000078060000780E0000FFFC0000F00C0000F00C0000 F00C0001E0080001E0080001E0080001E0000003C0000003C0000003C0000003C0000007800000 0780000007800000078000000F800000FFFC000020227DA120>70 D<00007F0080000380810000 0E00630000380027000070001F0000E0000E0001C0000E000380000E000700000E000F00000400 0E000004001E000004003C000004003C00000800780000000078000000007800000000F0000000 00F000000000F000000000F000000000F0003FFC00E00001E000E00001E000E00001E000E00003 C000E00003C000F00003C000700003C0007000078000380007800018000F80001C001380000600 2300000381C1000000FE000000212479A226>I<00FFF87FFC000F000780000F000780000F0007 80001E000F00001E000F00001E000F00001E000F00003C001E00003C001E00003C001E00003C00 1E000078003C000078003C000078003C000078003C0000FFFFF80000F000780000F000780000F0 00780001E000F00001E000F00001E000F00001E000F00003C001E00003C001E00003C001E00003 C001E000078003C000078003C000078003C000078003C0000F8007C000FFF87FFC0026227DA124 >I<00FFF8000F00000F00000F00001E00001E00001E00001E00003C00003C00003C00003C0000 780000780000780000780000F00000F00000F00000F00001E00001E00001E00001E00003C00003 C00003C00003C0000780000780000780000780000F8000FFF80015227DA113>I<0007FFC00000 3C0000003C0000003C00000078000000780000007800000078000000F0000000F0000000F00000 00F0000001E0000001E0000001E0000001E0000003C0000003C0000003C0000003C00000078000 000780000007800000078000000F0000000F0000380F0000780F0000F81E0000F81E0000F03C00 00403800004070000021E000001F8000001A237CA11A>I<00FF800007FC000F80000F80000F80 001780000F80001780001780002F000013C0002F000013C0004F000013C0008F000023C0009E00 0023C0011E000023C0011E000023C0021E000043C0043C000043C0043C000043C0083C000041E0 083C000081E01078000081E02078000081E02078000081E04078000101E040F0000101E080F000 0101E100F0000101E100F0000200F201E0000200F201E0000200F401E0000200F801E0000400F8 03C0000400F003C0000400F003C0000C00E003C0001E00C007C000FFC0C07FFC002E227DA12C> 77 D<00FFFFE0000F0038000F001E000F000E001E0007001E0007001E0007001E0007003C000F 003C000F003C000F003C001E0078001E0078003C00780078007800E000F003C000FFFE0000F000 0000F0000001E0000001E0000001E0000001E0000003C0000003C0000003C0000003C000000780 00000780000007800000078000000F800000FFF8000020227DA121>80 D<0001F020000E0C4000 1802C0003001C0006001C000E0018000C0018001C0018001C0018003C0010003C0010003C00000 03C0000003E0000001F8000001FF000000FFE000007FF000001FF8000003FC0000007C0000003C 0000001E0000001E0000001E0020001C0020001C0020001C002000180060003800600030007000 60007000C000C8018000C607000081FC00001B247DA21B>83 D<1FFFFFF81E03C0381803C01830 03C01820078018200780184007801040078010400F0010800F0010800F0010000F0000001E0000 001E0000001E0000001E0000003C0000003C0000003C0000003C00000078000000780000007800 000078000000F0000000F0000000F0000000F0000001E0000001E0000001E0000001E0000003E0 0000FFFF00001D2277A123>I<01020408102020404080B8FCFCF870080F76A20F>96 D<00F8C00185C00705C00E03800E03801C03803C0380380700780700780700780700F00E00F00E 00F00E00F00E10F01C20701C20703C20305C40308C400F078014157B9419>I<03C03F80038003 80038007000700070007000E000E000E000E001C001CF81D0C1E0E3C0638073807380F700F700F 700F700FE01EE01EE01EE03CE038E038607060E031C01F0010237BA216>I<007E0001C1000301 800703800E07801C07803C0000380000780000780000780000F00000F00000F00000F00000F001 00700100700200300C001830000FC00011157B9416>I<00003C0003F800003800003800003800 00700000700000700000700000E00000E00000E00000E00001C000F9C00185C00705C00E03800E 03801C03803C0380380700780700780700780700F00E00F00E00F00E00F00E10F01C20701C2070 3C20305C40308C400F078016237BA219>I<00F803840E021C023C0238027804F018FFE0F000F0 00E000E000E000E000E002E0026004701830600F800F157A9416>I<00003E0000470000CF0001 8F000186000380000380000380000700000700000700000700000700000E0000FFF0000E00000E 00000E00001C00001C00001C00001C00001C000038000038000038000038000038000070000070 0000700000700000700000E00000E00000E00000E00000C00001C00001C000718000F18000F300 006200003C0000182D82A20F>I<001F180030B800E0B801C07001C0700380700780700700E00F 00E00F00E00F00E01E01C01E01C01E01C01E01C01E03800E03800E0780060B8006170001E70000 0700000700000E00000E00000E00701C00F01800F0300060E0003F8000151F7E9416>I<00F000 0FE00000E00000E00000E00001C00001C00001C00001C000038000038000038000038000070000 071F0007218007C0C00F00E00F00E00E00E00E00E01C01C01C01C01C01C01C01C0380380380380 380380380704700708700E08700E10700610E006206003C016237DA219>I<00C001E001C001C0 000000000000000000000000000000001C002300430043008700870087000E000E001C001C001C 00380038003840708070807080710032001C000C217BA00F>I<01E01FC001C001C001C0038003 800380038007000700070007000E000E000E000E001C001C001C001C0038003800380038007000 700070007100E200E200E200E200640038000B237CA20C>108 D<1C0F80F8002610C10C004760 66060087807807008780780700870070070087007007000E00E00E000E00E00E000E00E00E000E 00E00E001C01C01C001C01C01C001C01C01C001C01C03820380380384038038070403803807080 380380308070070031003003001E0024157B9428>I<1C0F002631C04740C08780E08780E08700 E08700E00E01C00E01C00E01C00E01C01C03801C03801C03801C0704380708380E08380E103806 107006203003C017157B941B>I<007E0001C3000381800701C00E01C01C01E03C01E03801E078 01E07801E07801E0F003C0F003C0F00380F00780700700700E00700C0030180018700007C00013 157B9419>I<01C1F002621804741C08780C08700E08700E08701E00E01E00E01E00E01E00E01E 01C03C01C03C01C03C01C07803807003807003C0E003C1C0072380071E000700000700000E0000 0E00000E00000E00001C00001C00001C0000FFC000171F7F9419>I<1C1F002620804741C08783 C08703C08701808700000E00000E00000E00000E00001C00001C00001C00001C00003800003800 0038000038000070000030000013157B9415>114 D<00FC000183000200800401800C03800C03 000C00000F00000FF00007FC0003FE00003E00000F00000700700700F00600F00600E004004008 002030001FC00011157D9414>I<00C001C001C001C001C003800380038003800700FFF8070007 000E000E000E000E001C001C001C001C003800380038003810702070207040708031001E000D1F 7C9E10>I<1E00602300E04380E04381C08381C08701C08701C00703800E03800E03800E03801C 07001C07001C07001C07081C0E10180E101C0E101C1E200C262007C3C016157B941A>I<1E0030 2300704380704380E08380E08700E08700E00701C00E01C00E01C00E01C01C03801C03801C0380 1C03801C07001C07001C07001C0F000C3E0003CE00000E00000E00001C00601C00F03800F03000 E0600080C0004380003E0000151F7B9418>121 D E /Fb 1 81 df80 D E /Fc 3 21 df0 D2 D<0000000C0000003C000000F0000003C000000F0000003C000000F0000007C000001F0000 0078000001E00000078000001E00000078000000E0000000780000001E0000000780000001E000 0000780000001F00000007C0000000F00000003C0000000F00000003C0000000F00000003C0000 000C0000000000000000000000000000000000000000000000000000000000000000FFFFFFFCFF FFFFFC1E277C9F27>20 D E /Fd 3 52 df<07C018303018701C600C600CE00EE00EE00EE00EE0 0EE00EE00EE00EE00E600C600C701C30181C7007C00F157F9412>48 D<0F8030E040708030C038 E0384038003800700070006000C00180030006000C08080810183FF07FF0FFF00D157E9412>50 D<0FE030306018701C701C001C00180038006007E000300018000C000E000EE00EE00EC00C4018 30300FE00F157F9412>I E /Fe 3 106 df<07F0000FE000F0001E0000B8001E0000B8002E0000 B8004E000138005C000138009C000138011C00011C011C00021C023800021C043800021C043800 021C083800041C107000040E107000040E207000040E407000080E40E000080E80E000080F00E0 00080700E000180601C000FE040FF80023177F9622>77 D<07F007F800F000C000B8008000B800 80009C0080011C0100011E0100010E0100010E0100020702000207020002038200020382000401 C4000401C4000400E4000400E4000800780008007800080038000800380018001000FE0010001D 177F961C>I<0300038003000000000000000000000000001C002400460046008C000C00180018 00180031003100320032001C0009177F960C>105 D E /Ff 9 84 df<0F00187F00380F00380E 00700E00700E00700E00E01C00E01C01C01C01801C0380380700380600380C0038180070300070 600070C000730000FC0000F0000015157D9418>23 D<003F0001FFC00381E00400400800001000 001000001000001060000B98000FF800100000200000400000400000400000800000C000804000 80600100380E001FFC0007E00013177F9517>34 D<70F8F8F87005057C840E>58 D<70F8FCFC7404040404080810102040060F7C840E>I<00007F00400003C0C080000E00218000 1C0013800070000F8000E000070001C0000700038000070007000007000F000002000E00000200 1E000002003C000002003C00000400780000000078000000007800000000F000000000F0000000 00F000000000F000000000F0003FFF00E00000F000E00000F000E00000F000E00001E000F00001 E000F00001E000700001E000700003C000380003C000180007C0000C0009C00006001180000380 E08000007F00000022247DA226>71 D<007FC00001FF0007C00003E00007C00005E00007C00005 E00009E0000BC00009E0000BC00009E00013C00009E00023C00011E00027800011E00047800011 E00047800011E00087800021E0010F000020F0010F000020F0020F000020F0040F000040F0041E 000040F0081E000040F0081E000040F0101E000080F0203C00008078203C00008078403C000080 78803C0001007880780001007900780001007900780001007A00780002007C00F00002007C00F0 0002003800F00006003800F0000F003001F000FFE0203FFF0030227EA12F>77 D<007FC003FF0007C000780007C000600005E000200009E000400009E000400008F000400008F0 00400010F800800010780080001078008000103C008000203C010000203E010000201E01000020 1E010000400F020000400F020000400F020000400782000080078400008007C400008003C40000 8003C400010001E800010001E800010001F800010000F800020000F00002000070000200007000 06000070000F00002000FFE000200028227EA127>I<007FFFF0000007801C000007800F000007 800700000F000380000F000380000F000380000F000380001E000780001E000780001E00078000 1E000F00003C000F00003C001E00003C003C00003C007000007801E000007FFF00000078000000 007800000000F000000000F000000000F000000000F000000001E000000001E000000001E00000 0001E000000003C000000003C000000003C000000003C000000007C0000000FFFC00000021227E A11F>80 D<0003F010000E0C2000180260002001E0004000E000C000C0008000C0018000C00180 00C00380008003800080038000000380000003C0000001F0000001FE000000FFE000007FF00000 1FF8000001FC0000003C0000001C0000000E0000000E0000000E0020000C0020000C0020000C00 2000080060001800600010007000200070004000C8008000C603000081FC00001C247DA21E>83 D E /Fg 31 122 df45 D<60F0F06004047C830C>I<030007003F00C7 000700070007000700070007000700070007000700070007000700070007000700070007000700 0700070007000F80FFF80D1C7C9B15>49 D<07C01830201C400C400EF00FF80FF807F807700700 0F000E000E001C001C00380070006000C00180030006010C01180110023FFE7FFEFFFE101C7E9B 15>I<00F0030C06040C0E181E301E300C700070006000E3E0E430E818F00CF00EE006E007E007 E007E007E007600760077006300E300C18180C3003E0101D7E9B15>54 D<03C00C301818300C70 0C600EE006E006E007E007E007E007E0076007700F300F18170C2707C700060006000E300C780C 78187010203030C00F80101D7E9B15>57 D<000600000006000000060000000F0000000F000000 0F00000017800000178000001780000023C0000023C0000023C0000041E0000041E0000041E000 0080F0000080F0000180F8000100780001FFF80003007C0002003C0002003C0006003E0004001E 0004001E000C001F001E001F00FF80FFF01C1D7F9C1F>65 D<001F808000E06180018019800700 07800E0003801C0003801C00018038000180780000807800008070000080F0000000F0000000F0 000000F0000000F0000000F0000000F0000000F000000070000080780000807800008038000080 1C0001001C0001000E000200070004000180080000E03000001FC000191E7E9C1E>67 D78 D<07E0801C1980300580700380600180E00180E00080E00080E00080F00000F800007C00 007FC0003FF8001FFE0007FF0000FF80000F800007C00003C00001C08001C08001C08001C0C001 80C00180E00300D00200CC0C0083F800121E7E9C17>83 D<7FFFFFC0700F01C0600F00C0400F00 40400F0040C00F0020800F0020800F0020800F0020000F0000000F0000000F0000000F0000000F 0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F000000 0F0000000F0000000F0000001F800003FFFC001B1C7F9B1E>I<1FC000307000783800781C0030 1C00001C00001C0001FC000F1C00381C00701C00601C00E01C40E01C40E01C40603C40304E801F 870012127E9115>97 D<07E00C301878307870306000E000E000E000E000E000E0006000700430 0418080C3007C00E127E9112>99 D<003F00000700000700000700000700000700000700000700 00070000070000070003E7000C1700180F00300700700700600700E00700E00700E00700E00700 E00700E00700600700700700300700180F000C370007C7E0131D7E9C17>I<03E00C301818300C 700E6006E006FFFEE000E000E000E00060007002300218040C1803E00F127F9112>I<00F8018C 071E061E0E0C0E000E000E000E000E000E00FFE00E000E000E000E000E000E000E000E000E000E 000E000E000E000E000E000E007FE00F1D809C0D>I<00038003C4C00C38C01C3880181800381C 00381C00381C00381C001818001C38000C300013C0001000003000001800001FF8001FFF001FFF 803003806001C0C000C0C000C0C000C06001803003001C0E0007F800121C7F9215>II<18003C003C0018000000000000000000000000000000FC001C001C 001C001C001C001C001C001C001C001C001C001C001C001C001C001C00FF80091D7F9C0C>I107 DIII<03F0000E1C00180600300300700380600180E001C0E001C0E001C0E001C0E001C0 E001C06001807003803003001806000E1C0003F00012127F9115>II 114 D<1F9030704030C010C010E010F8007F803FE00FF000F880388018C018C018E010D0608FC0 0D127F9110>I<04000400040004000C000C001C003C00FFE01C001C001C001C001C001C001C00 1C001C001C101C101C101C101C100C100E2003C00C1A7F9910>II119 D121 D E /Fh 2 122 df<040004000400C460E4E0 3F800E003F80E4E0C4600400040004000B0D7E8D11>3 D<0C000C000C000C000C000C00FFC0FF C00C000C000C000C000C000C000C000C000C000C000C000C000C000C000C000C000C000C000A1A 7E9310>121 D E /Fi 41 122 df<0007F80FF000007FFE7FFC0001F80FF80E0003E00FE01F00 07C01FC03F000F801F803F000F801F803F000F800F801E000F800F800C000F800F8000000F800F 8000000F800F8000000F800F800000FFFFFFFFFF00FFFFFFFFFF000F800F801F000F800F801F00 0F800F801F000F800F801F000F800F801F000F800F801F000F800F801F000F800F801F000F800F 801F000F800F801F000F800F801F000F800F801F000F800F801F000F800F801F000F800F801F00 0F800F801F000F800F801F000F800F801F007FF07FF0FFE07FF07FF0FFE02B237FA22F>14 D<387CFEFFFF7F3B03030706060C1C18702008117CA210>39 D<387CFEFEFE7C3807077C8610> 46 D<00180000780001F800FFF800FFF80001F80001F80001F80001F80001F80001F80001F800 01F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F800 01F80001F80001F80001F80001F8007FFFE07FFFE013207C9F1C>49 D<03FC000FFF003C1FC070 07E07C07F0FE03F0FE03F8FE03F8FE01F87C01F83803F80003F80003F00003F00007E00007C000 0F80001F00003E0000380000700000E01801C0180380180700180E00380FFFF01FFFF03FFFF07F FFF0FFFFF0FFFFF015207D9F1C>I<00FE0007FFC00F07E01E03F03F03F03F81F83F81F83F81F8 1F03F81F03F00003F00003E00007C0001F8001FE0001FF000007C00001F00001F80000FC0000FC 3C00FE7E00FEFF00FEFF00FEFF00FEFF00FC7E01FC7801F81E07F00FFFC001FE0017207E9F1C> I<0000E00001E00003E00003E00007E0000FE0001FE0001FE00037E00077E000E7E001C7E00187 E00307E00707E00E07E00C07E01807E03807E07007E0E007E0FFFFFEFFFFFE0007E00007E00007 E00007E00007E00007E00007E000FFFE00FFFE17207E9F1C>I<1000201E01E01FFFC01FFF801F FF001FFE001FF8001BC00018000018000018000018000019FC001FFF001E0FC01807E01803E000 03F00003F00003F80003F83803F87C03F8FE03F8FE03F8FC03F0FC03F07007E03007C01C1F800F FF0003F80015207D9F1C>I<001F8000FFE003F07007C0F00F01F81F01F83E01F83E01F87E00F0 7C00007C0000FC0800FC7FC0FCFFE0FD80F0FF00F8FE007CFE007CFC007EFC007EFC007EFC007E 7C007E7C007E7C007E3C007C3E007C1E00F80F00F00783E003FFC000FF0017207E9F1C>I<0000 70000000007000000000F800000000F800000000F800000001FC00000001FC00000003FE000000 03FE00000003FE00000006FF000000067F0000000E7F8000000C3F8000000C3F800000183FC000 00181FC00000381FE00000300FE00000300FE00000600FF000006007F00000E007F80000FFFFF8 0000FFFFF800018001FC00018001FC00038001FE00030000FE00030000FE000600007F00060000 7F00FFE00FFFF8FFE00FFFF825227EA12A>65 D<0003FE0080001FFF818000FF01E38001F8003F 8003E0001F8007C0000F800F800007801F800007803F000003803F000003807F000001807E0000 01807E00000180FE00000000FE00000000FE00000000FE00000000FE00000000FE00000000FE00 000000FE000000007E000000007E000001807F000001803F000001803F000003801F800003000F 8000030007C000060003F0000C0001F800380000FF00F000001FFFC0000003FE000021227DA128 >67 DII72 DI76 DI80 D82 D<01FC0407FF8C1F03FC3C007C7C003C78001C78001CF8000CF8000CFC000CFC00 00FF0000FFE0007FFF007FFFC03FFFF01FFFF80FFFFC03FFFE003FFE0003FF00007F00003F0000 3FC0001FC0001FC0001FE0001EE0001EF0003CFC003CFF00F8C7FFE080FF8018227DA11F>I<07 FC001FFF803F07C03F03E03F01E03F01F01E01F00001F00001F0003FF003FDF01FC1F03F01F07E 01F0FC01F0FC01F0FC01F0FC01F07E02F07E0CF81FF87F07E03F18167E951B>97 DI<00FF8007FFE00F83 F01F03F03E03F07E03F07C01E07C0000FC0000FC0000FC0000FC0000FC0000FC00007C00007E00 007E00003E00301F00600FC0E007FF8000FE0014167E9519>I<0001FE000001FE0000003E0000 003E0000003E0000003E0000003E0000003E0000003E0000003E0000003E0000003E0000003E00 01FC3E0007FFBE000F81FE001F007E003E003E007E003E007C003E00FC003E00FC003E00FC003E 00FC003E00FC003E00FC003E00FC003E00FC003E007C003E007C003E003E007E001E00FE000F83 BE0007FF3FC001FC3FC01A237EA21F>I<00FE0007FF800F87C01E01E03E01F07C00F07C00F8FC 00F8FC00F8FFFFF8FFFFF8FC0000FC0000FC00007C00007C00007E00003E00181F00300FC07003 FFC000FF0015167E951A>I<003F8000FFC001E3E003C7E007C7E00F87E00F83C00F80000F8000 0F80000F80000F80000F8000FFFC00FFFC000F80000F80000F80000F80000F80000F80000F8000 0F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80007FF8007FF800 13237FA211>I<03FC1E0FFF7F1F0F8F3E07CF3C03C07C03E07C03E07C03E07C03E07C03E03C03 C03E07C01F0F801FFF0013FC003000003000003800003FFF801FFFF00FFFF81FFFFC3800FC7000 3EF0001EF0001EF0001EF0001E78003C7C007C3F01F80FFFE001FF0018217E951C>II<1C003F007F007F007F003F001C 000000000000000000000000000000FF00FF001F001F001F001F001F001F001F001F001F001F00 1F001F001F001F001F001F001F001F00FFE0FFE00B247EA310>I108 DII<00FE0007FFC00F83E01E00 F03E00F87C007C7C007C7C007CFC007EFC007EFC007EFC007EFC007EFC007EFC007E7C007C7C00 7C3E00F81F01F00F83E007FFC000FE0017167E951C>II<00FE030007FF87000FC1C7001F006F003F003F007E003F007E001F007C001F00FC001F00FC 001F00FC001F00FC001F00FC001F00FC001F00FC001F007E001F007E001F003E003F001F007F00 0FC1DF0007FF9F0001FC1F0000001F0000001F0000001F0000001F0000001F0000001F0000001F 0000001F000000FFE00000FFE01B207E951E>II<0FF3003FFF00781F00600700E00300E00300F00300FC0000 7FE0007FF8003FFE000FFF0001FF00000F80C00780C00380E00380E00380F00700FC0E00EFFC00 C7F00011167E9516>I<0180000180000180000180000380000380000780000780000F80003F80 00FFFF00FFFF000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80 000F81800F81800F81800F81800F81800F830007C30003FE0000F80011207F9F16>II119 D121 D E /Fj 2 122 df<020002000200C218F2783AE00F800F803AE0F278C2180200020002000D0E 7E8E12>3 D<06000600060006000600060006000600FFF0FFF006000600060006000600060006 000600060006000600060006000600060006000600060006000C1D7E9611>121 D E /Fk 83 124 df<001F83E000706E3000C07C780180F8780380F07807007000070070000700 7000070070000700700007007000070070000700700007007000FFFFFFC0070070000700700007 007000070070000700700007007000070070000700700007007000070070000700700007007000 070070000700700007007000070070000700700007007000070078007FE3FF801D2380A21C>11 D<001FC0000070200000C010000180380003807800070078000700300007000000070000000700 000007000000070000000700000007000000FFFFF8000700780007003800070038000700380007 003800070038000700380007003800070038000700380007003800070038000700380007003800 07003800070038000700380007003800070038007FE1FF80192380A21B>I<001FD80000703800 00C078000180780003807800070038000700380007003800070038000700380007003800070038 000700380007003800FFFFF8000700380007003800070038000700380007003800070038000700 380007003800070038000700380007003800070038000700380007003800070038000700380007 00380007003800070038007FF3FF80192380A21B>I<000FC07F00007031C08000E00B00400180 1E00E003803E01E007003C01E007001C00C007001C000007001C000007001C000007001C000007 001C000007001C000007001C0000FFFFFFFFE007001C01E007001C00E007001C00E007001C00E0 07001C00E007001C00E007001C00E007001C00E007001C00E007001C00E007001C00E007001C00 E007001C00E007001C00E007001C00E007001C00E007001C00E007001C00E007001C00E07FF1FF CFFE272380A229>I<7038F87CFC7EFC7E743A0402040204020402080408041008100820104020 0F0F7EA218>34 D<0780000C001840001C0018200018003010007800701C00F0006013FF6000E0 0800E000E00800C000E00801C000E008038000E008030000E008070000E0080E0000E0080C0000 60101C000070101800003010380000182070000018406000000780E03C000001C04200000180C1 00000381810000070380800006030080000E030040000C070040001C0700400038070040003007 0040007007004000E007004000C007004001C00300400180030080038003808007000181000600 00C1000E000042000C00003C0022287DA429>37 D<70F8FCFC7404040404080810102040060F7C A20E>39 D<00200040008001000300060004000C000C0018001800300030003000700060006000 6000E000E000E000E000E000E000E000E000E000E000E000E000E000E000600060006000700030 0030003000180018000C000C0004000600030001000080004000200B327CA413>I<8000400020 00100018000C000400060006000300030001800180018001C000C000C000C000E000E000E000E0 00E000E000E000E000E000E000E000E000E000E000C000C000C001C00180018001800300030006 00060004000C00180010002000400080000B327DA413>I<000180000001800000018000000180 000001800000018000000180000001800000018000000180000001800000018000000180000001 80000001800000018000FFFFFFFEFFFFFFFE000180000001800000018000000180000001800000 018000000180000001800000018000000180000001800000018000000180000001800000018000 000180001F227D9C26>43 D<70F8FCFC7404040404080810102040060F7C840E>II<70F8F8F87005057C840E>I<00018000018000038000030000030000070000060000 0600000E00000C00000C00001C0000180000180000180000380000300000300000700000600000 600000E00000C00000C00001C0000180000180000380000300000300000700000600000600000E 00000C00000C00000C00001C0000180000180000380000300000300000700000600000600000E0 0000C00000C0000011317DA418>I<01F000071C000C06001803003803803803807001C07001C0 7001C07001C0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001E0 F001E0F001E0F001E07001C07001C07001C07803C03803803803801C07000C0600071C0001F000 13227EA018>I<008003800F80F380038003800380038003800380038003800380038003800380 03800380038003800380038003800380038003800380038003800380038007C0FFFE0F217CA018 >I<03F0000C1C001007002007804003C04003C08003E0F003E0F801E0F801E0F801E02003E000 03E00003C00003C0000780000700000E00001C0000180000300000600000C00001800001000002 00200400200800201800603000403FFFC07FFFC0FFFFC013217EA018>I<03F8000C1E00100700 2007804007C07807C07803C07807C03807C0000780000780000700000F00000E0000380003F000 001C00000F000007800007800003C00003C00003E02003E07003E0F803E0F803E0F003C04003C0 400780200780100F000C1C0003F00013227EA018>I<000200000600000E00000E00001E00001E 00002E00004E00004E00008E00008E00010E00020E00020E00040E00040E00080E00100E00100E 00200E00200E00400E00800E00FFFFF8000E00000E00000E00000E00000E00000E00000E00001F 0001FFF015217FA018>I<1000801E07001FFF001FFE001FF80013E00010000010000010000010 000010000010000010F800130E001407001803801003800001C00001C00001E00001E00001E000 01E07001E0F001E0F001E0E001C08001C04003C04003802007001006000C1C0003F00013227EA0 18>I<007E0001C1000300800601C00E03C01C03C0180180380000380000780000700000700000 F0F800F30C00F40600F40300F80380F801C0F001C0F001E0F001E0F001E0F001E0F001E07001E0 7001E07001E03801C03801C01803801C03000C0600070C0001F00013227EA018>I<4000006000 007FFFE07FFFC07FFFC0400080C001008001008002008002000004000008000008000010000030 0000200000600000600000600000E00000C00000C00001C00001C00001C00001C00003C00003C0 0003C00003C00003C00003C00003C00003C00001800013237DA118>I<01F800060E0008030010 01802001802000C06000C06000C06000C07000C07801803E01003F02001FC4000FF80003F80003 FC00067F00083F80100F803007C06001C06000E0C000E0C00060C00060C00060C0006060004060 00C03000801803000E0E0003F00013227EA018>I<01F000060C000C0600180700380380700380 700380F001C0F001C0F001C0F001E0F001E0F001E0F001E0F001E07001E07003E03803E01805E0 0C05E00619E003E1E00001C00001C00001C0000380000380300300780700780600700C00201800 1030000FC00013227EA018>I<70F8F8F870000000000000000000000070F8F8F87005157C940E> I61 D<07E01838201C400E800FF00FF00FF00F000F000E00 1C00380030006000C000C000800080018001000100010001000100010000000000000000000000 038007C007C007C0038010237DA217>63 D<0001800000018000000180000003C0000003C00000 03C0000005E0000005E000000DF0000008F0000008F0000010F800001078000010780000203C00 00203C0000203C0000401E0000401E0000401E0000800F0000800F0000FFFF0001000780010007 80030007C0020003C0020003C0040003E0040001E0040001E00C0000F00C0000F03E0001F8FF80 0FFF20237EA225>65 DI<0007E0 100038183000E0063001C00170038000F0070000F00E0000701E0000701C0000303C0000303C00 00307C0000107800001078000010F8000000F8000000F8000000F8000000F8000000F8000000F8 000000F800000078000000780000107C0000103C0000103C0000101C0000201E0000200E000040 070000400380008001C0010000E0020000381C000007E0001C247DA223>IIII<0007F008003C0C18 00E0021801C001B8038000F8070000780F0000381E0000381E0000183C0000183C0000187C0000 087800000878000008F8000000F8000000F8000000F8000000F8000000F8000000F8000000F800 1FFF780000F8780000787C0000783C0000783C0000781E0000781E0000780F0000780700007803 8000B801C000B800E00318003C0C080007F00020247DA226>III<03FFF0001F00000F00000F00000F00000F00000F00000F00000F00000F 00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F 00000F00000F00000F00700F00F80F00F80F00F80E00F01E00401C0020380018700007C0001423 7EA119>IIIII<000FE00000783C0000E00E0003C007800780 03C00F0001E00E0000E01E0000F03C0000783C0000787C00007C7C00007C7800003C7800003CF8 00003EF800003EF800003EF800003EF800003EF800003EF800003EF800003EF800003E7800003C 7C00007C7C00007C3C0000783E0000F81E0000F00F0001E00F0001E0078003C003C0078000E00E 0000783C00000FE0001F247DA226>II<000FE00000783C0000E00E0003C00780078003C00F0001E00E0000E01E0000F03E0000F8 3C0000787C00007C7C00007C7800003C7800003CF800003EF800003EF800003EF800003EF80000 3EF800003EF800003EF800003EF800003E7800003C7C00007C7C00007C3C0000783C0000781E03 80F00E0420E00F0801E0078813C003C8178000E80E00007C3C02000FEC0200000C0200000C0200 000E0600000F0E000007FC000007FC000007F8000003F8000001E01F2D7DA226>II<03F0200C0C601802603001E07000E0600060 E00060E00060E00020E00020E00020F00000F000007800007F00003FF0001FFE000FFF0003FF80 003FC00007E00001E00000F00000F0000070800070800070800070800070C00060C00060E000C0 F000C0C80180C6070081FC0014247DA21B>I<7FFFFFF878078078600780184007800840078008 40078008C007800C80078004800780048007800480078004000780000007800000078000000780 000007800000078000000780000007800000078000000780000007800000078000000780000007 800000078000000780000007800000078000000780000007800000078000000FC00003FFFF001E 227EA123>IIII<7FF807FF0007E001F80003C000E00003E000C00001E000800000F001000000F803 00000078020000007C040000003E0C0000001E080000001F100000000FB000000007A000000007 C000000003E000000001E000000001F000000003F80000000278000000047C0000000C3E000000 081E000000101F000000300F80000020078000004007C00000C003E000008001E000010001F000 030000F000070000F8001F8001FC00FFE007FFC022227FA125>II91 D<080410082010201040204020 8040804080408040B85CFC7EFC7E7C3E381C0F0F7AA218>II<08102020404080808080B8FCFC7C38060F7DA20E>96 D<0FE0001838003C0C003C0E0018 070000070000070000070000FF0007C7001E07003C0700780700700700F00708F00708F00708F0 0F087817083C23900FC1E015157E9418>I<0E0000FE00001E00000E00000E00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E1F000E61C00E80600F00300E00380E003C 0E001C0E001E0E001E0E001E0E001E0E001E0E001E0E001E0E001C0E003C0E00380F00700C8060 0C41C0083F0017237FA21B>I<01FE000703000C07801C0780380300780000700000F00000F000 00F00000F00000F00000F00000F000007000007800403800401C00800C010007060001F8001215 7E9416>I<0000E0000FE00001E00000E00000E00000E00000E00000E00000E00000E00000E000 00E00000E00000E001F8E00704E00C02E01C01E03800E07800E07000E0F000E0F000E0F000E0F0 00E0F000E0F000E0F000E07000E07800E03800E01801E00C02E0070CF001F0FE17237EA21B>I< 01FC000707000C03801C01C03801C07801E07000E0F000E0FFFFE0F00000F00000F00000F00000 F000007000007800203800201C00400E008007030000FC0013157F9416>I<003C00C6018F038F 030F070007000700070007000700070007000700FFF80700070007000700070007000700070007 0007000700070007000700070007000700070007807FF8102380A20F>I<00007001F198071E18 0E0E181C07001C07003C07803C07803C07803C07801C07001C07000E0E000F1C0019F000100000 1000001800001800001FFE000FFFC00FFFE03800F0600030400018C00018C00018C00018600030 6000303800E00E038003FE0015217F9518>I<0E0000FE00001E00000E00000E00000E00000E00 000E00000E00000E00000E00000E00000E00000E00000E1F800E60C00E80E00F00700F00700E00 700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00 700E0070FFE7FF18237FA21B>I<1C001E003E001E001C00000000000000000000000000000000 000E00FE001E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00 0E00FFC00A227FA10E>I<01C003E003E003E001C00000000000000000000000000000000001E0 0FE001E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000 E000E000E000E000E000E060E0F0C0F18061803E000B2C82A10F>I<0E0000FE00001E00000E00 000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E03FC0E01F00E01 C00E01800E02000E04000E08000E10000E38000EF8000F1C000E1E000E0E000E07000E07800E03 C00E01C00E01E00E00F00E00F8FFE3FE17237FA21A>I<0E00FE001E000E000E000E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E 000E000E000E000E000E000E00FFE00B237FA20E>I<0E1FC07F00FE60E183801E807201C00F00 3C00E00F003C00E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E 003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E0 0E003800E0FFE3FF8FFE27157F942A>I<0E1F80FE60C01E80E00F00700F00700E00700E00700E 00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E0070FF E7FF18157F941B>I<01FC000707000C01801800C03800E0700070700070F00078F00078F00078 F00078F00078F00078F000787000707800F03800E01C01C00E038007070001FC0015157F9418> I<0E1F00FE61C00E80600F00700E00380E003C0E001C0E001E0E001E0E001E0E001E0E001E0E00 1E0E001E0E003C0E003C0E00380F00700E80E00E41C00E3F000E00000E00000E00000E00000E00 000E00000E00000E00000E0000FFE000171F7F941B>I<01F8200704600E02601C01603801E078 00E07800E0F000E0F000E0F000E0F000E0F000E0F000E0F000E07000E07800E03801E01C01E00C 02E0070CE001F0E00000E00000E00000E00000E00000E00000E00000E00000E00000E0000FFE17 1F7E941A>I<0E3CFE461E8F0F0F0F060F000E000E000E000E000E000E000E000E000E000E000E 000E000E000F00FFF010157F9413>I<0F8830786018C018C008C008E008F0007F803FE00FF001 F8003C801C800C800CC00CC008E018D0308FC00E157E9413>I<02000200020002000600060006 000E001E003E00FFF80E000E000E000E000E000E000E000E000E000E000E000E040E040E040E04 0E040E040708030801F00E1F7F9E13>I<0E0070FE07F01E00F00E00700E00700E00700E00700E 00700E00700E00700E00700E00700E00700E00700E00700E00700E00F00E00F006017003827800 FC7F18157F941B>I IIII<3FFFC03803803007802007 00600E00401C00403C0040380000700000E00001E00001C0000380400700400F00400E00C01C00 80380080780180700780FFFF8012157F9416>II E /Fl 23 123 df<000003800000000007C00000000007C0000000000FE0000000000FE000000000 0FE0000000001FF0000000001FF0000000003FF8000000003FF8000000003FF80000000073FC00 00000073FC00000000F3FE00000000E1FE00000000E1FE00000001C0FF00000001C0FF00000003 C0FF80000003807F80000007807FC0000007003FC0000007003FC000000E003FE000000E001FE0 00001E001FF000001C000FF000001FFFFFF000003FFFFFF800003FFFFFF80000780007FC000070 0003FC0000700003FC0000E00001FE0000E00001FE0001E00001FF0001C00000FF0001C00000FF 00FFFE001FFFFEFFFE001FFFFEFFFE001FFFFE2F297EA834>65 D69 D77 D80 D<01FF800007FFF0000F81FC001FC0FE001FC07F001FC07F001FC03F800F80 3F8000003F8000003F8000003F80000FFF8000FFFF8007FC3F801FE03F803F803F807F803F807F 003F80FE003F80FE003F80FE003F80FE007F80FF007F807F00FFC03F83DFFC0FFF0FFC01FC03FC 1E1B7E9A21>97 D<001FF80000FFFE0003F01F000FE03F801FC03F803F803F803F803F807F801F 007F000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF00 00007F0000007F8000003F8001C03FC001C01FC003C00FE0078003F01F0000FFFC00001FE0001A 1B7E9A1F>99 D<00003FF80000003FF80000003FF800000003F800000003F800000003F8000000 03F800000003F800000003F800000003F800000003F800000003F800000003F800000003F80000 0003F800001FE3F80000FFFBF80003F03FF8000FE00FF8001FC007F8003F8003F8003F8003F800 7F8003F8007F0003F800FF0003F800FF0003F800FF0003F800FF0003F800FF0003F800FF0003F8 00FF0003F800FF0003F800FF0003F8007F0003F8007F0003F8003F8003F8003F8007F8001FC00F F8000FE01FF80003F03FFF8000FFF3FF80003FC3FF80212A7EA926>I<003FE00001FFF80003F0 7E000FE03F001FC01F803F800FC03F800FC07F000FC07F0007E0FF0007E0FF0007E0FF0007E0FF FFFFE0FFFFFFE0FF000000FF000000FF000000FF0000007F0000007F8000003F8000E03F8001E0 1FC001C00FE003C003F81F8000FFFE00001FF0001B1B7E9A20>I<0007F0003FFC00FE3E01FC7F 03F87F03F87F07F07F07F03E07F00007F00007F00007F00007F00007F00007F000FFFFC0FFFFC0 FFFFC007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F000 07F00007F00007F00007F00007F00007F00007F00007F00007F0007FFF807FFF807FFF80182A7E A915>I<00FF81F003FFE7FC0FC1FE7C1F80FC7C3F80FE7C3F007E107F007F007F007F007F007F 007F007F007F007F007F007F003F007E003F80FE001F80FC000FC1F8001FFFE00018FF80003800 00003C0000003C0000003E0000003FFFF8003FFFFF001FFFFFC00FFFFFE007FFFFF01FFFFFF07E 0007F87C0001F8F80001F8F80000F8F80000F8F80000F8FC0001F87E0003F03F0007E00FC01F80 03FFFE00007FF0001E287E9A22>I<07001FC01FE03FE03FE03FE01FE01FC00700000000000000 0000000000000000FFE0FFE0FFE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00F E00FE00FE00FE00FE00FE00FE00FE00FE0FFFEFFFEFFFE0F2B7DAA14>105 D108 DII<003FE00001FFFC0003F07E000FC01F801F800FC03F800FE03F0007E07F0007F07F0007F0 7F0007F0FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F87F0007 F07F0007F03F800FE03F800FE01F800FC00FC01F8007F07F0001FFFC00003FE0001D1B7E9A22> II114 D<03FE300FFFF03E03F07800F07000F0F00070F0 0070F80070FC0000FFE000FFFE007FFFC03FFFE01FFFF007FFF800FFFC0003FC0000FCE0007CE0 003CF0003CF0003CF80078FC0078FF01F0F7FFC0C1FF00161B7E9A1B>I<007000007000007000 00700000F00000F00000F00001F00003F00003F00007F0001FFFF0FFFFF0FFFFF007F00007F000 07F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F03807F038 07F03807F03807F03807F03807F03803F87001F8F000FFE0001F8015267FA51B>II120 DI<3FFFFF803FFFFF803F00FF803C00FF003801FE007803FC00 7807FC007007F800700FF000701FE000001FE000003FC000007F800000FF800000FF000001FE03 8003FC038003FC038007F803800FF007801FF007801FE007003FC00F007F801F00FF807F00FFFF FF00FFFFFF00191B7E9A1F>I E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin @letter /letter where {pop letter} if %%EndSetup %%Page: 1 1 bop 432 658 a Fl(A)23 b(Mo)r(del)f(for)i(Executing)e(Multidisci)o(pl)o(i)o (nary)663 732 y(and)i(Multizonal)d(Programs)391 841 y Fk(Eric)16 b(Barszcz)658 823 y Fj(\003)792 841 y Fk(Sisira)g(K.)g(W)l(eeratunga)1247 823 y Fj(y)1383 841 y Fk(Eddy)g(Pramono)1705 823 y Fj(y)719 960 y Fk(NAS)g(Applied)f(Researc)o(h)g(Branc)o(h)731 1018 y(NASA)g(Ames)f (Researc)o(h)i(Cen)o(ter)788 1076 y(Mo\013ett)g(Field,)f(CA)h(94035)884 1240 y(Marc)o(h)g(26,)h(1993)941 1647 y Fi(Abstract)345 1733 y Fk(This)10 b(pap)q(er)i(presen)o(ts)e(the)h(mo)q(del)e(of)j(computing)d (for)i(m)o(ultidiscipli)o(nary)d(and/or)271 1793 y(m)o(ultizonal)24 b(applications)i(on)h(parallel)e(mac)o(hines)f(b)q(eing)i(pursued)h(at)f (NASA)271 1853 y(Ames)15 b(Researc)o(h)h(Cen)o(ter.)21 b(The)16 b(mo)q(del)f(describ)q(es)h(execution)f(on)i(a)g(parallel)f(ma-)271 1914 y(c)o(hine)c(with)h(some)f(n)o(um)o(b)q(er)g(of)h(pro)q(cessors,)i(eac)o (h)d(with)h(lo)q(cal)g(memory)l(,)e(connected)271 1974 y(b)o(y)20 b(a)g(net)o(w)o(ork.)31 b(It)20 b(do)q(es)g(not)g(discuss)g(pro)q(cessor)h (rates,)g(size)e(of)h(memory)l(,)d(net-)271 2034 y(w)o(ork)h(latency)l(,)g (net)o(w)o(ork)f(bandwidth,)i(net)o(w)o(ork)e(connectivit)o(y)l(,)f(disk)i (space,)g(I/O)271 2094 y(bandwidth)k(or)f(mass)f(storage.)36 b(It)20 b(also)h(do)q(es)h(not)f(discuss)g(language)g(and)h(dis-)271 2154 y(cipline)17 b(sp)q(eci\014c)g(algorithm)g(issues.)27 b(Mo)q(del)18 b(rationalit)o(y)f(and)i(functionalit)o(y)e(are)271 2214 y(presen)o(ted.)p 149 2594 719 2 v 205 2625 a Fh(\003)224 2640 y Fg(The)d(author)g(is)g(an)g(emplo)o(y)o(ee)e(of)i(NASA.)207 2675 y Fh(y)224 2690 y Fg(The)k(author)f(is)g(an)g(emplo)o(y)o(ee)f(of)h (Computer)f(Sciences)j(Co.)28 b(This)17 b(w)o(ork)g(w)o(as)g(funded)g (through)g(NASA)149 2740 y(con)o(tract)e(NAS)f(2-12961)p eop %%Page: 2 2 bop 149 42 a Fi(1)56 b(In)n(tro)r(duction)149 151 y Fk(The)18 b(problem)e(domain)g(of)i(particular)f(in)o(terest)f(to)h(NASA)f(Ames)g (Researc)o(h)g(Cen)o(ter)h(is)g(aero-)149 211 y(nautical)i(researc)o(h)g(and) h(engineering.)29 b(Some)18 b(of)h(the)g(disciplines)f(in)o(v)o(olv)o(ed)f (in)h(mo)q(deling)g(an)149 271 y(aircraft)h(though)h(its)e(\015igh)o(t)h(en)o (v)o(elop)q(e)e(are)i(\015uid)g(dynamics,)e(structural)i(dynamics,)f(thermal) 149 332 y(analysis,)i(propulsion,)g(con)o(trol)f(theory)l(,)g (electromagnetic)e(analysis,)j(and)f(acoustics.)31 b(All)18 b(of)149 392 y(the)c(computational)g(aeroscience)f(\(CAS\))h(grand)h(c)o (hallenges)e(of)i(the)f(HPCC)g(program)g(in)o(v)o(olv)o(e)149 452 y(coupling)j(t)o(w)o(o)f(or)g(more)f(of)i(these)f(disciplines.)223 512 y(In)d(addition,)h(geometrically)d(complex)h(aircraft)i (con\014gurations,)h(often)f(with)g(one)g(or)g(more)149 572 y(comp)q(onen)o(ts)d(in)f(relativ)o(e)f(motion)h(with)h(resp)q(ect)f(to)h (others,)h(require)e(comp)q(osite)f(mesh)h(metho)q(d-)149 633 y(ologies)20 b(for)f(e\016cien)o(t)e(spatial)i(discretization)e([4].)29 b(A)19 b(m)o(ultizonal)d(application)j(meshes)e(indi-)149 693 y(vidual)j(comp)q(onen)o(ts)f(of)h(the)f(con\014guration)i(separately)f(and)g (forms)f(a)h(comp)q(osite)f(mesh)f(b)o(y)149 753 y(o)o(v)o(erlapping)c(comp)q (onen)o(t)f(meshes.)19 b(The)c(o)o(v)o(erlapping)e(meshes)g(in)o(teract)g (through)j(data)f(in)o(ter-)149 813 y(p)q(olation)i(at)g(in)o(ter-mesh)d(b)q (oundary)k(p)q(oin)o(ts)e([8].)223 873 y(Curren)o(tly)l(,)22 b(there)g(are)g(no)h(applications)f(that)h(com)o(bine)d(all)h(of)i(the)f (disciplines.)38 b(Ho)o(w-)149 934 y(ev)o(er,)22 b(sev)o(eral)f(co)q(des)h (exist)f(that)h(com)o(bine)e(at)i(least)f(t)o(w)o(o)h(of)g(these)f (disciplines.)37 b(Some)20 b(ex-)149 994 y(amples)14 b(in)g(the)h (aeronautical)g(domain)f(are)g(\015uid-thermal,)f(\015uid-structures,)i (\015uid-acoustics,)149 1054 y(and)i(\015uid-electromagnetic.)j(A)o(t)15 b(NASA)h(Ames)e(Researc)o(h)i(Cen)o(ter,)f(\015uid-thermal)g([5],)h(\015uid-) 149 1114 y(structures)i([10],)e(and)i(m)o(ultizonal)d(\015uid)i(dynamics)f ([11])h(co)q(des)g(ha)o(v)o(e)g(b)q(een)g(implem)o(en)n(ted)d(on)149 1174 y(a)j(message)f(passing)h(parallel)e(computer,)g(the)h(In)o(tel)e (iPSC/860.)223 1234 y(This)21 b(pap)q(er)i(presen)o(ts)e(the)g(mo)q(del)g(of) h(computing)e(for)i(m)o(ultidiscipl)o(inary)d(and/or)k(m)o(ul-)149 1295 y(tizonal)g(applications)f(on)h(parallel)f(mac)o(hines)e(b)q(eing)j (pursued)g(at)g(NASA)e(Ames)g(Researc)o(h)149 1355 y(Cen)o(ter.)40 b(The)22 b(mo)q(del)f(describ)q(es)i(execution)e(on)i(a)g(parallel)f(mac)o (hine)e(with)i(some)g(n)o(um)o(b)q(er)149 1415 y(of)h(pro)q(cessors,)i(eac)o (h)d(with)g(lo)q(cal)g(memory)l(,)f(connected)h(b)o(y)g(a)h(net)o(w)o(ork.)39 b(It)22 b(do)q(es)h(not)g(dis-)149 1475 y(cuss)c(pro)q(cessor)h(rates,)f (size)e(of)i(memory)l(,)d(net)o(w)o(ork)i(latency)l(,)f(net)o(w)o(ork)h (bandwidth,)i(net)o(w)o(ork)149 1535 y(connectivit)o(y)l(,)e(disk)i(space,)g (I/O)g(bandwidth,)h(or)f(mass)f(storage.)33 b(It)20 b(also)g(do)q(es)g(not)h (discuss)149 1596 y(language)d(and)f(discipline)d(sp)q(eci\014c)h(algorithm)h (issues.)223 1656 y(The)g(pap)q(er)h(has)f(three)g(sections.)21 b(Section)16 b(2)g(describ)q(es)g(general)g(m)o(ultidisci)o(plinary)d(prob-) 149 1716 y(lem)24 b(c)o(haracteristics.)47 b(Section)25 b(3)h(discusses)f(p)q (ossible)h(solution)f(approac)o(hes.)50 b(Section)24 b(4)149 1776 y(presen)o(ts)16 b(three)g(computational)f(mo)q(dels)h(and)g(their)g (requiremen)o(ts.)149 1908 y Fi(2)56 b(Problem)17 b(Characteristics)149 2017 y Fk(A)i(m)o(ultidiscipli)o(nary)e(problem)g(is)i(comp)q(osed)g(of)h(a)g (collection)d(of)j(t)o(w)o(o)f(or)h(more)e(in)o(teracting)149 2078 y(disciplines.)31 b(A)19 b(giv)o(en)g(discipline)f(ma)o(y)h(in)o(teract) f(directly)g(with)i(one)g(or)g(more)f(of)h(the)g(other)149 2138 y(disciplines.)k(Also,)17 b(these)g(in)o(teractions)g(o)q(ccur)h(sim)o (ultaneously)d(across)j(all)f(in)o(terdisciplinary)149 2198 y(b)q(oundaries.)30 b(Similar)16 b(remarks)i(apply)g(to)i(comp)q(onen)o(t)d (meshes)h(in)g(a)h(m)o(ultizonal)d(computa-)149 2258 y(tion.)223 2318 y(A)11 b(m)o(ultidiscipli)o(nary)e(problem)i(is)h(naturally)g(decomp)q (osed)f(in)o(to)h(di\013eren)o(t)g(in)o(teracting)f(dis-)149 2379 y(ciplines,)17 b(eac)o(h)h(of)h(whic)o(h)f(ma)o(y)f(mo)q(del)g(widely)h (di\013eren)o(t)f(ph)o(ysical)h(phenomena.)27 b(Therefore,)149 2439 y(the)d(equations)f(used)h(to)f(describ)q(e)g(the)g(underlying)g(ph)o (ysics)g(\(PDEs/ODEs)i(and/or)g(in)o(te-)149 2499 y(gral)19 b(equations\))f(v)m(ary)g(b)q(et)o(w)o(een)f(disciplines.)25 b(As)18 b(a)h(result,)e(the)h(computational)f(tec)o(hniques)149 2559 y(used)i(for)g(computer)e(sim)o(ulations)f(ma)o(y)h(ha)o(v)o(e)h(widely) f(di\013eren)o(t)h(n)o(umerical)d(algorithm)j(c)o(har-)149 2619 y(acteristics,)c(data)h(structures,)f(and)h(computational)e(requiremen)o (ts.)18 b(In)c(addition,)g(on)h(parallel)149 2680 y(computers)e(with)i(ph)o (ysically)d(distributed)i(memory)l(,)d(the)j(ab)q(o)o(v)o(e)g(di\013erences)g (ma)o(y)e(induce)i(dis-)149 2740 y(tinct)19 b(data)h(partitioning)f (strategies)g(for)h(eac)o(h)e(discipline.)28 b(F)l(or)19 b(example,)e(some)i (disciplines)1036 2864 y(2)p eop %%Page: 3 3 bop 149 42 a Fk(ma)o(y)17 b(b)q(e)h(mo)q(deled)e(using)i(\014nite)f (di\013erence)g(discretization)g(on)h(logically)f(structured)g(meshes)149 102 y(while)i(others)g(ma)o(y)f(use)h(a)g(\014nite)g(elemen)o(t)d (discretization)i(on)h(unstructured)g(meshes.)29 b(Also,)149 162 y(the)23 b(computational)g(requiremen)n(ts)e(within)h(eac)o(h)h (discipline)e(ma)o(y)g(v)m(ary)j(dynamically)c(due)149 222 y(to)g(solution)g(adaptiv)o(e)f(mesh)g(re\014nemen)o(t/coarsening,)f(mo)o (ving)g(b)q(oundaries,)j(and)f(ev)o(olving)149 282 y(material)15 b(non-linearities,)g(etc.)149 415 y Fi(3)56 b(Solution)19 b(Approac)n(hes)149 525 y Fk(Giv)o(en)h(a)h(m)o(ultidiscipl)o(inary)d(and/or)j(m)o(ultizonal)e (problem,)g(what)j(w)o(ould)e(b)q(e)h(the)f(b)q(est)h(ap-)149 585 y(proac)o(h)g(to)f(solving)g(it)g(on)g(a)h(parallel)e(computer?)32 b(On)20 b(a)g(serial)g(computer,)f(the)h(disciplines)149 645 y(and/or)15 b(comp)q(onen)o(t)c(meshes)h(m)o(ust)f(b)q(e)i(pro)q(cessed)g (serially)l(.)19 b(This)13 b(is)g(accomplished)e(b)o(y)h(ha)o(ving)149 706 y(a)21 b(doubly)f(nested)g(lo)q(op.)34 b(The)21 b(outer)f(lo)q(op)h (iterates)f(for)g(the)g(appropriate)h(n)o(um)o(b)q(er)d(of)j(time)149 766 y(steps)c(and)g(the)f(inner)f(lo)q(op)i(iterates)f(o)o(v)o(er)f(the)h (disciplines)f(\(meshes\).)223 826 y(F)l(or)d(a)h(parallel)e(computer,)h(the) g(disciplines)e(\(meshes\))h(ma)o(y)g(b)q(e)h(executed)f(in)h(parallel)g(or)h (se-)149 886 y(quen)o(tially)l(.)19 b(In)14 b(the)g(parallel)f(execution)h (mo)q(de,)f(eac)o(h)h(discipline)e(\(mesh\))h(ma)o(y)g(b)q(e)h(partitioned) 149 946 y(across)19 b(a)f(di\013eren)o(t)f(set)h(of)g(pro)q(cessors)h(and)g (computation)e(pro)q(ceeds)h(in)f(parallel)g(within)h(dis-)149 1007 y(ciplines)i(\(meshes\))g(and)i(across)h(disciplines)c(\(meshes\).)36 b(Both)21 b(data)h(parallelism)e(and)i(task)149 1067 y(parallelism)f(are)h (used.)40 b(In)23 b(the)f(sequen)o(tial)f(execution)g(mo)q(de,)i(eac)o(h)f (discipline)f(\(mesh\))g(is)149 1127 y(partitioned)15 b(across)g(all)f(pro)q (cessors)i(and)f(the)f(computation)f(pro)q(ceeds)i(in)f(parallel)g(within)g (dis-)149 1187 y(ciplines)h(\(meshes\))g(but)h(sequen)o(tially)f(across)i (disciplines)e(\(meshes\).)k(Only)d(data)h(parallelism)149 1247 y(is)f(used.)223 1307 y(The)k(decision)g(whether)g(or)h(not)f(to)h(use)g (task)f(parallelism)e(across)k(disciplines)c(\(meshes\))149 1368 y(is)g(in\015uenced)e(b)o(y)h(sev)o(eral)f(factors:)24 b(memory)15 b(requiremen)o(t)o(s,)g(memory)f(e\016ciency)l(,)h(computa-)149 1428 y(tional)d(requiremen)o(ts,)d(Amdahl's)h(La)o(w,)j(soft)o(w)o(are)f (engineering)f(issues,)i(and)f(m)o(ultidisci)o(plinary)149 1488 y(coupling)17 b(issues.)223 1548 y(In)g(the)g(discussion)g(b)q(elo)o(w)g (when)g(comparing)g(t)o(w)o(o)g(approac)o(hes,)h(a)f(\014xed)g(n)o(um)o(b)q (er)f(of)h(pro-)149 1608 y(cessors)f(and)f(a)g(\014xed)f(problem)f(size)h (are)h(assumed.)20 b(A)14 b(\014xed)h(n)o(um)o(b)q(er)e(of)i(pro)q(cessors)h (is)e(a)h(v)m(alid)149 1669 y(assumption)20 b(b)q(ecause)h(at)f(an)o(y)h (instan)o(t,)f(the)g(goal)h(is)f(to)h(use)f(the)g(a)o(v)m(ailable)f (resources)i(most)149 1729 y(e\016cien)o(tly)l(.)27 b(A)18 b(\014xed)h(problem)e(size)h(is)h(appropriate)g(b)q(ecause)h(once)e(the)h (relev)m(an)o(t)f(ph)o(ysics)g(is)149 1789 y(resolv)o(ed)f(to)g(the)g (desired)g(lev)o(el)e(of)j(accuracy)l(,)e(further)h(re\014nemen)o(t)e(\\w)o (astes")k(computational)149 1849 y(resources)e(\(memory)c(and)k(CPU)f (cycles\).)149 1977 y Fi(3.1)56 b(Memory)17 b(Requirem)o(en)n(ts)149 2069 y Fk(Memory)j(requiremen)o(ts)f(include)i(the)g(size)g(of)h(the)g (executable)e(and)i(the)g(size)f(of)h(the)f(data.)149 2130 y(F)l(or)i(a)f(SIMD)g(mac)o(hine,)f(there)g(is)h(a)g(single)g(cop)o(y)f(of)i (the)f(executable)e(and)j(the)f(disciplines)149 2190 y(are)d(pro)q(cessed)g (sequen)o(tially)e(\(data)i(parallelism)d(only\).)29 b(On)18 b(a)h(MIMD)f(mac)o(hine,)f(there)h(are)149 2250 y(m)o(ultiple)g(copies)i(of)h (the)f(executable.)33 b(If)20 b(the)g(disciplines)f(are)i(pro)q(cessed)g(in)f (parallel)g(\(task)149 2310 y(and)i(data)g(parallelism\),)e(the)h(set)g(of)h (pro)q(cessors)g(asso)q(ciated)g(with)f(a)h(discipline)d(only)i(need)149 2370 y(the)c(executable)f(for)i(that)f(discipline.)23 b(If)16 b(the)h(disciplines)f(are)h(pro)q(cessed)h(sequen)o(tially)d(\(data)149 2431 y(parallelism)j(only\),)j(eac)o(h)e(pro)q(cessor)j(needs)d(a)i(cop)o(y)f (of)g(the)g(executable)f(for)h(all)g(disciplines.)149 2491 y(Dep)q(ending)g(on)f(the)g(amoun)o(t)f(of)h(lo)q(cal)g(memory)l(,)e(whether) h(or)i(not)f(virtual)f(memory)e(is)j(sup-)149 2551 y(p)q(orted,)d(and)f(the)g (parallel)f(I/O)h(bandwidth,)h(this)f(ma)o(y)e(or)i(ma)o(y)f(not)h(b)q(e)g (an)h(issue.)k(Curren)o(tly)l(,)149 2611 y(most)13 b(massiv)o(ely)d(parallel) i(pro)q(cessors)j(should)e(b)q(e)g(though)o(t)h(of)f(as)h(ph)o(ysical)e (memory)e(mac)o(hines)149 2671 y(and)17 b(are)g(considered)e(suc)o(h)h(in)g (this)g(pap)q(er.)1036 2864 y(3)p eop %%Page: 4 4 bop 223 42 a Fk(The)19 b(memory)d(required)i(to)h(store)h(data)g(also)f(v)m (aries.)30 b(On)19 b(a)h(SIMD)f(mac)o(hine,)e(since)h(the)149 102 y(amoun)o(t)d(of)g(data)h(for)g(eac)o(h)e(discipline)g(\(mesh\))f(v)m (ary)l(,)i(some)f(data)i(structures)g(ma)o(y)d(b)q(e)i(padded)149 162 y(dep)q(ending)j(on)g(sp)q(eci\014c)g(mac)o(hine)d(or)j(compiler)d (requiremen)o(ts)g([12].)25 b(Also,)18 b(temp)q(orary)f(v)m(ari-)149 222 y(ables)g(are)g(often)g(the)g(size)f(of)h(the)g(whole)f(mesh.)22 b(On)17 b(a)g(MIMD)f(mac)o(hine,)f(when)i(a)g(domain)f(is)149 282 y(partitioned)c(across)h(m)o(ultiple)c(pro)q(cessors,)14 b(some)d(data)h(are)g(often)h(duplicated)e(to)h(sa)o(v)o(e)g(comm)o(u-)149 342 y(nication)18 b(costs)g([2].)25 b(Also,)18 b(temp)q(orary)e(v)m (ariables,)i(t)o(ypically)e(scalar)i(or)g(v)o(ector)e(temp)q(oraries,)149 403 y(and)k(small)e(global)h(data)h(structures)f(\(i.e.,)f(small)g(lo)q(ok)h (up)h(tables\))f(are)g(replicated)f(to)h(a)o(v)o(oid)149 463 y(b)q(ottlenec)o(ks.)32 b(Bu\013er)20 b(space)g(is)g(also)h(needed)e(on)i (eac)o(h)e(pro)q(cessor)i(to)g(bu\013er)f(messages)g(in)f(a)149 523 y(message)d(passing)h(en)o(vironmen)o(t.)223 583 y(If)f(the)h (disciplines)f(\(meshes\))g(are)h(pro)q(cessed)h(in)e(parallel,)h(eac)o(h)f (discipline)g(\(mesh\))g(has)h(a)149 643 y(b)q(etter)g(surface)f(to)h(v)o (olume)d(ratio)j(for)g(its)f(data)h(implying)d(less)j(duplicated)e(data.)23 b(\(Assuming)149 704 y(a)18 b(\014xed)f(n)o(um)o(b)q(er)f(of)i(pro)q(cessors) g(for)g(the)f(whole)g(application.\))25 b(Also,)17 b(the)g(memory)d(required) 149 764 y(for)23 b(temp)q(orary)f(v)m(ariables)g(dep)q(ends)h(up)q(on)g(the)f (individual)g(discipline.)37 b(If)22 b(the)h(disciplines)149 824 y(\(meshes\))d(are)h(pro)q(cessed)g(sequen)o(tially)l(,)f(the)h(lo)q(cal) g(memory)d(asso)q(ciated)k(with)e(a)i(pro)q(cessor)149 884 y(m)o(ust)17 b(b)q(e)h(shared)g(b)o(y)f(all)g(disciplines)f(\(meshes\).)24 b(This)18 b(implies)d(a)j(higher)f(surface)h(to)g(v)o(olume)149 944 y(ratio)f(for)f(eac)o(h)f(discipline)f(\(mesh\))h(since)g(eac)o(h)h (discipline)e(\(mesh\))g(is)i(spread)g(o)o(v)o(er)g(more)e(pro-)149 1005 y(cessors.)21 b(This)13 b(in)g(turn)g(implies)d(more)i(duplicated)g (data)i(to)f(a)o(v)o(oid)f(comm)o(unication.)17 b(Also,)c(the)149 1065 y(memory)j(requiremen)o(t)g(for)j(temp)q(orary)f(v)m(ariables)h(is)f (dep)q(enden)o(t)h(on)g(the)f(discipline)f(\(mesh\))149 1125 y(with)g(the)f(largest)g(requiremen)o(ts.)149 1252 y Fi(3.2)56 b(Memory)17 b(E\016ciency)149 1344 y Fk(Greater)22 b(parallelism)e(implies)f (more)h(memory)f(is)j(required)f(to)h(solv)o(e)f(the)h(application)f(e\016-) 149 1405 y(cien)o(tly)l(.)f(Let)c(memory)d(e\016ciency)h(b)q(e)j(de\014ned)f (as)932 1531 y Ff(")955 1538 y Fe(M)1008 1531 y Fk(=)1072 1497 y Ff(M)1119 1504 y Fd(0)p 1065 1519 81 2 v 1065 1565 a Ff(M)1112 1572 y Fe(N)1151 1531 y Ff(;)149 1656 y Fk(where)24 b Ff(")321 1663 y Fe(M)383 1656 y Fk(is)g(the)f(memory)d(e\016ciency)l(,)j Ff(M)1002 1663 y Fd(0)1045 1656 y Fk(is)h(amoun)o(t)e(of)i(the)f(memory)e (required)h(to)i(\014t)149 1716 y(the)d(problem)d(on)j(the)f(smallest)f(n)o (um)o(b)q(er)f(of)j(pro)q(cessors)h(\()p Ff(P)1311 1723 y Fd(0)1331 1716 y Fk(\),)f(and)g Ff(M)1531 1723 y Fe(N)1585 1716 y Fk(is)f(the)g(amoun)o (t)g(of)149 1777 y(memory)12 b(used)j(on)g Ff(P)543 1784 y Fe(N)592 1777 y Fk(pro)q(cessors)h(\()p Ff(P)873 1784 y Fd(0)907 1777 y Fc(\024)d Ff(P)990 1784 y Fe(N)1024 1777 y Fk(\).)21 b(Then)15 b(the)f(claim)f(is)h(that)h(memory)d(e\016ciency)149 1837 y(decreases)19 b(as)g(the)g(n)o(um)o(b)q(er)e(of)i(pro)q(cessors)h (increase.)28 b(All)18 b(that)h(is)g(necessary)f(for)h(this)g(claim)149 1897 y(to)f(b)q(e)f(true)g(is)f(for)i(a)f(single)g(v)m(ariable/compiler)d (temp)q(orary)j(to)g(b)q(e)g(replicated)f(across)i(t)o(w)o(o)f(or)149 1957 y(more)e(pro)q(cessors.)223 2017 y(As)f(an)h(example)e(of)i(ho)o(w)g (memory)d(requiremen)o(ts)g(can)j(gro)o(w)g(with)g(increased)f(parallelism,) 149 2078 y(consider)h(a)g(32)8 b Fc(\002)g Fk(32)g Fc(\002)g Fk(32)17 b(mesh.)i(\(F)l(airly)14 b(small)f(mesh)h(sizes)g(o)q(ccur)h (frequen)o(tly)e(in)h(m)o(ultizonal)149 2138 y(applications.\))34 b(F)l(orm)19 b(an)i(indep)q(enden)o(t)e(scalar)i(tridiagonal)g(system)e(out)h (of)h(eac)o(h)f(column.)149 2198 y(Then)i(there)f(are)h(32)547 2180 y Fd(2)589 2198 y Fk(indep)q(enden)o(t)f(tridiagonal)h(systems,)f(eac)o (h)g(of)h(length)f(32.)38 b(Compare)149 2258 y(the)20 b(memory)c(requiremen)o (ts)h(to)j(solv)o(e)f(the)g(systems)f(on)i(a)g(serial)f(mac)o(hine,)f(a)i (single)f(v)o(ector)149 2318 y(pro)q(cessor,)j(and)e(a)h(parallel)e(v)o (ector)g(mac)o(hine.)30 b(The)20 b(solution)g(tec)o(hnique)f(will)f(b)q(e)j (Gaussian)149 2379 y(elimination)14 b(without)j(piv)o(oting)e(where)h(the)g (input)g(ma)o(y)f(b)q(e)h(o)o(v)o(erwritten.)223 2439 y(On)g(a)h(serial)e (mac)o(hine,)f(32)725 2421 y Fd(3)757 2439 y Fk(+)d(3)g Fc(\002)g Fk(32)17 b(w)o(ords)g(are)f(required,)f(32)1430 2421 y Fd(3)1467 2439 y Fk(for)h(the)g(righ)o(t)g(hand)h(side)149 2499 y(and)i(3)13 b Fc(\002)f Fk(32)19 b(for)g(a)g(single)e(tridiagonal)i(system.)26 b(The)18 b(systems)f(are)h(formed)f(and)i(solv)o(ed)f(one)149 2559 y(at)f(a)g(time.)223 2619 y(On)g(a)i(single)e(v)o(ector)g(pro)q(cessor,) i(32)910 2601 y Fd(3)942 2619 y Fk(+)13 b(3)f Fc(\002)g Fk(32)1128 2601 y Fd(2)1166 2619 y Fk(w)o(ords)19 b(are)f(required,)e(32)1645 2601 y Fd(3)1684 2619 y Fk(for)i(the)f(righ)o(t)149 2680 y(hand)f(side)f(and) g(3)9 b Fc(\002)f Fk(32)588 2661 y Fd(2)623 2680 y Fk(to)16 b(form)e(32)h(indep)q(enden)o(t)f(tridiagonal)i(systems.)j(A)o(t)14 b(least)h(32)h(of)f(the)149 2740 y(systems)g(are)i(formed)e(at)h(a)h(time)d (to)j(allo)o(w)f(for)g(v)o(ectorization)f(across)i(indep)q(enden)o(t)f (systems.)1036 2864 y(4)p eop %%Page: 5 5 bop 223 42 a Fk(On)13 b(a)h(32)h(no)q(de)f(parallel)f(v)o(ector)g(mac)o (hine,)e(4)6 b Fc(\002)g Fk(32)1157 23 y Fd(3)1191 42 y Fk(w)o(ords)14 b(are)g(required,)f(32)1658 23 y Fd(3)1692 42 y Fk(for)h(the)f(righ)o(t)149 102 y(hand)20 b(side)f(and)h(3)13 b Fc(\002)g Fk(32)610 84 y Fd(3)650 102 y Fk(to)19 b(form)f(32)878 84 y Fd(2)918 102 y Fk(indep)q(enden)o(t)g(tridiagonal)i(systems.)28 b(Thirt)o(y)19 b(t)o(w)o(o)g(of)149 162 y(the)d(systems)f(are)i(formed)e(at)h(a)h(time)d(on) j(eac)o(h)f(pro)q(cessor)h(to)f(allo)o(w)g(for)h(v)o(ectorization.)223 222 y(If)c(the)h(serial)g(mac)o(hine)e(has)j(a)f(memory)d(e\016ciency)h(of)j (1)p Ff(:)p Fk(0,)f(then)g(the)g(single)g(v)o(ector)f(pro)q(ces-)149 282 y(sor)h(and)f(parallel)f(v)o(ector)g(mac)o(hine)e(ha)o(v)o(e)i(memory)e (e\016ciencies)g(of)k(0)p Ff(:)p Fk(92)f(and)g(0)p Ff(:)p Fk(25)h(resp)q (ectiv)o(ely)l(.)149 410 y Fi(3.3)56 b(Computational)18 b(Requirem)o(en)n(ts) 149 502 y Fk(Computational)d(requiremen)o(ts)d(for)j(di\013eren)o(t)e (disciplines)g(and)j(di\013eren)o(t)e(meshes)f(v)m(ary)l(.)21 b(Also,)149 563 y(mesh)16 b(sizes)g(ma)o(y)g(v)m(ary)h(widely)f(in)g(m)o (ultizonal)f(computations)h(yielding)g(v)o(ery)g(di\013eren)o(t)g(com-)149 623 y(putational)f(requiremen)n(ts)c(for)j(di\013eren)o(t)f(meshes.)19 b(This)14 b(leads)f(to)h(load)g(balancing)g(issues.)21 b(F)l(or)149 683 y(example,)14 b(computational)h(requiremen)o(ts)e(in)i(the)h(con)o(trols) g(discipline)e(is)i(m)o(uc)o(h)d(smaller)h(than)149 743 y(the)i (computational)g(requiremen)o(ts)d(to)k(compute)e(a)h(\015uid)g(\015o)o(w)h (\014eld.)223 803 y(If)g(disciplines)f(\(meshes\))h(are)g(pro)q(cessed)i(in)e (parallel)g(\(task)h(and)h(data)g(parallelism\))c(then)149 864 y(the)h(n)o(um)o(b)q(er)e(of)i(pro)q(cessors)h(assigned)f(to)g(eac)o(h)g (discipline)d(\(mesh\))i(can)h(b)q(e)g(adjusted)g(to)g(com-)149 924 y(putationally)g(load)h(balance)f(the)g(application.)21 b(\(Sub)s(ject)16 b(to)g(memory)e(constrain)o(ts.\))223 984 y(If)i(disciplines)f(\(meshes\))g(are)i(pro)q(cessed)g(sequen)o(tially)d (\(data)k(parallelism)c(only\),)i(for)h(dis-)149 1044 y(ciplines)d (\(meshes\))h(with)g(small)f(memory)f(requiremen)n(ts)g(and)j(small)e (computational)h(require-)149 1104 y(men)o(ts,)g(trying)h(to)h(use)g(all)f(a) o(v)m(ailable)g(pro)q(cessors)i(ma)o(y)d(actually)h(slo)o(w)h(do)o(wn)g(the)f (application)149 1165 y(due)h(to)g(the)g(decrease)f(in)g(the)h(computation)f (to)h(comm)o(unicati)o(on)e(ratio.)23 b(If)16 b(the)h(computation)149 1225 y(asso)q(ciated)d(with)f(a)g(discipline)e(\(mesh\))g(is)h(unable)h(to)g (use)g(all)f(a)o(v)m(ailable)g(pro)q(cessors,)i(then)f(some)149 1285 y(pro)q(cessors)18 b(will)d(b)q(e)h(idle)g(during)g(that)h(phase)f(of)h (the)f(computation.)223 1345 y(Ho)q(c)o(kney)h(and)i(Jesshop)q(e)g([7])f (discuss)g(trade)h(o\013s)g(in)f(algorithm)f(selection.)26 b(Whic)o(h)18 b(algo-)149 1405 y(rithm)c(is)i(\\b)q(est")h(dep)q(ends)f(up)q (on)g(the)g(mac)o(hine)d(arc)o(hitecture,)h(actual)i(n)o(um)o(b)q(er)e(of)i (pro)q(cessors)149 1466 y(b)q(eing)e(used,)f(and)h(the)f(problem)e(size.)19 b(An)13 b(algorithm)f(with)h(more)f(inheren)o(t)g(parallelism)f(migh)o(t)149 1526 y(not)20 b(execute)d(as)i(fast)g(as)h(an)f(algorithm)f(with)g(less)h (inheren)o(t)e(parallelism)f(due)j(to)g(an)g(o)o(v)o(erall)149 1586 y(increase)d(in)g(op)q(erations.)149 1714 y Fi(3.4)56 b(Amdahl's)18 b(La)n(w)149 1806 y Fk(E\016ciency)c(also)i(v)m(aries)f(dep)q (ending)h(on)f(whether)g(the)h(disciplines)d(\(meshes\))h(are)h(pro)q(cessed) h(in)149 1866 y(parallel)g(or)g(sequen)o(tially)l(.)j(Amdahl's)14 b(La)o(w)j(sa)o(ys)f(that)h(sp)q(eedup)f(is)g(b)q(ounded)h(b)o(y)e(the)h (fraction)149 1926 y(of)h(the)f(co)q(de)g(that)h(is)f(serial:)847 1998 y Ff(S)h Fc(\024)1069 1965 y Ff(N)p 951 1987 280 2 v 951 2033 a Fk(\()p Ff(N)g Fc(\000)11 b Fk(1\))p Ff(\027)j Fk(+)d(1)1236 1998 y Ff(;)149 2111 y Fk(where)18 b Ff(S)i Fk(is)e(the)f(sp)q(eedup,)h Ff(N)23 b Fk(is)17 b(the)h(n)o(um)o(b)q(er)e(of)i(pro)q(cessors,)h(and)f Ff(\027)j Fk(is)c(the)h(fraction)f(of)h(the)149 2171 y(co)q(de)f(that)g(is)f (serial.)k(E\016ciency)l(,)14 b Ff(")p Fk(,)i(is)g(de\014ned)g(as)793 2302 y Ff(")d Fk(=)892 2269 y Ff(S)p 886 2291 45 2 v 886 2336 a(N)949 2302 y Fk(=)1133 2269 y(1)p 1006 2291 280 2 v 1006 2336 a(\()p Ff(N)j Fc(\000)11 b Fk(1\))p Ff(\027)k Fk(+)c(1)1290 2302 y Ff(:)149 2437 y Fk(Amdahl's)h(La)o(w)i(is)f(relev)m(an)o(t)f(since)h (these)g(discussions)g(compare)g(and)h(con)o(trast)f(the)g(pro)q(cessing)149 2498 y(of)k(a)g(\014xed)f(n)o(um)o(b)q(er)e(of)j(m)o(ultiple)c(disciplines)h (\(meshes\).)223 2558 y(If)k(the)h(disciplines)f(\(meshes\))f(are)i(pro)q (cessed)h(in)f(parallel,)f(assume)h(that)g(the)g(n)o(um)o(b)q(er)f(of)149 2618 y(pro)q(cessors)j(assigned)e(to)h(eac)o(h)e(discipline)f(\(mesh\),)h Ff(G)1190 2625 y Fe(i)1204 2618 y Fk(,)i(is)e(suc)o(h)h(that)g(there)g(is)f (near)i(p)q(erfect)149 2678 y(load)c(balance)f(across)h(disciplines)d (\(meshes\).)19 b(Giv)o(en)14 b(the)h(same)f(n)o(um)o(b)q(er)f(of)j(pro)q (cessors)g(\()p Ff(N)j Fk(=)149 2705 y Fb(P)202 2738 y Ff(G)240 2745 y Fe(i)254 2738 y Fk(\),)14 b(pro)q(cessing)g(disciplines)d(\(meshes\))h (sequen)o(tially)f(m)o(ust)h(ha)o(v)o(e)h(a)h(lo)o(w)o(er)e(e\016ciency)l(.) 18 b(Eac)o(h)1036 2864 y(5)p eop %%Page: 6 6 bop 149 42 a Fk(discipline)13 b(\(mesh\))g(is)h(spread)h(o)o(v)o(er)f(more)f (pro)q(cessors)j(and)f(Amdahl's)d(La)o(w)j(implies)d(e\016ciency)149 102 y(decreases)k(as)h(the)f(n)o(um)o(b)q(er)f(of)h(pro)q(cessors)i (increases)e(for)g(a)h(\014xed)f(problem)e(size.)223 162 y(F)l(or)g(example,) e(assume)i(that)h(a)g(m)o(ultizonal)d(application)i(has)h(t)o(w)o(o)g(iden)o (tical)e(sized)g(meshes)149 222 y(with)g(the)g(same)f(computational)g(load)i (for)f(eac)o(h.)20 b(F)l(urther,)13 b(assume)f(that)h(the)g(computation)f(on) 149 282 y(eac)o(h)k(is)f(99.9\045)h(parallel.)k(Giv)o(en)15 b(1000)i(pro)q(cessors,)g(if)e(the)g(meshes)g(are)g(pro)q(cessed)h(in)g (parallel)149 342 y(\(500)g(pro)q(cessors)f(assigned)g(to)f(eac)o(h)f(mesh\)) g(then)h(the)g(e\016ciency)d(is)j(66)p Ff(:)p Fk(7\045.)21 b(If)13 b(the)h(meshes)f(are)149 403 y(pro)q(cessed)k(sequen)o(tially)e (\(1000)j(pro)q(cessors)f(assigned)h(to)e(eac)o(h)g(mesh\))f(then)i(the)f (e\016ciency)e(is)149 463 y(50)p Ff(:)p Fk(0\045.)21 b(This)14 b(means)e(that)i(pro)q(cessing)h(the)e(meshes)f(sequen)o(tially)f(will)i(tak) o(e)g(33\045)h(longer)f(than)149 523 y(pro)q(cessing)k(them)e(in)h(parallel.) 223 583 y(Other)c(sources)h(of)g(ine\016ciency)d(are)i(an)h(increased)f(comm) o(unication)e(to)j(computation)f(ratio)149 643 y(and)24 b(an)g(increase)f(in) g(redundan)o(t)h(computations)f(as)h(the)f(p)q(ercen)o(tage)g(of)g (duplicated)g(data)149 704 y(increases.)149 831 y Fi(3.5)56 b(Soft)n(w)n(are)20 b(Engineering)149 924 y Fk(Some)d(of)i(the)e(soft)o(w)o (are)i(engineering)e(issues)h(are)g(mo)q(dularit)o(y)l(,)e(indep)q(endence,)h (extensibilit)o(y)l(,)149 984 y(mo)q(di\014abilit)o(y)l(,)d(readabilit)o(y)l (,)h(and)i(main)o(tainabilit)o(y)l(.)i(Soft)o(w)o(are)e(engineering)e(ma)o(y) g(b)q(e)i(though)o(t)149 1044 y(of)h(as)f(complexit)o(y)d(managemen)o(t.)20 b(Tw)o(o)e(primary)d(w)o(a)o(ys)i(h)o(umans)f(deal)h(with)f(complexit)o(y)e (are)149 1104 y(divide-and-conquer)22 b(and)h(abstraction.)41 b(The)23 b(trend)f(o)o(v)o(er)g(the)g(last)g(thirt)o(y)g(y)o(ears)g(in)g (soft-)149 1165 y(w)o(are)17 b(engineering)f(has)h(b)q(een)g(to)o(w)o(ards)g (greater)g(co)q(de)g(mo)q(dularit)o(y)e(and)i(greater)g(information)149 1225 y(hiding.)24 b(Structured)16 b(programming,)g(top-do)o(wn)i(design,)f (program)g(mo)q(dules,)e(abstract)j(data)149 1285 y(structures,)c(ob)s(ject)g (orien)o(ted)f(programming)f(are)i(all)g(examples)e(of)i(this)g(trend.)20 b(They)14 b(encour-)149 1345 y(age)i(the)e(partitioning)h(of)g(problems)f(in) o(to)g(simpler)f(pieces,)g(in)o(terfaces)h(to)h(b)q(e)g(de\014ned,)g(and)g (the)149 1405 y(impleme)o(n)o(tation)f(details)h(to)i(b)q(e)f(hidden)g(b)q (ehind)g(the)g(in)o(terfaces.)149 1533 y Fi(3.6)56 b(Multidisciplinary)16 b(Coupling)149 1626 y Fk(The)d(principal)e(metho)q(ds)h(for)g(coupling)h(m)o (ultiple)c(disciplines)h(are)j(global)f(explicit)f(in)o(tegration,)149 1686 y(global)18 b(implicit)c(in)o(tegration,)j(mo)q(dal)f(equation)i(in)o (tegration,)e(and)i(partitioned)f(analysis)h([6].)149 1746 y(Global)c(explicit)d(in)o(tegration)h(applies)h(an)h(explicit)d(time)f(in)o (tegration)j(algorithm)f(to)i(the)f(global)149 1806 y(set)19 b(of)g(equations.)29 b(Ho)o(w)o(ev)o(er,)17 b(stabilit)o(y)g(is)i(a)g (concern)f(and)h(the)g(time)d(step)j(m)o(ust)e(b)q(e)i(c)o(hosen)149 1866 y(based)e(on)g(the)f(discipline)e(with)i(the)g(sti\013est)h(requiremen)n (ts.)223 1926 y(Mo)q(dal)12 b(equation)f(in)o(tegration)h(represen)o(ts)f (the)g(resp)q(onse)i(of)f(a)g(system)e(b)o(y)h(mo)q(des)h(go)o(v)o(erned)149 1987 y(b)o(y)g(mo)q(dal)g(equations)h(of)f(motion.)19 b(Ho)o(w)o(ev)o(er,)11 b(it)h(is)g(hard)h(to)g(determining)d(a)j(set)f(of)h(basis)f(mo)q(des)149 2047 y(for)17 b(a)g(nonlinear)f(system.)223 2107 y(Global)j(implicit)d (coupling)i(sc)o(heme)f(is)i(used,)g(the)g(problem)f(can)h(b)q(e)g(expressed) g(in)f(blo)q(c)o(k)149 2167 y(matrix)d(form)g(where)h(the)f(main)g(diagonal)i (blo)q(c)o(ks)f(represen)o(t)f(the)h(individual)f(disciplines)f(and)149 2227 y(the)e(o\013-diagonal)i(blo)q(c)o(ks)d(represen)o(t)g(the)g(coupling)h (b)q(et)o(w)o(een)f(the)g(disciplines.)19 b(Then)11 b(the)h(whole)149 2288 y(system)18 b(can)g(b)q(e)h(solv)o(ed)f(as)i(a)f(large)f(single)g (matrix.)27 b(Ho)o(w)o(ev)o(er,)17 b(in)h(man)o(y)g(instances,)g(it)g(ma)o(y) 149 2348 y(not)c(b)q(e)g(p)q(ossible)g(to)g(directly)d(couple)i(t)o(w)o(o)h (disciplines.)k(This)c(is)f(due)h(to)f(di\016culties)f(asso)q(ciated)149 2408 y(with)k(the)g(explicit)e(ev)m(aluation)i(of)g(o\013-diagonal)i(blo)q(c) o(ks.)j(F)l(or)16 b(example,)e(it)h(w)o(ould)h(b)q(e)g(di\016cult)149 2468 y(to)j(directly)e(couple)h(a)h(particle)e(sim)o(ulation)g(of)i(a)f (rare\014ed)h(gas)g(to)g(the)f(structural)h(and)g(ther-)149 2528 y(mal)d(analysis)h(of)g(the)g(b)q(o)q(dy)h(o)o(v)o(er)e(whic)o(h)g(it)g (\015o)o(ws.)24 b(Ev)o(en)16 b(if)g(all)h(o\013-diagonal)i(coupling)d(terms) 149 2589 y(can)k(b)q(e)f(ev)m(aluated)g(explicitly)l(,)d(direct)i(in)o(v)o (ersion)f(of)j(suc)o(h)e(a)i(large)f(sparse)g(matrix)f(w)o(ould)h(b)q(e)149 2649 y(prohibitiv)o(ely)14 b(exp)q(ensiv)o(e)h(in)h(terms)e(of)j(b)q(oth)g (memory)c(and)j(CPU)h(time)c(for)k(problems)e(of)h(an)o(y)149 2709 y(consequence)i(to)h(the)f(aeronautics)h(comm)o(unit)o(y)-5 b(.)26 b(As)18 b(a)h(result,)f(it)g(is)h(necessary)f(to)h(resort)g(to)1036 2864 y(6)p eop %%Page: 7 7 bop 149 42 a Fk(some)17 b(form)f(of)h(iterativ)o(e)e(solution)j(approac)o(h)f (based)h(on)g(the)f(concept)f(of)i(sub-structuring)g(or)149 102 y(partitioned)g(analysis.)26 b(A)17 b(natural)h(form)f(of)h (sub-structuring)g(across)h(discipline)d(b)q(oundaries)149 162 y(w)o(ould)24 b(b)q(e)f(inevitable)f(due)h(to)g(widely)f(di\013eren)o(t)h (n)o(umerical)d(c)o(haracteristics)i(across)i(disci-)149 222 y(plines.)c(A)14 b(monolithic)e(iterativ)o(e)h(solv)o(er)g(is)i(unlik)o(ely)d (to)i(b)q(e)h(able)f(to)h(cop)q(e)f(with)g(all)g(the)g(div)o(erse)149 282 y(disciplines)h(in)h(an)h(e\016cien)o(t)d(manner.)223 342 y(In)h(partitioned)f(analysis,)i(the)f(e\013ect)f(of)i(o\013-diagonal)h(blo)q (c)o(ks)e(are)g(represen)o(ted)f(as)i(forcing)149 403 y(functions)23 b(on)g(the)g(righ)o(t)f(hand)i(side.)40 b(The)23 b(computation)f(of)h(these)f (forcing)h(terms)e(w)o(ould)149 463 y(induce)13 b(in)o(ter-disciplinary)e (comm)o(unication.)18 b(P)o(artitioned)13 b(analysis)g(leads)h(to)g(blo)q(c)o (k)f(iterativ)o(e)149 523 y(metho)q(ds)k(\(blo)q(c)o(k)g(Jacobi,)g(blo)q(c)o (k)g(Gauss-Seidel,)g(etc.\).)23 b(Also,)17 b(with)g(partitioned)g(analysis,)g (it)149 583 y(is)i(easier)g(to)g(extend)f(the)h(application)f(to)h(include)f (new)h(disciplines.)27 b(Ho)o(w)o(ev)o(er,)18 b(robust)h(and)149 643 y(e\016cien)o(t)c(m)o(ultidiscipl)o(inary)f(and/or)k(m)o(ultizonal)c (coupling)i(tec)o(hniques)g(is)g(an)h(activ)o(e)e(area)j(of)149 704 y(researc)o(h.)223 764 y(P)o(ark)j(and)h(F)l(elippa)e([9])h(list)g(man)o (y)e(of)j(the)f(problems)f(asso)q(ciated)i(with)g(global)f(implicit)149 824 y(coupling)16 b(and)h(the)f(adv)m(an)o(tages)i(of)e(partitioned)g (analysis.)21 b(F)l(rom)15 b(a)i(soft)o(w)o(are)f(viewp)q(oin)o(t,)f(the)149 884 y(list)i(of)h(disadv)m(an)o(tages)g(of)g(direct)f(coupling)g(and)h(adv)m (an)o(tages)h(of)e(partitioned)g(analysis)h(could)149 944 y(b)q(e)f(said)f (ab)q(out)i(monolithic)c(programming)h(v)o(ersus)h(mo)q(dular)f(programming)g (resp)q(ectiv)o(ely)l(.)149 1078 y Fi(4)56 b(Computational)18 b(Mo)r(dels)149 1187 y Fk(Giv)o(en)d(the)g(ab)q(o)o(v)o(e)h(considerations)g (\(memory)c(requiremen)o(ts,)h(computational)h(requiremen)o(ts,)149 1247 y(Amdahl's)19 b(La)o(w,)j(soft)o(w)o(are)f(engineering,)f(and)i(m)o (ultidisci)o(pli)o(nary)c(and/or)k(m)o(ultizonal)c(cou-)149 1307 y(pling\))d(the)f(most)g(\\natural")j(approac)o(h)e(to)g(implem)o(en)o (ti)o(ng)e(a)i(m)o(ultidisci)o(plinary)d(and/or)k(m)o(ul-)149 1368 y(tizonal)i(application)g(on)h(parallel)e(pro)q(cessor)i(is)f (partitioned)g(analysis)g(using)g(a)h(MIMD)e(st)o(yle)149 1428 y(of)i(execution)d(across)j(disciplines)e(\(meshes\).)24 b(This)18 b(allo)o(ws)g(n)o(umerical)e(algorithms)h(and)h(pro-)149 1488 y(grams)e(p)q(ertaining)g(to)g(individual)f(disciplines)f(to)j(b)q(e)f(dev)o (elop)q(ed)f(and)h(tested)g(indep)q(enden)o(tly)l(.)149 1548 y(After)h(eac)o(h)g(program)h(has)g(b)q(een)g(v)m(alidated,)f(an)h (additional)g(mo)q(dule)e(is)h(added)h(to)g(eac)o(h)f(that)149 1608 y(deals)g(with)f(the)g(required)f(in)o(terdisciplinary)e(\(in)o (terzonal\))j(comm)o(uni)o(cation.)223 1669 y(Sev)o(eral)g(mo)q(dels)h(of)i (computation)e(supp)q(ort)i(the)f(MIMD)f(st)o(yle)g(of)h(execution)f(across)i (dis-)149 1729 y(ciplines.)24 b(Whic)o(h)17 b(one)g(is)h(b)q(est)g(dep)q (ends)f(up)q(on)i(the)e(ph)o(ysical)g(problem,)f(solution)h(algorithm,)149 1789 y(hardw)o(are)f(resources,)g(and)g(op)q(erating)g(system)e(supp)q(ort)j (a)o(v)m(ailable.)j(Belo)o(w)15 b(w)o(e)g(presen)o(t)f(three)149 1849 y(mo)q(dels)i(and)g(describ)q(e)g(the)g(functionalit)o(y)f(that)i(is)f (desired)f(from)g(eac)o(h.)223 1909 y(The)h(discussions)g(assume)g(the)g (follo)o(wing)f(de\014nitions:)21 b Ff(N)5 b(P)24 b Fk(equals)16 b(the)g(total)g(n)o(um)o(b)q(er)f(of)149 1970 y(pro)q(cessors)g(used,)f(a)g (group)h Ff(G)714 1977 y Fe(i)742 1970 y Fk(is)f(the)g(set)f(of)h(pro)q (cessors)h(assigned)g(to)f(a)g(particular)f(discipline,)149 2030 y Ff(N)5 b(G)17 b Fk(is)e(the)h(total)g(n)o(um)o(b)q(er)e(of)i(groups,)g (and)h Ff(GS)1061 2037 y Fe(i)1091 2030 y Fk(is)e(the)h(n)o(um)o(b)q(er)e(of) i(pro)q(cessors)h(in)e(group)i Ff(G)1919 2037 y Fe(i)1933 2030 y Fk(.)223 2090 y(In)c(all)f(of)i(the)f(mo)q(dels,)f(it)h(is)g(assumed)f (that)i(the)f(user)g(has)h(the)f(option)h(to)g(con)o(trol)e(the)h(set)h(of) 149 2150 y(ph)o(ysical)g(pro)q(cessors)h(assigned)g(to)g(a)g(group)g(and)g (the)f(data)i(la)o(y)o(out)d(on)i(that)g(set)f(of)h(pro)q(cessors.)149 2278 y Fi(4.1)56 b(Static)18 b(Mo)r(del)149 2370 y Fk(In)g(the)g(static)g(mo) q(del,)f(all)h(resource)g(requiremen)o(ts)d(are)j(kno)o(wn)h(at)f(startup)h (\(i.e.,)e Ff(N)5 b(P)i Fk(,)19 b Ff(N)5 b(G)p Fk(,)149 2431 y(and)18 b Ff(GS)313 2438 y Fe(i)345 2431 y Fk(are)f(all)f(kno)o(wn\).)24 b(F)l(unctional)16 b(requiremen)o(ts)e(are)j(that)g(the)g(n)o(um)o(b)q(er)e (of)j(pro)q(cessors)149 2491 y(assigned)23 b(to)e(eac)o(h)g(discipline)f(is)h (indep)q(enden)o(t)f(of)i(the)f(other)h(disciplines,)e(a)i(di\013eren)o(t)f (exe-)149 2551 y(cutable)e(ma)o(y)f(b)q(e)i(loaded)g(in)o(to)f(eac)o(h)g (group,)i(eac)o(h)e(group)i(has)f(its)f(o)o(wn)h(logical)f(n)o(um)o(b)q (ering,)149 2611 y(global)f(op)q(erations)h(suc)o(h)e(as)h(SUM)f(or)g(MAX)g (apply)g(to)h(the)f(lo)q(cal)g(group,)h(and)g(some)e(mec)o(ha-)149 2671 y(nism)f(is)i(pro)o(vided)e(to)i(establish)f(comm)o(unication)e(b)q(et)o (w)o(een)h(an)i(individual)e(pro)q(cessor)j(in)e(one)149 2731 y(group)i(and)e(an)h(individual)e(pro)q(cessor)i(in)f(another)h(group.)1036 2864 y(7)p eop %%Page: 8 8 bop 223 42 a Fk(The)12 b(logical)g(n)o(um)o(b)q(ering)f(within)h(a)h(group,)h (global)e(op)q(erations)i(applying)f(only)f(to)h(the)f(lo)q(cal)149 102 y(group,)24 b(and)f(di\013eren)o(t)e(executables)g(all)g(supp)q(ort)i (indep)q(enden)o(t)f(dev)o(elopmen)n(t)d(and)k(testing)149 162 y(of)g(disciplines.)36 b(The)22 b(indep)q(enden)o(t)f(assignmen)o(t)g(of) h(pro)q(cessors)h(to)f(disciplines)e(allo)o(ws)i(the)149 222 y(application)f(to)g(b)q(e)f(statically)g(load)h(balanced.)34 b(P)o(oin)o(t)20 b(to)h(p)q(oin)o(t)f(comm)o(unication)e(b)q(et)o(w)o(een)149 282 y(groups)g(allo)o(w)e(coupling)h(information)e(to)i(b)q(e)f(passed)h (directly)e(b)q(et)o(w)o(een)h(the)g(pro)q(cessors)h(that)149 342 y(need)f(to)h(kno)o(w)f(the)g(information.)223 403 y(Groups)j(are)f (neither)f(created)h(nor)h(deleted)e(dynamically)l(.)24 b(Ev)o(erything)18 b(is)g(initialized)e(at)149 463 y(startup)25 b(and)f(con)o(tin)o(ues)f(un)o (til)g(application)g(termination.)42 b(Curren)o(tly)l(,)24 b(there)f(are)h(sev)o(eral)149 523 y(applications)19 b([11)q(,)f(10)q(,)h(3]) f(using)i(this)f(mo)q(del)e(at)j(NASA)d(Ames)g(Researc)o(h)i(Cen)o(ter)f (based)h(on)149 583 y(the)d(in)o(tercub)q(e)g(comm)o(uni)o(cation)e(soft)o(w) o(are)i(dev)o(elop)q(ed)f(for)i(the)f(In)o(tel)e(iPSC/860)k([1].)223 643 y(There)11 b(are)h(sev)o(eral)f(problems)f(with)i(the)g(iPSC/860)h (implem)o(en)o(tation)c(of)j(the)g(static)g(mo)q(del.)149 704 y(Debuggers)21 b(do)f(not)g(w)o(ork)f(for)h(m)o(ultiple)d(indep)q(enden)o(t)h (groups)j(of)f(pro)q(cessors.)32 b(Groups)21 b(are)149 764 y(restricted)11 b(to)i(a)f(p)q(o)o(w)o(er-of-t)o(w)o(o)h(n)o(um)o(b)q(er)e (of)h(pro)q(cessors)h(and)g(only)f(one)g(group)h(ma)o(y)e(b)q(e)h(assigned) 149 824 y(to)21 b(a)f(pro)q(cessor)i(thereb)o(y)d(limiti)o(ng)f(the)i(abilit) o(y)f(to)h(load)h(balance)f(the)f(computation.)33 b(Also,)149 884 y(existing)24 b(batc)o(h)g(job)g(submission)g(soft)o(w)o(are,)h(Net)o(w)o (ork)e(Queuing)h(System)f(\(NQS\),)g(do)h(not)149 944 y(supp)q(ort)18 b(this)e(st)o(yle)f(of)i(computation.)149 1072 y Fi(4.2)56 b(Dynamic)17 b(Mo)r(del)149 1165 y Fk(In)g(the)f(dynamic)f(mo)q(del,)g (groups)i(ma)o(y)e(b)q(e)i(created)f(or)h(deleted)e(dynamically)l(.)20 b Ff(N)5 b(P)24 b Fk(and)17 b Ff(N)5 b(G)149 1225 y Fk(v)m(ary)19 b(dynamically)l(.)24 b Ff(GS)622 1232 y Fe(i)655 1225 y Fk(is)18 b(\014xed)f(at)i(group)g(creation.)26 b(F)l(unctional)18 b(requiremen)o(ts)d (are)j(that)149 1285 y(the)c(n)o(um)o(b)q(er)e(of)i(pro)q(cessors)h(assigned) f(to)h(eac)o(h)e(discipline)f(is)h(indep)q(enden)o(t)g(of)h(the)g(other)f (disci-)149 1345 y(plines,)i(a)i(di\013eren)o(t)e(executable)g(ma)o(y)f(b)q (e)i(loaded)h(in)o(to)e(eac)o(h)h(group,)g(eac)o(h)g(group)h(has)g(its)f(o)o (wn)149 1405 y(logical)g(n)o(um)o(b)q(ering,)e(global)i(op)q(erations)h (apply)f(to)h(the)e(lo)q(cal)h(group,)h(mec)o(hanism)o(s)d(for)i(group)149 1466 y(creation)j(and)f(deletion)g(are)g(needed,)g(and)h(some)e(mec)o(hanism) e(to)k(establish)f(comm)o(unic)o(ation)149 1526 y(b)q(et)o(w)o(een)d(an)h (individual)e(pro)q(cessor)j(in)e(one)g(group)i(and)f(an)g(individual)e(pro)q (cessor)i(in)f(another)149 1586 y(group.)223 1646 y(The)k(reasoning)i(for)f (the)g(functionalit)o(y)e(is)i(the)f(same)g(as)h(in)g(the)f(static)h(mo)q (del)e(with)i(the)149 1706 y(additional)f(requiremen)n(ts)d(of)i(dynamic)f (creation)h(and)g(deletion)g(of)g(groups.)28 b(The)19 b(size)e(of)h(an)149 1766 y(individual)23 b(group)h(is)f(\014xed)g(at)h(group)g(creation.)43 b(If)22 b(a)i(group)h(needs)e(increase)f(its)i(size,)f(a)149 1827 y(new)18 b(group)h(ma)o(y)d(b)q(e)i(created)g(and)g(the)g(relev)m(an)o (t)f(information)g(copied)g(from)g(the)g(old)h(group.)149 1887 y(Then)f(the)g(old)g(group)g(name)f(is)h(reassigned)g(to)g(the)g(new)f(group) i(and)f(the)g(old)g(group)h(deleted.)149 1947 y(The)i(create,)g(cop)o(y)f (and)h(delete)e(approac)o(h)j(to)f(dynamic)e(group)i(size)f(is)h(acceptable)f (b)q(ecause)149 2007 y(disciplines)d(often)h(assume)f(a)h(particular)f(data)i (to)f(pro)q(cessor)h(la)o(y)o(out.)23 b(Adding)16 b(pro)q(cessors)j(to)149 2067 y(the)f(curren)o(t)e(group)j(w)o(ould)e(also)i(require)d(a)i(data)g (reorganization.)26 b(Mo)o(ving)17 b(to)h(a)g(new)f(set)h(of)149 2128 y(pro)q(cessors)k(ma)o(y)c(ease)i(the)g(pro)q(cess.)33 b(This)20 b(form)f(of)h(pro)q(cess)h(migration)e(requires)g(that)h(the)149 2188 y(c)o(hange)d(in)f(lo)q(cation)g(of)h(the)f(group)h(b)q(e)f(transmitted) f(to)i(all)f(groups)h(that)g(need)e(to)i(kno)o(w.)149 2316 y Fi(4.3)56 b(Hybrid)18 b(Mo)r(del)149 2408 y Fk(In)e(the)g(h)o(ybrid)g(mo)q (del,)e(the)i(total)h(n)o(um)o(b)q(er)d(of)j(pro)q(cessors)g(used)g(b)o(y)e (the)h(application,)g Ff(N)5 b(P)i Fk(,)16 b(is)149 2468 y(\014xed)k(at)f (application)h(startup.)31 b(Ho)o(w)o(ev)o(er,)18 b(the)h(n)o(um)o(b)q(er)f (of)i(groups,)h Ff(N)5 b(G)p Fk(,)20 b(and)g(their)f(sizes,)149 2528 y Ff(GS)217 2535 y Fe(i)232 2528 y Fk(,)j(v)m(ary)f(at)g(run)o(time.)33 b(F)l(unctional)21 b(requiremen)o(ts)d(are)j(that)g(the)g(n)o(um)o(b)q(er)e (of)j(pro)q(cessors)149 2589 y(assigned)c(to)g(eac)o(h)e(discipline)f(is)i (indep)q(enden)o(t)f(of)h(other)h(disciplines,)d(a)i(di\013eren)o(t)f (executable)149 2649 y(ma)o(y)k(b)q(e)i(loaded)g(in)o(to)f(eac)o(h)g(group,)i (eac)o(h)e(group)i(has)f(its)f(o)o(wn)h(logical)f(n)o(um)o(b)q(ering,)g (global)149 2709 y(op)q(erations)k(apply)f(to)g(the)f(lo)q(cal)g(group,)j (mec)o(hanisms)21 b(for)j(group)g(creation)g(and)g(deletion)1036 2864 y(8)p eop %%Page: 9 9 bop 149 42 a Fk(are)17 b(needed,)e(and)i(some)f(mec)o(hanism)d(to)k (establish)f(comm)o(unication)e(b)q(et)o(w)o(een)h(an)i(individual)149 102 y(pro)q(cessor)h(in)e(one)g(group)h(and)g(an)g(individual)e(pro)q(cessor) i(in)f(another)h(group.)223 162 y(Ha)o(ving)h(a)h(\014xed)f(n)o(um)o(b)q(er)f (of)i(pro)q(cessors)h(assigned)g(to)f(the)f(application)h(simpli\014es)d (some)149 222 y(problems)d(and)h(creates)g(others.)20 b(Keeping)14 b(the)f(group)i(information)e(consisten)o(t)g(across)i(groups)149 282 y(is)h(simpli\014ed)e(b)q(ecause)j(it)e(is)h(a)h(b)q(ounded)g(problem)d (and)j(tables)f(ma)o(y)f(b)q(e)h(k)o(ept)g(on)g(a)h(pro)q(cessor)149 342 y(basis)e(rather)f(than)h(a)g(group)g(basis.)21 b(Ho)o(w)o(ev)o(er,)12 b(what)j(happ)q(ens)g(when)f(a)h(group)g(is)f(created)f(and)149 403 y(there)19 b(are)g(no)g(idle)f(pro)q(cessors?)30 b(P)o(ossible)19 b(solutions)g(are)g(to)g(either)f(ab)q(ort)i(the)f(application,)149 463 y(inform)d(the)g(application)h(that)g(it)f(is)h(out)g(of)g(pro)q (cessors,)h(or)f(allo)o(w)g(m)o(ultiple)c(groups)18 b(p)q(er)f(pro-)149 523 y(cessor)i(and)h(time)c(slice)i(and)h(share)g(lo)q(cal)g(memory)d(\(mem)o (ory)g(slice\))i(across)h(groups.)30 b(Recall)149 583 y(that)17 b(parallel)f(pro)q(cessors)h(are)f(b)q(eing)h(considered)e(ph)o(ysical)h (memory)d(mac)o(hines.)223 643 y(The)j(most)f(\015exible)g(solution)h(is)g (informing)f(the)h(application)g(that)g(it)g(has)h(run)f(out)h(of)f(idle)149 704 y(pro)q(cessors)24 b(and)e(let)f(it)h(decide)f(if)g(it)h(w)o(an)o(ts)g (to)g(memory)d(slice)i(or)h(ab)q(ort.)40 b(When)22 b(memory)149 764 y(slicing,)16 b(the)g(user)h(should)g(ha)o(v)o(e)f(the)h(option)g(to)g (sp)q(ecify)f(a)i(logical)e(pro)q(cessor)i(la)o(y)o(out)e(and)h(the)149 824 y(set)i(of)g(ph)o(ysical)f(pro)q(cessors)i(that)f(are)g(to)g(b)q(e)g (memory)d(sliced.)28 b(This)19 b(implies)d(the)i(abilit)o(y)g(of)149 884 y(an)f(application)f(to)h(in)o(terrogate)f(the)g(state)g(of)h(an)o(y)f (pro)q(cessor.)223 944 y(With)j(mem)o(ory)e(slicing,)h(it)h(is)g(also)h (desirable)e(to)i(ha)o(v)o(e)e(group)i(migration)e(as)i(an)g(option.)149 1005 y(This)e(is)f(an)h(issue)g(where)f(solution)h(adaptiv)o(e)f(algorithms)f (are)i(used.)25 b(Assume)16 b(that)i(a)g(region)149 1065 y(needs)h (re\014nemen)o(t.)27 b(Then)19 b(a)h(new)f(grid)g(with)g(greater)g (re\014nemen)o(t)d(can)j(b)q(e)h(placed)e(o)o(v)o(er)g(the)149 1125 y(region.)32 b(The)19 b(new)h(grid)f(is)g(treated)h(as)g(a)g(logically)e (separate)i(group.)32 b(In)19 b(other)h(places,)f(the)149 1185 y(grid)k(ma)o(y)d(b)q(e)i(coarsened.)40 b(In)21 b(those)i(places,)g(groups)g (terminate.)37 b(As)22 b(groups)h(terminate,)149 1245 y(pro)q(cessors)c(ma)o (y)c(b)q(ecome)h(idle)g(while)g(other)h(pro)q(cessors)i(are)e(memory)d (slicing.)23 b(Under)16 b(these)149 1306 y(conditions,)g(the)g(user)h(ma)o(y) d(w)o(an)o(t)i(groups)i(to)e(migrate)f(to)i(the)f(idle)f(pro)q(cessors.)223 1366 y(One)d(of)h(the)g(adv)m(an)o(tages)h(of)f(the)f(h)o(ybrid)g(mo)q(del)g (is)g(that)i(it)e(could)g(b)q(e)h(submitted)e(as)j(a)f(batc)o(h)149 1426 y(job.)22 b(The)16 b(resources)h(required)e(are)h(\014xed)g(and)h(sp)q (eci\014able)e(at)i(application)f(instan)o(tiation.)149 1559 y Fi(5)56 b(Conclusion)149 1669 y Fk(Based)22 b(on)f(memory)e(requiremen)n (ts,)h(memory)e(e\016ciency)l(,)i(Amdahl's)f(La)o(w,)k(computational)149 1729 y(requiremen)o(ts,)d(computational)h(e\016ciency)l(,)f(soft)o(w)o(are)i (engineering,)g(and)g(m)o(ultidisci)o(plinary)149 1789 y(\(m)o(ultizonal\))14 b(coupling)i(a)g(computing)f(mo)q(del)g(that)h(supp)q(orts)h(a)g(MIMD)e(st)o (yle)g(of)h(computing)149 1849 y(across)i(disciplines)c(is)i(the)g(most)g (natural.)223 1909 y(Three)c(computational)h(mo)q(dels)f(w)o(ere)g(in)o(tro)q (duced.)20 b(The)13 b(static)g(mo)q(del)e(is)i(curren)o(tly)f(b)q(eing)149 1970 y(used)20 b(at)h(NASA)d(Ames)g(for)i(m)o(ultidiscipli)o(nary)d(and)k(m)o (ultizonal)c(applications.)32 b(W)l(e)20 b(exp)q(ect)149 2030 y(to)k(mo)o(v)o(e)c(to)o(w)o(ards)j(the)g(h)o(ybrid)f(mo)q(del)f(in)h(the)h (next)f(y)o(ear)g(to)h(supp)q(ort)h(solution)f(adaptiv)o(e)149 2090 y(metho)q(ds.)149 2223 y Fi(References)174 2333 y Fk([1])h(E.)15 b(Barszcz.)k(In)o(tercub)q(e)14 b(comm)o(unicati)o(on)f(for)j(the)f (ipsc/860.)21 b(In)15 b Fa(Sc)n(alable)k(High)e(Perfor-)250 2393 y(manc)n(e)h(Computing)g(Confer)n(enc)n(e)g(`92)f(pr)n(o)n(c)n(e)n(e)n (dings)p Fk(,)e(pages)i(307{313,)h(April)d(1992.)174 2495 y([2])24 b(E.)c(Barszcz,)h(T.)g(F.)f(Chan,)i(D.)f(C.)f(Jesp)q(ersen,)i(and)f(R.)g(S.)f (T)l(uminaro.)34 b(Flo52)22 b(on)f(h)o(y-)250 2555 y(p)q(ercub)q(es:)27 b(P)o(erformance)18 b(and)h(mo)q(delling)f(of)i(a)f(m)o(ultigrid)e(euler)h (co)q(de.)30 b Fa(International)250 2615 y(Journal)17 b(of)g(High)h(Sp)n(e)n (e)n(d)f(Computing)p Fk(,)g(1\(3\):481{503,)h(1989.)1036 2864 y(9)p eop %%Page: 10 10 bop 174 42 a Fk([3])24 b(E.)17 b(Barszcz,)g(S.)g(K.)h(W)l(eeratunga,)g(and)g (R.)f(L.)h(Meaking.)25 b(Dynamic)16 b(o)o(v)o(erset)h(grid)h(com-)250 102 y(m)o(unication)c(on)j(distributed)f(memory)d(parallel)j(pro)q(cessors.) 23 b(T)l(o)17 b(b)q(e)g(presen)o(ted)e(at)i(11th)250 162 y(AIAA)d (Computational)i(Fluid)g(Dynamics)e(Conference,)i(Orlando)g(Florida,)g(July)g (1993.)174 264 y([4])24 b(J.)c(A.)g(Benek,)h(P)l(.)f(G.)h(Buning,)g(and)h(J.) e(L.)h(Steger.)35 b(A)20 b(3-d)i(c)o(himera)d(grid)i(em)o(b)q(edding)250 324 y(tec)o(hnique.)k(American)15 b(Institute)j(of)g(Aeronautics)g(and)h (Astronautics)f(pap)q(er)h(85-1523,)250 384 y(July)c(1985.)174 486 y([5])24 b(R.)11 b(F.)g(V)l(an)g(der)h(Wijngaart.)i(E\016cien)o(t)c (implem)o(en)o(tation)f(of)i(a)h(3-dimensional)f(adi)h(metho)q(d)250 546 y(on)22 b(the)f(ipsc/860.)39 b(Submitted)20 b(to)i(Sup)q(erComputing)g (`93",)h(NASA)e(Ames)f(Researc)o(h)250 606 y(Cen)o(ter,)15 b(Mo\013ett)h(Field,)f(CA)h(94035.)174 708 y([6])24 b(C.)f(A.)f(F)l(elippa)h (and)g(T.)h(L.)f(Geers.)42 b(P)o(artitioned)23 b(analysis)g(for)h(coupled)e (mec)o(hanical)250 768 y(systems.)31 b(College)20 b(of)g(Engineering)g(and)h (Applied)e(Science,)g(Univ)o(ersit)o(y)e(of)k(Colorado,)250 828 y(Boulder,)15 b(CO)h(80309.)174 930 y([7])24 b(R.)19 b(W.)g(Ho)q(c)o (kney)g(and)h(C.)g(R.)f(Jesshop)q(e.)32 b Fa(Par)n(al)r(lel)22 b(Coputers)p Fk(.)31 b(Adam)19 b(Hilger,)g(Bristol)250 990 y(and)e(Philadelphia,)e(2nd)h(edition,)g(1988.)174 1092 y([8])24 b(R.)16 b(L.)i(Meakin.)23 b(A)16 b(new)i(metho)q(d)e(for)i(establishing)f(in) o(ter-grid)f(comm)o(unic)o(ation)f(among)250 1152 y(systems)k(of)i(o)o(v)o (erset)e(grids.)35 b(American)18 b(Institute)i(of)g(Aeronautics)h(and)g (Astronautics)250 1212 y(pap)q(er)c(91-1586,)h(June)e(1991.)174 1314 y([9])24 b(K.)e(C.)h(P)o(ark)h(and)f(C.)g(A.)g(F)l(elippa.)41 b(P)o(artitioned)23 b(analysis)g(of)h(coupled)e(systems.)41 b(In)250 1374 y(T.)18 b(Belytsc)o(hk)o(o)f(and)i(T.)f(J.)h(R.)f(Hughes,)h (editors,)g Fa(Computational)h(Metho)n(ds)f(for)g(T)l(r)n(an-)250 1434 y(sient)d(A)o(nalysis)p Fk(,)f(c)o(hapter)f(3,)h(pages)g(157{219.)i (Elsevier)d(Scienc)f(Publishers)h(B.V.,)f(North)250 1494 y(Holland,)i (Amsterdam,)e(New)j(Y)l(ork,)f(and)i(Oxford,)f(1983.)149 1596 y([10])25 b(E.)14 b(Pramono)g(and)i(S.)e(K.)g(W)l(eeratunga.)19 b(Aero)q(elastic)13 b(computations)h(for)h(wings)g(through)250 1656 y(direct)27 b(coupling)h(on)h(distributed-mem)o(ory)d(mim)o(d)g (parallel)h(computers.)56 b(W)l(ork)28 b(in)250 1716 y(progress)20 b(at)f(the)g(Applied)f(Researc)o(h)h(Branc)o(h,)g(NAS)f(System)g(Division,)h (NASA)f(Ames)250 1777 y(Researc)o(h)d(Cen)o(ter,)g(Mo\013ett)i(Field,)d(CA)i (94035.)149 1878 y([11])25 b(J.S.)13 b(Ry)o(an)g(and)h(S.)g(K.)f(W)l (eeratunga.)k(P)o(arallel)c(computation)g(of)h(3-d)g(na)o(vier-stok)o(es)f (\015o)o(w-)250 1939 y(\014eld)f(for)i(sup)q(ersonic)f(v)o(ehicles.)h (American)d(Institute)i(of)g(Aeronautics)g(and)h(Astronautics)250 1999 y(pap)q(er)j(93-0064,)h(Jan)o(uary)f(1993.)149 2100 y([12])25 b(Thinking)c(Mac)o(hines)g(Corp)q(oration,)j(Cam)o(bridge,)d(MA.)36 b Fa(CM)22 b(F)l(ortr)n(an)g(Pr)n(o)n(gr)n(amming)250 2161 y(Guide)p Fk(,)16 b(v)o(ersion)f(1.0)i(edition,)e(Jan)o(uary)i(1991.)1024 2864 y(10)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-comm@CS.UTK.EDU Sun Mar 28 16:01:54 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA16813; Sun, 28 Mar 93 16:01:54 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28455; Sun, 28 Mar 93 15:59:48 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sun, 28 Mar 1993 15:59:47 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28447; Sun, 28 Mar 93 15:59:45 -0500 Date: Sun, 28 Mar 93 21:59:29 BST Message-Id: <23593.9303282059@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Multidisciplinary Position Paper To: barszcz@nas.nasa.gov (Eric Barszcz), mpi-comm@cs.utk.edu In-Reply-To: Eric Barszcz's message of Fri, 26 Mar 93 16:40:19 -0800 Reply-To: lyndon@epcc.ed.ac.uk Hi Eric Thanks for mailing this. It's a good example of a "parallel module graph" program. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Mon Mar 29 09:46:10 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12851; Mon, 29 Mar 93 09:46:10 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22553; Mon, 29 Mar 93 09:43:12 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 29 Mar 1993 09:43:11 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22526; Mon, 29 Mar 93 09:43:09 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA16002; Mon, 29 Mar 1993 09:43:08 -0500 Date: Mon, 29 Mar 1993 09:43:08 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9303291443.AA16002@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Revised agenda for MPI meeting Provisional Agenda for MPI Meeting, March 31-April 2, 1993 Wednesday 1:30-3:00 Intro to context sub-committee proposals - Skjellum (10 mins.) Proposal VII - Littlefield and Clarke present (30 mins.) Discussion on VII - lead by Littlefield and Clarke (15 mins.) Proposal I - Snir (30 mins.) Discussion on I - lead by Snir (15 mins.) Proposals III and VIII - Skjellum presents (30 mins.) Discussion on III and VIII - lead by Skjellum and Sears (15 mins.) Overall discussion, Recommendations, and Ranking poll (35 mins.) 3:30-6:00 Discussion of Snir, Gropp, Lusk point-to-point proposal (everyone) (Snir) 6:00-7:30 Unofficial dinner break 7:30-10:30 Break up for subcommittee meetings Thursday 9:00-12:00 Continued discussion of Snir, Gropp, Lusk point-to-point proposal (Snir) OR Discussion of Snir & Geist collection communication proposal (Otto) 12:00-1:30 Lunch (provided) 1:30-3:00 Discussion of Snir & Geist collection communication proposal (Otto) 3:00-6:00 Full group meeting for presentation of process topology subcommittee ideas and proposals. (Hempel) 6:00-8:00 Dinner (attendees pay, but hotel provides transport to area restaurant) 8:00-10:00 Continued informal subcommittee meetings if necessary Friday 9:00-11:00 Full group meeting with the intent of taking binding votes on point-to-point and collective communication proposals, or sending proposals back to subcommittees for revision. (Snir) 11:00-12:00 Full group meeting for defining timetable for producing MPI (or subset) by deadline in July. (Dongarra) From owner-mpi-comm@CS.UTK.EDU Mon Mar 29 18:14:27 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27088; Mon, 29 Mar 93 18:14:27 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20945; Mon, 29 Mar 93 18:02:00 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 29 Mar 1993 18:01:59 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20937; Mon, 29 Mar 93 18:01:56 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA03153; Mon, 29 Mar 93 23:01:48 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA01225; Mon, 29 Mar 93 16:00:29 MST Date: Mon, 29 Mar 93 16:00:29 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303292300.AA01225@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Resend of unofficial "opinion survey" Hi everyone, Some folks have requested that the unofficial "MPI Opinion Survey" be distributed again. Here it is. Please ignore if you don't need to see it again! :-) Tom ----------------------------------------------------------------------------- Hi all, Steve Huss-Lederman writes: > I would like to propose that we have a meeting of all participants at > the start of the next meeting to settle the issue of what MPI-1 will > cover. Several mail messages have expressed concern that we cannot > conclude by June if we try to incorporate everything that everyone > wants. Since the scope and time frame will effect most of the other > discussions I think it would be best to begin with this. My only > concern is that it will take too long to decide the global picture. > I would hope it could be done in 1-2 hours and help to refocus the > group. I fully support this idea. Here's a proposal for an opinion survey that could shorten and focus the discussion. This proposal originated with Jon Flower and has had input from Ian Glendinning, Steve Huss-Lederman, Leslie Hart, myself, and others. If you think this is a good idea, please complete the survey and return to me at: hender@fsl.noaa.gov If you think this is a bad idea, please say so! We will collect responses and distribute a summary via email before the next meeting if there is sufficient interest. Tom Henderson ----------------------------------------------------------------------------- MPI PRIORITIES OPINION SURVEY Several MPI committee members have expressed concern about the current and future directions of MPI. The MPI committee began discussion using an existing document (Dongarra et al) as a starting point. This document contains background rationale and a statement of direction for the MPI standardization effort. Since these issues were consistent with the original MPI proposal, they have not recently been discussed in detail by the MPIF. The current MPI proposals are very different from the original MPI document. As a result, we feel that the original rationale and direction need to be re-examined. In particular we need to prioritize and limit the issues that MPI will try to resolve so that o the initial MPI standard document ("MPI-1") is delivered in a timely fashion and o the functions that it defines can (and will) be implemented on major platforms within a reasonable time frame. To get discussion of these issues started, we propose an informal, email "opinion survey". The objective of the survey is to find out which MPI features are most important. High-priority features would be included in MPI-1. Lower-priority features might be deferred to a later version of MPI. Some features might never be included. The compiled results of this survey could be used as a basis for discussion at the next MPI meeting. *** This survey is in no way binding and is a completely unofficial "straw poll" arranged by its authors for their own personal information. It is not intended to distract attention from the current proposals. Any data collected will be made available to the group at large if requested. *** Following is a ballot containing possible MPI features and goals. Please rate each item (or sub-item) using the following rating scheme: 0 This feature should not appear in any version of MPI ever. 1-2 MPIF should consider including this feature in some future version of MPI (after MPI-1). 3-4 This feature should be included as an Annex to MPI-1 for possible inclusion in a future version of MPI. 5-6 This feature should be included as an Annex to MPI-1 for definite inclusion in a future version of MPI. 7-10 This feature should be included in MPI-1. In this context an "Annex" is a part of the standard document that will not be required by a conforming implementation of MPI. Rather it should be taken as a recommendation by the MPIF to vendors for the way in which particular features should be implemented, if they are implemented at all. This way features that are not usefully implementable on particular hardware can be eliminated while machines that do support these features do so in a portable manner. When replying to the survey, please use the response form attached to the end of this message. (This will make compilation of the results a lot less painful!) :-) +-------------------------------------------------------+ | POSSIBLE FEATURES FOR INITIAL AND FUTURE MPI VERSIONS | +-------------------------------------------------------+ Ordering is for convenience only. This list is not meant to suggest any priority. GENERAL FEATURES 1) Routines for static process creation: A) Same program in every node. B) Host-node programming model in which there is one "special node". C) Static model but with potentially different programs in different groups of nodes. D) Static model with a way to create more than one process per processor. (Heavy duty processes, not threads). 2) Routines for identification of statically created processes. 3) Associated environmental inquiry routines. 4) Some way of avoiding message tag conflicts between independently developed libraries and user code (contexts). 5) Heterogeneous communication. A) All communication routines should support heterogeneous communication. B) A special set of communication routines should support heterogeneous communication. 6) Creation/destruction of "static" groups (processes cannot join/leave an existing group). Groups are used to limit the scope of collective operations. A) No logical process topology is associated with a group. B) Grid/toroidal process topology is associated with a group at the time the group is created. (The default "ALL" group has a default topology.) C) General graph topology is also supported. D) Provide access to all relevant information (via topological environmental inquiry routines) to allow users to define custom topologies. 7) Support for communication between processes in different groups without reference to the "ALL" group. 8) Dynamic groups (processes can join/leave an existing group while the total number of processes remains static.) 9) Dynamic process creation and identification. 10) Additional environmental inquiry and "setting" routines for performance optimization. A) Inquire/suggest system buffering resource allocation. B) Inquire relative node performance. C) Inquire relative interconnect performance. 11) Termination of programs. A) Kill all others nodes. B) Wait for all the other nodes to complete. C) Automatically de-allocate resources. 12) Standard organization. A) "Multi-level" organization of routines (users may use simple features without having to understand all of MPI). B) Simple "core" standard with (optional) Annexes. C) Simple "core" standard without Annexes. D) Should an MPI "subset" be defined that can be quickly implemented as a layer on top of existing systems? (This question was inspired by section 1.5 of Bill and Rusty's "A Test Implementation of the MPI Draft Message-Passing Standard", ANL-92/47.) 13) Should there be future version(s) of MPI (MPI-2, etc.) (yes/no)? POINT-TO-POINT COMMUNICATION 14) Blocking send and receive. A) Contiguous data. B) Strided data. C) Buffer-descriptor (multiple data locations, multiple data types). 15) Non-blocking receive. A) Contiguous data. B) Strided data. C) Buffer-descriptor (multiple data locations, multiple data types). D) Check for completion (status()). E) Wait for completion (wait()). F) Cancel operation (cancel()). 16) Non-blocking send. A) Contiguous data. B) Strided data. C) Buffer-descriptor (multiple data locations, multiple data types). D) Check for completion (status()). E) Wait for completion (wait()). F) Cancel operation (cancel()). 17) Low-level functionality (from Marc Snir's proposal) visible to users. A) Initialize. B) Start. C) Complete. D) Free. 18) "SECURE" versions of send and receive as proposed by Lyndon Clarke on 3/3/93. (Correct programs using SECURE routines are guaranteed to work regardless of system buffering resources.) 19) Ready-send. 20) Ready-receive. 21) Message handlers (a la iNTEL hrecv, TMC active messages, Express exhandle). A) Simple definition sufficient to support anticipated MPI requirements and make a self-consistent system. B) More advanced routines for remote procedure calls, etc... 22) Additional point-to-point primitives. A) Exchange between two processors. B) "Shift-exchange": read from one, send to another (superset of "A"). C) Contiguous data. D) Strided data. E) Buffer-descriptor (multiple data locations, multiple data types). F) Non-blocking versions? G) Check for completion (status()). H) Wait for completion (wait()). I) Cancel operation (cancel()). COLLECTIVE OPERATIONS 23) Blocking collective communication. A) Broadcast. B) Concatenate. C) Barrier. D) Sum. E) Extrema (max, min). F) Average. G) Sum of squared differences (error). H) Should strided data be allowed for these operations? I) Should buffer-descriptor be allowed for these operations? 24) "User-defined" blocking collective operations (Express "excombine()" style). 25) Non-blocking collective communication, collective operations, and user-defined operations (if this is desired, detailed ranking can be done later). 26) If any non-blocking collective operations are supported, what should completion/cancel options be? A) Check for completion (status()). B) Wait for completion (wait()). C) "Multiple cancel" operation (cancel()). 27) Should status()/wait()/cancel() syntax be consistent across all types of non-blocking operations? OTHER COMMUNICATION OPERATIONS 28) Probe (and associated "precv()" routines, if necessary). I/O 29) Standard I/O is supported in the standard fashion for the language on one process (no file I/O). This process can be identified by an environmental inquiry routine. (This is the current proposal.) 30) Serial I/O is supported in the standard fashion for the language on one process. This process can be identified by an environmental inquiry routine. (Rik Littlefield's proposal.) 31) Multiplexed I/O similar to that currently provided in systems such as CMMD (TMC), VERTEX (nCUBE) and Express (ParaSoft). Basically a way of defining the behavior of multi-node I/O statements which interact with a sequential device. No special considerations for parallel I/O channels. 32) Blocking parallel I/O. Intended to take advantage of parallel I/O channels, where available. 33) Non-blocking parallel I/O. Intended to take advantage of parallel I/O channels, where available. LANGUAGE BINDINGS 34) Ada 35) C 36) C++ 37) Fortran 77 38) Fortran 90 39) Other IMPLEMENTATION PERIOD There will be a delay between the generation of the standard and its implementation on various platforms. If this delay is too long users will continue to rely more and more heavily on vendor routines and specialized calls and MPI will become less and less attractive, particularly if new technology such as Fortran 90/HPF becomes effective. 40) What is the maximum delay that you think MPI can stand and still succeed (please feel free to comment if you wish)? ---------------------------------------------------------------------------- SURVEY RESPONSE FORM 1) A) B) C) D) 2) 3) 4) 5) A) B) 6) A) B) C) D) 7) 8) 9) 10) A) B) C) 11) A) B) C) 12) A) B) C) D) 13) (yes/no) 14) A) B) C) 15) A) B) C) D) E) F) 16) A) B) C) D) E) F) 17) A) B) C) D) 18) 19) 20) 21) A) B) 22) A) B) C) D) E) F) G) H) I) 23) A) B) C) D) E) F) G) H) I) 24) 25) 26) A) B) C) 27) 28) 29) 30) 31) 32) 33) 34) 35) 36) 37) 38) 39) 40) (free form): Comments? From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 00:51:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02775; Tue, 30 Mar 93 00:51:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06540; Tue, 30 Mar 93 00:49:03 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 00:49:02 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Mail.Think.COM by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06532; Tue, 30 Mar 93 00:49:01 -0500 Received: from Basselope.Think.COM by mail.think.com; Tue, 30 Mar 93 00:48:58 -0500 From: Alan Mainwaring Received: by basselope.think.com (4.1/Think-1.2) id AA15508; Tue, 30 Mar 93 00:48:57 EST Date: Tue, 30 Mar 93 00:48:57 EST Message-Id: <9303300548.AA15508@basselope.think.com> To: mpi-comm@cs.utk.edu In-Reply-To: <9303291443.AA16002@rios2.epm.ornl.gov>, "David Walker's message of Mon, 29 Mar 1993 09:43:08 -0500" Subject: Revised agenda for MPI meeting Date: Mon, 29 Mar 1993 09:43:08 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Provisional Agenda for MPI Meeting, March 31-April 2, 1993 Wednesday 1:30-3:00 Intro to context sub-committee proposals - Skjellum (10 mins.) Proposal VII - Littlefield and Clarke present (30 mins.) Discussion on VII - lead by Littlefield and Clarke (15 mins.) Proposal I - Snir (30 mins.) Discussion on I - lead by Snir (15 mins.) Proposals III and VIII - Skjellum presents (30 mins.) Discussion on III and VIII - lead by Skjellum and Sears (15 mins.) Overall discussion, Recommendations, and Ranking poll (35 mins.) 3:30-6:00 Discussion of Snir, Gropp, Lusk point-to-point proposal (everyone) (Snir) 6:00-7:30 Unofficial dinner break 7:30-10:30 Break up for subcommittee meetings Thursday 9:00-12:00 Continued discussion of Snir, Gropp, Lusk point-to-point proposal (Snir) OR Discussion of Snir & Geist collection communication proposal (Otto) 12:00-1:30 Lunch (provided) 1:30-3:00 Discussion of Snir & Geist collection communication proposal (Otto) 3:00-6:00 Full group meeting for presentation of process topology subcommittee ideas and proposals. (Hempel) 6:00-8:00 Dinner (attendees pay, but hotel provides transport to area restaurant) 8:00-10:00 Continued informal subcommittee meetings if necessary Friday 9:00-11:00 Full group meeting with the intent of taking binding votes on point-to-point and collective communication proposals, or sending proposals back to subcommittees for revision. (Snir) 11:00-12:00 Full group meeting for defining timetable for producing MPI (or subset) by deadline in July. (Dongarra) After reviewing this schedule, we believe that the last item should be moved so that it is the first order of business at this meeting. If it is a goal for vendors to quickly implement and release MPI, then, it is imperative that we understand what functionality must be included in MPI1, and what functionality may be deferred to subsequent versions. Without this framework we seriously doubt that this process will yield a specification, against which vendors can implement production software this year. Respectfully, Alan and Moose From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 01:16:20 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03507; Tue, 30 Mar 93 01:16:20 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA07612; Tue, 30 Mar 93 01:15:09 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 01:15:08 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ssd.intel.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA07555; Tue, 30 Mar 93 01:15:05 -0500 Received: from tualatin.SSD.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA11172; Mon, 29 Mar 93 22:15:02 PST Date: Mon, 29 Mar 93 22:15:02 PST Message-Id: <9303300615.AA11172@SSD.intel.com> Received: by tualatin.SSD.intel.com (4.1/SMI-4.0) id AA15164; Mon, 29 Mar 93 22:14:59 PST From: Bob Knighten Sender: knighten@SSD.intel.com To: mpi-comm@cs.utk.edu Subject: Re: Revised agenda for MPI meeting In-Reply-To: <9303300548.AA15508@basselope.think.com> References: <9303291443.AA16002@rios2.epm.ornl.gov> <9303300548.AA15508@basselope.think.com> Reply-To: knighten@SSD.intel.com (Bob Knighten) Alan Mainwaring writes: > Date: Mon, 29 Mar 1993 09:43:08 -0500 > From: walker@rios2.epm.ornl.gov (David Walker) > > > Provisional Agenda for MPI Meeting, March 31-April 2, 1993 > > . . . > > > After reviewing this schedule, we believe that the last item should be > moved so that it is the first order of business at this meeting. > > If it is a goal for vendors to quickly implement and release MPI, then, it > is imperative that we understand what functionality must be included in MPI1, > and what functionality may be deferred to subsequent versions. Without this > framework we seriously doubt that this process will yield a specification, > against which vendors can implement production software this year. > > Respectfully, > > Alan and Moose > I strongly support this re-ordering of the agenda. -- Bob Robert L. Knighten | knighten@ssd.intel.com Intel Supercomputer Systems Division | 15201 N.W. Greenbrier Pkwy. | (503) 629-4315 Beaverton, Oregon 97006 | (503) 629-9147 [FAX] From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 10:35:53 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12387; Tue, 30 Mar 93 10:35:53 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA08221; Tue, 30 Mar 93 10:33:28 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 10:33:28 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from super.super.org by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA08206; Tue, 30 Mar 93 10:33:26 -0500 Received: from b125.super.org by super.super.org (4.1/SMI-4.1) id AA01022; Tue, 30 Mar 93 10:33:20 EST Received: by b125.super.org (4.1/SMI-4.1) id AA03597; Tue, 30 Mar 93 10:33:19 EST Date: Tue, 30 Mar 93 10:33:19 EST From: lederman@b125.super.org (Steve Huss-Lederman) Message-Id: <9303301533.AA03597@b125.super.org> To: mpi-comm@cs.utk.edu Subject: Re: Revised agenda for MPI meeting In response to the recent message by Alan and Moose, I want to reiterate that I support an early discussion of the goals and timetable for MPI. This will allow the general framework to be set when we decide on how to deal with specific proposals. Given the fact that some of the MPI members are already on travel to the meeting, I suspect that this change in agenda would have to be dealt with at the start of the meeting tomorrow. Steve From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 10:42:09 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12524; Tue, 30 Mar 93 10:42:09 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09326; Tue, 30 Mar 93 10:40:32 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 10:40:30 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from cs.sandia.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09303; Tue, 30 Mar 93 10:40:28 -0500 Received: from panther.cs.sandia.gov by cs.sandia.gov (4.1/SMI-4.1) id AA19731; Tue, 30 Mar 93 08:40:24 MST Received: by panther.cs.sandia.gov (Smail3.1.28.1 #1) id m0ndiQJ-0016bbC; Tue, 30 Mar 93 08:40 MST Message-Id: Date: Tue, 30 Mar 93 08:40 MST From: srwheat@cs.sandia.gov (Stephen R. Wheat) To: mpi-comm@cs.utk.edu Subject: Re: Revised agenda for MPI meeting I must disagree with the suggestion to change the schedule. Without any agreement to even basic, fundamental concepts how are we to decide what will go into MPI1 and what won't. I do indeed understand the desire to know the scope of MPI1, but I really believe the scope depends on an agreed method to achieve contexts, groups, etc. It is those definitions that will define how much work a given set of MPI routines will require. Stephen From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 10:48:58 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12768; Tue, 30 Mar 93 10:48:58 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09919; Tue, 30 Mar 93 10:47:25 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 10:47:23 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09910; Tue, 30 Mar 93 10:47:23 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA19794; Tue, 30 Mar 1993 10:47:22 -0500 Date: Tue, 30 Mar 1993 10:47:22 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9303301547.AA19794@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Revised agenda for MPI meeting I don't think it's a good idea to change the agenda. There seems little point on talking aboutwhat should be in MPI until everyone knows what has been proposed, particularly for contexts. Setting a timetable will be very difficulty if we can't largely agree on point-to-point, collective communication, and contexts, which is why these topics come first. David From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 10:51:19 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12832; Tue, 30 Mar 93 10:51:19 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA10133; Tue, 30 Mar 93 10:49:36 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 10:49:34 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA10095; Tue, 30 Mar 93 10:49:32 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AB04355; Tue, 30 Mar 93 15:49:27 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA02302; Tue, 30 Mar 93 08:48:08 MST Date: Tue, 30 Mar 93 08:48:08 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303301548.AA02302@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: Revised agenda for MPI meeting Alan Mainwaring writes, > After reviewing this schedule, we believe that the last item should be > moved so that it is the first order of business at this meeting. > > If it is a goal for vendors to quickly implement and release MPI, then, it > is imperative that we understand what functionality must be included in MPI1, > and what functionality may be deferred to subsequent versions. Without this > framework we seriously doubt that this process will yield a specification, > against which vendors can implement production software this year. > > Respectfully, > > Alan and Moose I agree. It would be very helpful if vendors would speak frankly about how long implementation might take during this discussion (wishful thinking? :-). Tom Henderson NOAA Forecast Systems Laboratory hender@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 11:45:20 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14240; Tue, 30 Mar 93 11:45:20 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12714; Tue, 30 Mar 93 11:43:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 11:43:40 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12705; Tue, 30 Mar 93 11:43:38 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA04547; Tue, 30 Mar 93 16:43:34 GMT Received: by nipmuc.fsl.noaa.gov (4.1/SMI-4.1) id AA08559; Tue, 30 Mar 93 09:46:14 MST Date: Tue, 30 Mar 93 09:46:14 MST From: hart@nipmuc.fsl.noaa.gov (Leslie Hart) Message-Id: <9303301646.AA08559@nipmuc.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: Revised agenda for MPI meeting I strongly agree with Steve (see below) on both points. I think that we need to discuss MPI priorities early at the meeting (definitely not the last agenda item!). I agree also since it is likely that a number of people are already traveling, the agenda itself should be the first agenda item. Regards, Leslie Hart (hart@fsl.noaa.gov) ----- Begin Included Message ----- From: lederman@b125.super.org (Steve Huss-Lederman) In response to the recent message by Alan and Moose, I want to reiterate that I support an early discussion of the goals and timetable for MPI. This will allow the general framework to be set when we decide on how to deal with specific proposals. Given the fact that some of the MPI members are already on travel to the meeting, I suspect that this change in agenda would have to be dealt with at the start of the meeting tomorrow. Steve ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 13:55:34 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17414; Tue, 30 Mar 93 13:55:34 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19024; Tue, 30 Mar 93 13:53:52 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 13:53:51 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19016; Tue, 30 Mar 93 13:53:50 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA14792; Tue, 30 Mar 93 12:43:16 CST Date: Tue, 30 Mar 93 12:43:16 CST From: Tony Skjellum Message-Id: <9303301843.AA14792@Aurora.CS.MsState.Edu> To: amm@Think.COM, mpi-comm@cs.utk.edu Subject: Re: Revised agenda for MPI meeting I have also asked David to make this change. I repeat that this step is important, and I would not mind being delayed in my context group's presentations, to make room for said discussion. Please note, however, contexts must finish by noon on Thursday, if the agenda is changed, because I have to leave (unexpectedly) at 1pm on Thursday. . . . . . . . . . . "There is no lifeguard at the gene pool." - C. H. Baldwin Anthony Skjellum, MSU/ERC, (601)325-8435; FAX: 325-8997; tony@cs.msstate.edu From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 16:15:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20542; Tue, 30 Mar 93 16:15:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26319; Tue, 30 Mar 93 16:13:22 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 16:13:20 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from cs.sandia.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26310; Tue, 30 Mar 93 16:13:19 -0500 Received: from panther.cs.sandia.gov by cs.sandia.gov (4.1/SMI-4.1) id AA27775; Tue, 30 Mar 93 14:13:17 MST Received: by panther.cs.sandia.gov (Smail3.1.28.1 #1) id m0ndncS-0016bbC; Tue, 30 Mar 93 14:13 MST Message-Id: Date: Tue, 30 Mar 93 14:13 MST From: srwheat@cs.sandia.gov (Stephen R. Wheat) To: mpi-comm@cs.utk.edu Subject: Re: Revised agenda for MPI meeting I too have other pressing matters that were neatly integrated with the original schedule. I must leave Wed at 4pm, to go elsewhere; I will be back Thursday night. I have used the agenda for its purpose, to make plans about. These plans are worthless if I cannot be at the context discussion. I know this is my own problem for using the agenda for the purposes of what an agenda is for. Furthermore, I repeat that I believe it to be fruitless to discuss the MPI1 contents without agreement on context and group. Stephen From owner-mpi-comm@CS.UTK.EDU Tue Apr 6 10:39:59 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18806; Tue, 6 Apr 93 10:39:59 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23928; Tue, 6 Apr 93 10:38:00 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 6 Apr 1993 10:37:59 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from super.super.org by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23920; Tue, 6 Apr 93 10:37:58 -0400 Received: from b125.super.org by super.super.org (4.1/SMI-4.1) id AA28082; Tue, 6 Apr 93 10:37:56 EDT Received: by b125.super.org (4.1/SMI-4.1) id AA06018; Tue, 6 Apr 93 10:37:56 EDT Date: Tue, 6 Apr 93 10:37:56 EDT From: lederman@b125.super.org (Steve Huss-Lederman) Message-Id: <9304061437.AA06018@b125.super.org> To: mpi-comm@cs.utk.edu Subject: subset subcommittee Hello MPIers, At the last MPI meeting, a group met to discuss the possibility of creating a recommended subset of MPI for initial implementation. The idea was presented to the whole group and there were no objections. I have volunteered to coordinate this effort. At the current time I am seeking input to the process and people interested in joining/monitoring this new subcommittee (subset subcommittee for lack of a better name). Send me e-mail (lederman@super.org) to sign up (and suggest a better name :-). At the end of this week I will send all the names of people to David to create a new mail alias. When it is formed you will receive an initial message. You can, of course, join at a later time. Steve From owner-mpi-comm@CS.UTK.EDU Tue Apr 6 13:30:00 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22321; Tue, 6 Apr 93 13:30:00 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02738; Tue, 6 Apr 93 13:28:41 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 6 Apr 1993 13:28:40 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02729; Tue, 6 Apr 93 13:28:38 -0400 Date: Tue, 6 Apr 93 18:28:33 BST Message-Id: <754.9304061728@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: interrupt driven To: mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-pt2pt@cs.utk.edu Dear MPI Colleagues We have, reasonably in my opinion, decided not to provide interrupt driven receive/send (as in Intel hrecv/hsend) in MPI, or active messages. I have heard, can't recall from whom, that this carries the consequence that we should not define any feature of MPI which asks for the implementor of MPI to make use of interrupt driven receives, or active messages. This seems to me to be a bogus argument. In my mind the primary reason for not providing hrecv/hsend type facilities is that it is very difficult to standardise these facilities across different operating systems, particularly in terms of what the handler procedure is and is not allowed to do (e.g., can it do I/O, can it use MPI, ...). It seems to me clear that on many (all) machines of interest the system itself will be making use of similar facilities. Who would write MPI for CM-5 without using active messages? It seems to me that for example non blocking communications really do ask for use of interrupt/active messages. Is there agreement, disagreement, or what on this point? Please do let me know. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Wed Apr 7 13:50:56 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18878; Wed, 7 Apr 93 13:50:56 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15424; Wed, 7 Apr 93 13:49:23 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 7 Apr 1993 13:49:22 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15367; Wed, 7 Apr 93 13:48:34 -0400 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 7 Apr 93 10:48 PDT Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA06918; Wed, 7 Apr 93 10:46:30 PDT Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA17058; Wed, 7 Apr 93 10:46:26 PDT Date: Wed, 7 Apr 93 10:46:26 PDT From: rj_littlefield@pnlg.pnl.gov Subject: Re: interrupt driven To: lyndon@epcc.ed.ac.uk, mpi-comm@cs.utk.edu Cc: d39135@carbon.pnl.gov, mpi-pt2pt@cs.utk.edu Message-Id: <9304071746.AA17058@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu, mpi-comm@cs.utk.edu Lyndon Clarke writes: > We have, reasonably in my opinion, decided not to provide interrupt > driven receive/send (as in Intel hrecv/hsend) in MPI, or active > messages. > > I have heard, can't recall from whom, that this carries the consequence > that we should not define any feature of MPI which asks for the > implementor of MPI to make use of interrupt driven receives, or active > messages. This seems to me to be a bogus argument. I (Rik Littlefield) have regularly made arguments that are close enough to this to be worth clarifying or defending. Think of the MPI specification as being divided into two parts: 1) things you can implement with portable code, and 2) things you can't. My bias is to make the POTENTIALLY portable part of MPI as large as possible. (I emphasize "potentially" to make it clear that I am talking about interface definitions. I encourage implementors to do whatever they want internally.) As an example, consider groups and collective communication. All other factors being equal, I would strongly prefer a specification that permitted groups and collective communication, including group formation and disbanding, to be implemented using just the standardized MPI point-to-point capabilities. This sort of layerability potentially has several good effects. Three of these are 1) earlier widespread availability via porting, 2) better focusing of vendor effort on tuning the point-to-point layer, and 3) the capability to create portable MPI-plug-compatible implementations that are tuned for different requirements, e.g., debugging versus performance. Of course, all other factors are not equal. Strict layerability also has some potentially bad effects. Two of these (depending on where you draw the line between layers) are 1) the loss of integrated syntax (e.g., being able to use rank directly in the point-to-point calls), and 2) loss of functionality that is not supported by lower layers (e.g., asynchronous name servers). Whether the good outweighs the bad is a matter of opinion. The applications that I deal with now and anticipate dealing with over the lifetime of MPI-1 can be characterized as follows: . sensitive to performance, . intended to run on lots of platforms, . not sensitive to syntax, and . not needing name server support. Applications like these are best served by layerability. Others' mileage may differ. Lyndon continues: > In my mind the primary reason for not providing hrecv/hsend type > facilities is that it is very difficult to standardise these facilities > across different operating systems, particularly in terms of what the > handler procedure is and is not allowed to do (e.g., can it do I/O, can > it use MPI, ...). It seems to me clear that on many (all) machines of > interest the system itself will be making use of similar facilities. > Who would write MPI for CM-5 without using active messages? It seems to > me that for example non blocking communications really do ask for use of > interrupt/active messages. > > Is there agreement, disagreement, or what on this point? Please do let > me know. I agree that it is important to distinguish between exporting a standardized hrecv/hsend and and exporting a standardized asynchronous name server interface. The former seems hopeless; the latter is doable, but at the cost of potentially detracting from other features such as performance and prompt widespread availability. Thus, I do not agree that it is bogus to argue against requiring asynchronous servers et.al. in MPI-1. --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-comm@CS.UTK.EDU Wed Apr 7 14:29:11 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20295; Wed, 7 Apr 93 14:29:11 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17859; Wed, 7 Apr 93 14:27:46 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 7 Apr 1993 14:27:44 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from arthur.cs.purdue.edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17851; Wed, 7 Apr 93 14:27:43 -0400 Received: from dragomirna.cs.purdue.edu by arthur.cs.purdue.edu (5.65c/PURDUE_CS-1.2) id ; Wed, 7 Apr 1993 13:27:40 -0500 Received: by dragomirna.cs.purdue.edu (5.65c/PURDUE_CS-1.2) id ; Wed, 7 Apr 1993 13:27:39 -0500 Date: Wed, 7 Apr 1993 13:27:39 -0500 From: dcm@cs.purdue.edu (Dan C Marinescu) Message-Id: <199304071827.AA28416@dragomirna.cs.purdue.edu> To: dcm@cs.purdue.edu, lyndon@epcc.ed.ac.uk, mpi-comm@cs.utk.edu Subject: interrupt driven messages Lyndon Clarke writes: > We have, reasonably in my opinion, decided not to provide interrupt > driven receive/send (as in Intel hrecv/hsend) in MPI, or active > messages. Interrupt driven messages are too important to be left out. Any attempt made either by the application or by the compiler to cache data in the local storage of any compute node can only be effective if interrupt driven messages are supported. The interrupt handling procedure should allow both MPI and I/O. Indeed, this opens a Pandora box. For example how many such messages should be queued for a suspended process, should one place a limit upon the size of an interrupt message and upon the time spent in the interrupt handling procedure and so on. Yet it seems reasonable to expect that the standardization effort in OS will make this task easier, for example one could expect that more vendors will base their OS on OSF/1. Moreover, interrupt driven messages are likely to be provided by most vendors. Dan Marinescu From owner-mpi-comm@CS.UTK.EDU Thu Apr 8 07:12:11 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07522; Thu, 8 Apr 93 07:12:11 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05594; Thu, 8 Apr 93 07:10:15 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 8 Apr 1993 07:10:14 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05576; Thu, 8 Apr 93 07:10:10 -0400 Date: Thu, 8 Apr 93 12:10:04 BST Message-Id: <2092.9304081110@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: mpi-comm: various (long) To: mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Dear MPI Colleagues I am sending this letter to mpi-comm as it seems to cover issues in many subcommittees and in particular: point-to-point, collective, context, environmental. This is a long, but I believe important, letter. The letter is a discussion of some issues primarily in point-to-point, collective and context, with some strong and powerful recommendations for each of these subcommittees. o--------------------o 1) In the recent meeting we discussed and narrowly accepted an amendment to the point-to-point chapter which included secure (aka synchronous, aka unbuffered) communication. Paul Pierce made the point that providing different point-to-point procedures does not seem to be the right model for this capability as it has a microscopic effect on MPI user software. This point is actually well taken (by myself at least). Paul went on to suggest that there should be application global control over whether point-to-point communications are all secure (aka synchronous, aka unbuffered) or all insecure (aka regular, aka asynchronous, aka buffered). I opposed this suggestion as it does not respect the requirements of library writers. There appeared to be a number of people who were concerned about the geometric increase in the number of point-to-point communication procedures. The discussion of secure send and secure receive became messy, and I promised to return some more sane discussion of the point. In this letter I will later (after consideration of some issues in collective communications and communication contexts) make a recommendation which: respects the well taken point of Paul; provides a mechanism for the global control suggested by Paul while respecting the independence of library writers from user program writers; avoids geometric growth in the number of point-to-point procedures; and completely side-steps the secure send and secure receive issue. 2) The question of whether collective communication procedures do or do not barrier synchronise the group has been raised a number of times. In the recent meeting this lead to Steve Otto suggesting that MPI user software should be written as though these operations do barrier synchronise although the MPI implementation is not obliged to make the operations so do. This is exactly analagous to the point-to-point insecure (aka regular, ...) communications which may or may not synchronise the sender and receiver depending on the implementation (read depending on system buffering resource availability if you prefer). In order to write safe (i.e. guaranteed portable) programs with insecure (aka regular, ...) communications the MPI user software should be written as though these communications were secure (aka synchronous, ...). In this letter I will later (after consideration of some issues in communication contexts) make a recommendation which: provides for both secure (i.e. barrier synchronous) and insecure (i.e. may or may not be barrier synchronous depending on implementation) collective communications; and avoids geometric growth in the number of collective communication procedures. 3) In the context subcommittee Mark Sears, in Proposal VIII and in presentation at the recent meeting, has pointed out that there are two classes of MPI user libraries. I will discuss this (from my point of view) briefly here for concreteness. There is a class of libraries which accept context(s) from the caller and use these context(s) for communication. Note that since the library uses the contexts(s) given by the caller then it is clearly the responsibility of the caller to ensure that there are no "dangling" communications, either in buffers or nonblocking, which will interfere with the communications of the library, and it is the responsibility of the library to ensure that there are no dangling communications between calls to the library which will interfere with communications of the user. Let me label this class as ClassA. There is a class of libraries which create context(s) for library private use and use only these context(s) for communication. Note that since the library uses private context(s) then the user does not have any concern regarding dangling communications which may interfere with the library communications, and equally the library may leave dangling communications between library calls since they cannot interfere with the user communications. Let me label this class as ClassB. ClassA libraries will generally operate within one (or more) group(s) and accept one (or more) group(s) from the caller, in addition to context(s). ClassB libraries will also generally operate within one (or more) group(s) and accept one (or more) group(s) from the caller, but do not accept context(s). This observation suggests that context and group should be different entities within MPI. (Whether we have an object which binds a group and a context is another issue to which I wish to return in a separate message of narrower circulation.) Observe that the (blocking) collective communications which are described in the MPI collective communication chapter draft are conformant with the ClassA library described and s single group. The kind of libraries which I am writing (and I suspect also Tony, although I am not sure) are conformant with the ClassB library described. 4) I now make a recommendation which ties together the points discussed above and delivers the promises made. First we observe that context is the MPI mechanism for distinguishing the communications of the user from (ClassB) libraries. The recommendation is that secure or insecure modes of operation are a a property of the communication context. This applies equally to both point-to-point communications. In point-to-point communications a secure context provides the secure point-to-point communication which I described at the recent meeting, and an insecure context provides the inseure point-to-point communications which is as discussed (regular) in the current chapter draft. (The semantics and desirability of ready-receiver communication in secure and insecure contexts is a subject to which I wish to return in a separate message of narrower circulation.) In collective communications a secure context provides barrier synchronisation across the group, and an insecure context may or may not provide barrier synchronisation which is as discussed in the recent meeting. There should be an application global default context property, either secure or insecure, which is specified by the user in an environmental management procedure (perhaps as argument to mpi_init()?). Creation of context should specify whether the created context is to have: default property; secure property; insecure property. It is possibe to provide these capabilities in terms of buffering in different ways. One possibility is that the property of a context is the buffering associated with that context, in which case a secure context is one with zero associated buffering. Another possibility is that the property of a context is whether or not it should use whatever buffering is available, in which case a secure context is one which does not use any buffering. My preference is for the latter since it does not prescribe the description and association of buffering. This recommendation does deliver the promises made. The requirements of the (ClassB) library writer are respected while providing global control for the user program writer. The capabilities required are provided without geometric growth of the number of point-to-point and/or collective communications. o--------------------o Free beer to all who can demonstrate that they have read and considered this (long long letter). Comments, questions, (flames :-) please?! Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Fri Apr 9 11:26:20 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29024; Fri, 9 Apr 93 11:26:20 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23758; Fri, 9 Apr 93 11:24:22 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 9 Apr 1993 11:24:21 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23750; Fri, 9 Apr 93 11:24:19 -0400 Date: Fri, 9 Apr 93 16:24:14 BST Message-Id: <3171.9304091524@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: mpi-comm: various (long) To: lyndon@epcc.ed.ac.uk, mpi-comm@cs.utk.edu In-Reply-To: L J Clarke's message of Thu, 8 Apr 93 12:10:04 BST Reply-To: lyndon@epcc.ed.ac.uk Dear MPI Colleagues This message is short. It is two little things which I forgot to say in the "mpi-comm: various (long)" message. We can loosely associate ClassA libraries as collections of procedures for which one can use the substitution model of evaluation, and ClassB procedures as encapsulated objects. The argument lists of of ClassA and ClassB libraries differ primarily in that ClassA libraries accept context(s) whereas ClassB libraries do not. This seems (to me) to convey just the right idea to the library user regarding the responsibilities for management of contexts and messages within those contexts. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Tue Apr 20 16:21:16 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08357; Tue, 20 Apr 93 16:21:16 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03609; Tue, 20 Apr 93 16:18:26 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 20 Apr 1993 16:18:25 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from gstws.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03601; Tue, 20 Apr 93 16:18:23 -0400 Received: by gstws.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA14309; Tue, 20 Apr 1993 16:18:20 -0400 Date: Tue, 20 Apr 1993 16:18:20 -0400 From: geist@gstws.epm.ornl.gov (Al Geist) Message-Id: <9304202018.AA14309@gstws.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: 2nd draft of Collective Communication proposal After studying the latest point-to-point draft by Marc and the votes from the last meeting, I have revised the original collective communication draft. Comments can be directed to the mpi-collcomm committee. Here is the LaTeX version, a postscript version will follow in a separate message. Al ----------------------------- __o /\ Al Geist _`\<,_ /\/ \ Oak Ridge National Laboratory (_)/ (_) / \ (615) 574-3153 gst@ornl.gov * * * * * * * * * * ----------------------------- % MPI Authors: % This is MY version of YOUR chapter. It has a wrapper so that you % should be able to simply LaTeX this. % % Please work from this text so that we are in synch. % % --Steve Otto \documentstyle[twoside,11pt]{report} \pagestyle{headings} %\markright{ {\em Draft Document of the MPI Standard,\/ \today} } \marginparwidth 0pt \oddsidemargin=.25in \evensidemargin .25in \marginparsep 0pt \topmargin=-.5in \textwidth=6.0in \textheight=9.0in \parindent=2em % ---------------------------------------------------------------------- % mpi-macs.tex --- man page macros, % discuss, missing, mpifunc macros % % ---------------------------------------------------------------------- % a couple of commands from Marc Snir, modified S. Otto \newlength{\discussSpace} \setlength{\discussSpace}{.7cm} \newcommand{\discuss}[1]{\vspace{\discussSpace} {\small {\bf Discussion:} #1} \vspace{\discussSpace} } \newcommand{\missing}[1]{\vspace{\discussSpace} {\small {\bf Missing:} #1} \vspace{\discussSpace} } \newlength{\codeSpace} \setlength{\codeSpace}{.3cm} \newcommand{\mpifunc}[1]{\vspace{\codeSpace} {\bf #1} \vspace{\codeSpace} } % ----------------------------------------------------------------------- % A few commands to help in writing MPI man pages % \def\twoc#1#2{ \begin{list} {\hbox to95pt{#1\hfil}} {\setlength{\leftmargin}{120pt} \setlength{\labelwidth}{95pt} \setlength{\labelsep}{0pt} \setlength{\partopsep}{0pt} \setlength{\parskip}{0pt} \setlength{\topsep}{0pt} } \item {#2} \end{list} } \outer\long\def\onec#1{ \begin{list} {} {\setlength{\leftmargin}{25pt} \setlength{\labelwidth}{0pt} \setlength{\labelsep}{0pt} \setlength{\partopsep}{0pt} \setlength{\parskip}{0pt} \setlength{\topsep}{0pt} } \item {#1} \end{list} } \def\manhead#1{\noindent{\bf{#1}}} \hyphenation{RE-DIS-TRIB-UT-ABLE sub-script mul-ti-ple} \begin{document} \setcounter{page}{1} \pagenumbering{roman} \title{ {\em D R A F T} \\ Document for a Standard Message-Passing Interface} \author{Scott Berryman, {\em Yale Univ} \\ James Cownie, {\em Meiko Ltd} \\ Jack Dongarra, {\em Univ. of Tennessee and ORNL} \\ Al Geist, {\em ORNL} \\ Bill Gropp, {\em ANL} \\ Rolf Hempel, {\em GMD} \\ Bob Knighten, {\em Intel} \\ Rusty Lusk, {\em ANL} \\ Steve Otto, {\em Oregon Graduate Inst} \\ Tony Skjellum, {\em Missisippi State Univ} \\ Marc Snir, {\em IBM T. J. Watson} \\ David Walker, {\em ORNL} \\ Steve Zenith, {\em Kuck \& Associates} } \date{April 20, 1993 \\ This work was supported by ARPA and NSF under contract number \#\#\#, by the National Science Foundation Science and Technology Center Cooperative Agreement No. CCR-8809615. } \maketitle \hfuzz=5pt %\tableofcontents %\begin{abstract} %We don't have an abstract yet. %\end{abstract} \setcounter{page}{1} \pagenumbering{arabic} \chapter{Collective Communication} \label{sec:coll} \begin{center} Al Geist \\ Marc Snir \end{center} \section{Introduction} This section is a draft of the current proposal for collective communication. Collective communication is defined to be communication that involves a group of processes. Examples are broadcast and global sum. A collective operation is executed by having all processes in the group call the communication routine, with matching parameters. Routines can (but are not required to) return as soon as their participation in the collective communication is complete. The completion of a call indicates that the caller is now free to access the locations in the communication buffer, or any other location that can be referenced by the collective operation. However, it does not indicate that other processes in the group have started the operation (unless otherwise indicated in the description of the operation). However, the successful completion of a collective communication call may depend on the execution of a matching call at all processes in the group. The syntax and semantics of the collective operations is defined so as to be consistent with the syntax and semantics of the point to point operations. The reader is referred to the point-to-point communication section of the current MPI draft for information concerning communication buffers and their manipulations. The context section describes the formation, manipulation, and query functions (such as group size) that are available for groups and group objects. The collective communication routines are built above the point-to-point routines. While vendors may optimize certain collective routines for their architectures, a complete library of the collective communication routines can be written entirely using point-to-point communication functions. We are using naive implementations of the collective calls in terms of point to point operations in order to provide an operational definition of their semantics. The following communication functions are proposed. \begin{itemize} \item Broadcast from one member to all members of a group. \item Barrier across all group members \item Gather data from all group members to one member. \item Scatter data from one member to all members of a group. \item Global operations such as sum, max, min, etc., were the result is known by all group members and a variation where the result is known by only one member. The ability to have user defined global operations. \item Simultaneous shift of data around the group, the simplest example being all members sending their data to (rank+1) with wrap around. \item Scan across all members of a group (also called parallel prefix). \item Broadcast from all members to all members of a group. \item Scatter data from all members to all members of a group (also called complete exchange or index). \end{itemize} To simplify the collective communication interface it is designed with two layers. The low level routines have all the generality of, and make use of, the communication buffer routines of the point-to-point section which allows arbitrarily complex messages to be constructed. The second level routines are similar to the upper level point-to-point routines in that they send only a contiguous buffer. \section{Group Functions} A full description of the group formation and manipulation functions can be found in the context chapter of the MPI document. Here we describe only those group functions that are used in the semantic description of the collective communication routines. An initial group containing all processes is supplied by default in MPI. MPI provides a procedure that returns the handle to this initial group. The processes in the inital group each have a unique rank in the group represented by integers (0, 1, 2, ..., [number-of-processes~-~1]. \mpifunc{MPI\_ALLGROUP(group)} Return the descriptor of the inital group containing all processes. \begin{description} \item[OUT group] handle to descriptor object of initial group. \end{description} \mpifunc{MPI\_RANK(group, rank)} Return the rank of the calling process within the specified group. \begin{description} \item[IN group] group handle \item[OUT rank] integer \end{description} \mpifunc{MPI\_GSIZE(group, size)} Return the number of processes that belong to the specified group. \begin{description} \item[IN group] group handle \item[OUT size] integer \end{description} \section{Communication Functions} The proposed communication functions are divided into two layers. The lowest level uses the same buffer descriptor objects available in point-to-point to create noncontiguous, multiple data type messages. The second level is similar to the block send/receive point-to-point operations in that it supports only contiguous buffers of data. For each communication operation, we list these two level of calls together. \section{Synchronization} \subsubsection*{Barrier synchronization} \mpifunc{MPI\_BARRIER( group )} MPI\_BARRIER blocks the calling process until all group members have called it; the call returns at any process only after all group members have entered the call. \begin{description} \item[IN group] group handle \end{description} {\tt MPI\_BARRIER( group )} is \begin{verbatim} MPI_CREATE_BUFFER(buffer_handle, MPI_BUFFER, MPI_PERSISTENT); MPI_GSIZE( group, &size ); MPI_RANK( group, &rank ); if (rank==0) { for (i=1; i < size; i++) MPI_RECV(buffer_handle, i, tag, group, return_handle); for (i=1; i < size; i++) MPI_SEND(buffer_handle, i, tag, group); } else { MPI_SEND(buffer_handle, 0, tag, group); MPI_RECV(buffer_handle, 0, tag, group, return_handle); } MPI_FREE(buffer_handle); \end{verbatim} \discuss{ since tag was voted out in the last meeting, a "safe" tag must be selected to use in the above description } \section{Data move functions} \subsubsection*{Broadcast} \mpifunc{ MPI\_BCAST( buffer\_handle, group, root )} {\tt MPI\_BCAST} broadcasts a message from the process with rank {\tt root} to all other processes of the group. It is called by all members of group using the same arguments for {\tt group, and root}. On return the contents of the buffer of the process with rank {\tt root} is contained in buffer of all group members. \begin{description} \item[INOUT buffer\_handle] Handle for buffer where from message is sent or received. \item[IN group] group handle \item[IN root] rank of broadcast root (integer) \end{description} \mpifunc{ MPI\_BCASTC( buf, len, group, root )} {\tt MPI\_BCASTC} behaves like broadcast, restricted to a block buffer. It is called by all processes with the same arguments for {\tt len, group} and {\tt root}. \begin{description} \item[INOUT buffer] Starting address of buffer (choice type) \item[IN len] Number of words in buffer (integer) \item[IN group] group handle \item[in root] rank of broadcast root (integer) \end{description} {\tt MPI\_BCAST( buffer\_handle, group, root )} is \begin{verbatim} MPI_GSIZE( group, &size ); MPI_RANK( group, &rank ); MPI_IRECV(handle, buffer_handle, root, tag, group, return_handle); if (rank==root) for (i=0; i < size; i++) MPI_SEND(buffer_handle, i, tag, group); MPI_WAIT(handle) \end{verbatim} \subsubsection*{Circular shift} \mpifunc{MPI\_CSHIFT( inbuf, outbuf, group, shift)} Process with rank {\tt i} sends the data in its input buffer to process with rank $\tt (i+ shift) \bmod group\_size$, who receives the data in its output buffer. All processes make the call with the same values for {\tt group}, and {\tt shift}. The {\tt shift} value can be positive, zero, or negative. \begin{description} \item[IN inbuf] handle to input buffer descriptor \item[OUT outbuf] handle to output buffer descriptor \item[IN group] handle to group \item[IN shift] integer \end{description} \mpifunc{MPI\_CSHIFT1( buf, group, shift)} Process with rank {\tt i} sends the data in its buffer to process with rank $\tt (i+ shift) \bmod group\_size$, who receives the data in the same buffer. All processes make the call with the same values for {\tt group}, and {\tt shift}. The {\tt shift} value can be positive, zero, or negative. \begin{description} \item[INOUT buf] handle to buffer descriptor \item[IN group] handle to group \item[IN shift] integer \end{description} \mpifunc{MPI\_CSHIFTC( inbuf, outbuf, len, group, shift)} Behaves like {\tt MPI\_CSHIFT}, with buffers restricted to be blocks of numeric units. All processes make the call with the same values for {\tt len, group}, and {\tt shift}. \begin{description} \item[IN inbuf] initial location of input buffer \item[OUT outbuf] initial location of output buffer \item[IN len] number of entries in input (and output) buffers \item[IN group] handle to group \item[IN shift] integer \end{description} \mpifunc{MPI\_CSHIFTC1( buf, len, group, shift)} Behaves like {\tt MPI\_CSHIFT1}, with buffers restricted to be blocks of numeric units. All processes make the call with the same values for {\tt len, group}, and {\tt shift}. \begin{description} \item[INOUT buf] initial location of buffer \item[IN len] number of entries in input (and output) buffers \item[IN group] handle to group \item[IN shift] integer \end{description} {\tt MPI\_CSHIFT( inbuf, outbuf, group, shift)} is \begin{verbatim} MPI_GSIZE( group, &size ); MPI_RANK( group, &rank ); MPI_ISEND( handle, inbuf, mod(rank+shift, size), tag, group); MPI_RECV( outbuf, mod(rank-shift,size), tag, group, return_handle) MPI_WAIT(handle); \end{verbatim} \subsubsection*{End-off shift} \mpifunc{MPI\_EOSHIFT( inbuf, outbuf, group, shift)} Process with rank {\tt i}, $\tt \max( 0, -shift) \le i < min( size, size - shift)$, sends the data in its input buffer to process with rank {\tt i+ shift}, who receives the data in its output buffer. The output buffer of processes which do not receive data is left unchanged. All processes make the call with the same values for {\tt group}, and {\tt shift}. \begin{description} \item[IN inbuf] handle to input buffer descriptor \item[OUT outbuf] handle to output buffer descriptor \item[IN group] handle to group \item[IN shift] integer \end{description} \mpifunc{MPI\_EOSHIFT1( buf, group, shift)} Process with rank {\tt i}, $\tt \max( 0, -shift) \le i < min( size, size - shift)$, sends the data in its buffer to process with rank {\tt i+ shift}, who receives the data in the same buffer. The output buffer of processes which do not receive data is left unchanged. All processes make the call with the same values for {\tt group}, and {\tt shift}. \begin{description} \item[IN inbuf] handle to input buffer descriptor \item[OUT outbuf] handle to output buffer descriptor \item[IN group] handle to group \item[IN shift] integer \end{description} \mpifunc{MPI\_EOSHIFTC( inbuf, outbuf, len, group, shift)} Behaves like {\tt MPI\_EOSHIFT}, with buffers restricted to be blocks of numeric units. All processes make the call with the same values for {\tt len, group}, and {\tt shift}. \begin{description} \item[IN inbuf] initial location of input buffer \item[OUT outbuf] initial location of output buffer \item[IN len] number of entries in input (and output) buffers \item[IN group] handle to group \item[IN shift] integer \end{description} \mpifunc{MPI\_EOSHIFTC1( buf, len, group, shift)} Behaves like {\tt MPI\_EOSHIFT1}, with buffer restricted to be blocks of numeric units. All processes make the call with the same values for {\tt len, group}, and {\tt shift}. \begin{description} \item[INOUT buf] initial location of buffer \item[IN len] number of entries in input (and output) buffers \item[IN group] handle to group \item[IN shift] integer \end{description} \subsubsection*{Gather} \mpifunc{MPI\_GATHER( inbuf, outbuf, group, root, return\_status) } Each process (including the root process) sends the content of its input buffer to the root process. The root process concatenates all the incoming messages in the order of the senders' rank and places the results in its output buffer. It is called by all members of group using the same arguments for {\tt group}, and {\tt root}. The input buffer of each process may have a different length. \begin{description} \item[IN inbuf] handle to input buffer descriptor \item[OUT outbuf] handle to output buffer descriptor -- significant only at root (choice) \item[IN group] group handle \item[IN root] rank of receiving process (integer) \item[OUT return\_status] return status handle \end{description} \mpifunc{MPI\_GATHERC( inbuf, inlen, outbuf, group, root) } {\tt MPI\_GATHERC} behaves like {\tt MPI\_GATHER} restricted to block buffers, and with the additional restriction that all input buffers should have the same length. All processes should provided the same values for {\tt inlen, group}, and {\tt root} . \begin{description} \item[IN inbuf] first variable of input buffer (choice) \item[IN inlen] Number of (word) variables in input buffer (integer) \item[OUT outbuf] first variable of output buffer -- significant only at root (choice) \item[IN group] group handle \item[IN root] rank of receiving process (integer) \end{description} {\tt MPI\_GATHERC( inbuf, inlen, outbuf, tag, group, root) } is \begin{verbatim} MPI_GSIZE( &size, group); MPI_RANK( &rank, group); MPI_ISENDC(handle, inbuf, inlen, root, tag, group); if (rank==root) for (i=0; i < size; i++) { MPI_RECVC(outbuf, inlen, i, tag, group, return\_status); outbuf += inlen; } MPI_WAIT(handle); \end{verbatim} \subsubsection*{Scatter} \mpifunc{MPI\_SCATTER( list\_of\_inbufs, outbuf, group, root, return\_status)} The root process sends the content of its {\tt i}-th input buffer to the process with rank {\tt i}; each process (including the root process) stores the incoming message in its output buffer. The routine is called by all members of the group using the same arguments for {\tt group}, and {\tt root}. \begin{description} \item[IN list\_of\_inbufs] list of buffer descriptor handles \item[OUT outbuf] buffer descriptor handle \item[IN group] handle \item[IN root] rank of sending process (integer) \item[OUT return\_status] return status handle \end{description} {\tt MPI\_SCATTER( list\_of\_inbufs, outbuf, group, root, return\_status)} is \begin{verbatim} MPI_GSIZE( group, &size ); MPI_RANK( group, &rank ); MPI_IRECV(handle, outbuf, root, tag, group); if (rank=root) for (i=0; i < size; i++) MPI_SEND(inbuf[i], i, tag, group); MPI_WAIT(handle, return\_status); \end{verbatim} \mpifunc{MPI\_SCATTERC( inbuf, outbuf, len, group, root)} {\tt MPI\_SCATTERC} behaves like {\tt MPI\_SCATTER} restricted to block buffers, and with the additional restriction that all output buffers have the same length. The input buffer block of the root process is partitioned into {\tt n} consecutive blocks, each consisting of {\tt len} words. The {\tt i}-th block is sent to the {\tt i}-th process in the group and stored in its output buffer. The routine is called by all members of the group using the same arguments for {\tt group, len}, and {\tt root}. \begin{description} \item[IN inbuf] first entry in input buffer -- significant only at root (choice). \item[OUT outbuf] first entry in output buffer (choice). \item[IN len] number of entries to be stored in output buffer (integer) \item[IN group] handle \item[IN root] rank of sending process (integer) \end{description} {\tt MPI\_SCATTERC( inbuf, outbuf, outlen, group, root) } is \begin{verbatim} MPI_GSIZE( &size, group); MPI_RANK( &rank, group); MPI_IRECVC( handle, outbuf, outlen, root, tag, group); if (rank=root) for (i=0; i < size; i++) { MPI_SENDC(inbuf, outlen, i, tag, group, return\_status); inbuf += outlen; } MPI_WAIT(handle); \end{verbatim} \subsubsection*{All-to-all scatter} \mpifunc{MPI\_ALLSCATTER( list\_of\_inbufs, outbuf, group, return\_status)} Each process in the group sends its {\tt i}-th buffer in its input buffer list to the process with rank {\tt i} (itself included); each process concatenates the incoming messages in its output buffer, in the order of the senders' ranks. The routine is called by all members of the group using the same arguments for {\tt group}. \begin{description} \item[IN list\_of\_inbufs] list of buffer descriptor handles \item[OUT outbuf] buffer descriptor handle \item[IN group] handle \item[OUT return\_status] return status handle \end{description} \mpifunc{MPI\_ALLSCATTERC( inbuf, outbuf, len, group)} {\tt MPI\_ALLSCATTERC} behaves like {\tt MPI\_ALLSCATTER} restricted to block buffers, and with the additional restriction that all blocks sent from one process to another have the same length. The input buffer block of each process is partitioned into {\tt n} consecutive blocks, each consisting of {\tt len} words. The {\tt i}-th block is sent to the {\tt it}-th process in the group. Each process concatenates the incoming messages, in the order of the senders' ranks, and store them in its output buffer. The routine is called by all members of the group using the same arguments for {\tt group}, and {\tt len}. \begin{description} \item[IN inbuf] first entry in input buffer (choice). root (integer) \item[OUT outbuf] first entry in output buffer (choice). \item[IN len] number of entries sent from each process to each other (integer). \item[IN group] handle \end{description} {\tt MPI\_ALLSCATTERC( inbuf, outbuf, len, group)} is \begin{verbatim} MPI_GSIZE( group, &size ); MPI_RANK( group, &rank ); for (i=0; i < rank; i++) { MPI_IRECVC(recv_handles[i], outbuf, len, tag, group); outbuf += len; } for (i=0; i < size; i++) { MPI_ISENDC(send_handle[i], inbuf, len, i, tag, group); inbuf += len; } MPI_WAITALL(send_handle); MPI_WAITALL(recv_handle); \end{verbatim} \subsubsection*{All-to-all broadcast} \mpifunc{MPI\_ALLCAST( inbuf, outbuf, group, return\_status)} Each process in the group broadcasts its input buffer to all processes (including itself); each process concatenates the incoming messages in its output buffer, in the order of the senders' ranks. The routine is called by all members of the group using the same arguments for {\tt group}. \begin{description} \item[IN inbuf] buffer descriptor handle for input buffer \item[OUT outbuf] buffer descriptor handle for output buffer \item[IN group] handle \item[OUT return\_status] return status handle \end{description} \mpifunc{MPI\_ALLCASTC( inbuf, outbuf, len, group)} {\tt MPI\_ALLCASTC} behaves like {\tt MPI\_ALLCAST} restricted to block buffers, and with the additional restriction that all blocks sent from one process to another have the same length. The routine is called by all members of the group using the same arguments for {\tt group}, and {\tt len}. \begin{description} \item[IN inbuf] first entry in input buffer (choice). root (integer) \item[OUT outbuf] first entry in output buffer (choice). \item[IN len] number of entries sent from each process to each other (including itself). \item[IN group] group handle \end{description} {\tt MPI\_ALLCASTC( inbuf, outbuf, len, group)} is \begin{verbatim} MPI_GSIZE( group, &size ); MPI_RANK( group, &rank ); for (i=0; i < rank; i++) { MPI_IRECVC(recv_handles[i], outbuf, len, tag, group); outbuf += len; } for (i=0; i < size; i++) { MPI_ISENDC(send_handle[i], inbuf, len, i, tag, group); } MPI_WAITALL(send_handle); MPI_WAITALL(recv_handle); \end{verbatim} \section{Global Compute Operations} \subsubsection*{Reduce} \mpifunc{MPI\_REDUCE( inbuf, outbuf, group, root, op)} Combines the values provided in the input buffer of each process in the group, using the operation {\tt op}, and returns the combined value in the output buffer of the process with rank {\tt root}. Each process can provide one value, or a sequence of values, in which case the combine operation is executed pointwise on each entry of the sequence. For example, if the operation is {\tt max} and input buffers contains two floating point numbers, then outbuf(1) $=$ global max(inbuf(1)) and outbuf(2) $=$ global max(inbuf(2)). All input buffers should define sequences of equal length of entries of types that match the type of the operands of {\tt op}. The output buffer should define a sequence of the same length of entries of types that match the type of the result of {\tt op}. (Note that, here as for all other communication operations, the type of entries inserted in a message depend on the information provided by the input buffer descriptor, and not on the declarations of these variables in the calling program. The types of the variables in the calling program need not match the types defined by the buffer descriptor, but in such case the outcome of a reduce operation may be implementation dependent.) The operation defined by {\tt op} is associative and commutative, and the implementation can take advantage of associativity and commutativity in order to change order of evaluation. The routine is called by all group members using the same arguments for {\tt group, root} and {\tt op}. \begin{description} \item[IN inbuf] handle to input buffer \item[OUT outbuf] handle to output buffer -- significant only at root \item[IN group] handle to group \item[IN root] rank of root process (integer) \item[IN op] operation (status) \end{description} We list below the operations which are supported for Fortran, each with the corresponding value of the {\tt op} parameter. \begin{description} \item[MPI\_IMAX] integer maximum \item[MPI\_RMAX] real maximum \item[MPI\_DMAX] double precision real maximum \item[MPI\_IMIN] integer minimum \item[MPI\_RMIN] real minimum \item[MPI\_DMIN] double precision real minimum \item[MPI\_ISUM] integer sum \item[MPI\_RSUM] real sum \item[MPI\_DSUM] double precision real sum \item[MPI\_CSUM] complex sum \item[MPI\_DCSUM] double precision complex sum \item[MPI\_IPROD] integer product \item[MPI\_RPROD] real product \item[MPI\_DPROD] double precision real product \item[MPI\_CPROD] complex product \item[MPI\_DCPROD] double precision complex product \item[MPI\_AND] logical and \item[MPI\_IAND] integer (bit-wise) and \item[MPI\_OR] logical or \item[MPI\_IOR] integer (bit-wise) or \item[MPI\_XOR] logical xor \item[MPI\_IXOR] integer (bit-wise) xor \item[MPI\_MAXLOC] rank of process with maximum integer value \item[MPI\_MAXRLOC] rank of process with maximum real value \item[MPI\_MAXDLOC] rank of process with maximum double precision real value \item[MPI\_MINLOC] rank of process with minimum integer value \item[MPI\_MINRLOC] rank of process with minimum real value \item[MPI\_MINDLOC] rank of process with minimum double precision real value \end{description} \mpifunc{MPI\_REDUCEC( inbuf, outbuf, len, group, root, op)} Is same as {\tt MPI\_REDUCE}, restricted to a block buffer. \begin{description} \item[IN inbuf] first location in input buffer \item[OUT outbuf] first location in output buffer -- significant only at root \item[IN len] number of entries in input and output buffer (integer) \item[IN group] handle to group \item[IN root] rank of root process (integer) \item[IN op] operation (status) \end{description} \mpifunc{MPI\_USER\_REDUCE( inbuf, outbuf, group, root, function)} Same as the reduce operation function above except that a user supplied function is used. {\tt function} is an associative and commutative function with two arguments. The types of the two arguments and of the returned value of the function, and the types of all entries in the input and output buffers all agree. The output buffer has the same length as the input buffer. \begin{description} \item[IN inbuf] handle to input buffer \item[OUT outbuf] handle to output buffer -- significant only at root \item[IN group] handle to group \item[IN root] rank of root process (integer) \item[IN function] user provided function \end{description} \mpifunc{MPI\_USER\_REDUCEC( inbuf, outbuf, len, group, root, function)} Is same as {\tt MPI\_\_USER\_REDUCE}, restricted to a block buffer. \begin{description} \item[IN inbuf] first location in input buffer \item[OUT outbuf] first location in output buffer -- significant only at root \item[IN len] number of entries in input and output buffer (integer) \item[IN group] handle to group \item[IN root] rank of root process (integer) \item[IN op] operation (status) \end{description} \discuss{ Do we also want a version of reduce that broadcasts the result to all processes in the group? (This can be achieved by a reduce followed by a broadcast, but a combined function may be somewhat more efficient.) These would be respectively: \mpifunc{MPI\_GOP( inbuf, outbuf, group, op)} \mpifunc{MPI\_GOPC( inbuf, outbuf, len, group, op)} \mpifunc{MPI\_USER\_GOP( inbuf, outbuf, group, function)} \mpifunc{MPI\_USER\_GOPC( inbuf, outbuf, len, group, function)} Do we want a user provided {\em function} (two IN parameters, one OUT value), or a user provided procedure that overwrites the second input (ie. one IN param, one INOUT param, the equivalent of C {\tt a op= b} type assignment)? The second choice may allow a more efficient implementation, without changing the semantics of the MPI call. } \subsubsection*{Scan} \mpifunc{ MPI\_SCAN( inbuf, outbuf, group, op )} MPI\_SCAN is used to perform a parallel prefix with respect to an associative reduction operation on data distributed across the group. The operation returns in the output buffer of the process with rank {\tt i} the reduction of the values in the input buffers of processes with ranks {\tt 0,...,i}. The type of operations supported and their semantic, and the constraints on input and output buffers are as for {\tt MPI\_REDUCE}. \begin{description} \item[IN inbuf] handle to input buffer \item[OUT outbuf] handle to output buffer \item[IN group] handle to group \item[IN op] operation (status) \end{description} \mpifunc{ MPI\_SCANC( inbuf, outbuf, len, group, op )} Same as {\tt MPI\_SCAN}, restricted to block buffers. \begin{description} \item[IN inbuf] first input buffer element (choice) \item[OUT outbuf] first output buffer element (choice) \item[IN len] number of entries in input and output buffer (integer) \item[IN group] handle to group \item[IN op] operation (status) \end{description} \mpifunc{ MPI\_USER\_SCAN( inbuf, outbuf, group, function )} Same as the scan operation function above except that a user supplied function is used. {\tt function} is an associative and commutative function with two arguments. The types of the two arguments and of the returned values all agree. \begin{description} \item[IN inbuf] handle to input buffer \item[OUT outbuf] handle to output buffer \item[IN group] handle to group \item[IN function] user provided function \end{description} \mpifunc{MPI\_USER\_SCANC( inbuf, outbuf, len, group, function)} Is same as {\tt MPI\_USER\_SCAN}, restricted to a block buffer. \begin{description} \item[IN inbuf] first location in input buffer \item[OUT outbuf] first location in output buffer \item[IN len] number of entries in input and output buffer (integer) \item[IN group] handle to group \item[IN function] user provided function \end{description} \discuss{ Do we want scan operations executed by segments? (The HPF definition of prefix and suffix operation might be handy -- in addition to the scanned vector of values there is a mask that tells where segments start and end.) } \section{Correctness} \discuss{ This is still very preliminary} The semantics of the collective communication operations can be derived from their operational definition in terms of point-to-point communication. It is assumed that messages pertaining to one operation cannot be confused with messages pertaining to another operation. Also messages pertaining to two distinct occurrences of the same operation cannot be confused, if the two occurrences have distinct parameters. The relevant parameters for this purpose are {\tt group}, {\tt root} and {\tt op}. messages pertaining to another occurrence of the same operation, with different parameters. The implementer can, of course, use another, more efficient implementation, as long as it has the same effect. \discuss{ This statement does not yet apply to the current, incomplete and somewhat careless definitions I provided in this draft. The definition above means that messages pertaining to a collective communication carry information identifying the operation itself, and the values of the {\tt group} and, where relevant, {\tt root} or {\tt op} parameters. Is this acceptable? } A few examples: \begin{verbatim} MPI_BCAST(buf, len, group, 0); MPI_BCAST(buf, len, group, 1); \end{verbatim} Two consecutive broadcasts, in the same group, with the same tag, but different roots. Since the operations are distinguishable, messages from one broadcast cannot be confused with messages from the other broadcast; the program is safe and will execute as expected. \begin{verbatim} MPI_BCAST(buf, len, group, 0); MPI_BCAST(buf, len, group, 0); \end{verbatim} Two consecutive broadcasts, in the same group, with the same tag and root. Since point-to-point communication preserves the order of messages here, too, messages from one broadcast will not be confused with messages from the other broadcast; the program is safe and will execute as intended. \begin{verbatim} MPI_RANK(&rank, group) if (rank==0) { MPI_BCASTC(buf, len, group, 0); MPI_SENDC(buf, len, 2, tag, group); } elseif (rank==1) { MPI_RECVC(buf, len, MPI_DONTCARE, tag, group); MPI_BCASTC(buf, len, group, 0); MPI_RECVC(buf, len, MPI_DONTCARE, tag, group); } else { MPI_SENDC(buf, len, 2, tag, group); MPI_BCASTC(buf, len, group, 0); } \end{verbatim} Process zero executes a broadcast followed by a send to process one; process two executes a send to process one, followed by a broadcast; and process one executes a receive, a broadcast and a receive. A possible outcome is for the operations to be matched as illustrated by the diagram below. \begin{verbatim} 0 1 2 / - > receive / - send / / broadcast / broadcast / broadcast / / send - receive < - \end{verbatim} The reason is that broadcast is not a synchronous operation; the call at a process may return before the other processes have entered the broadcast. Thus, the message sent by process zero can arrive to process one before the message sent by process two, and before the call to broadcast on process one. \end{document} From owner-mpi-comm@CS.UTK.EDU Tue Apr 20 16:21:18 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08359; Tue, 20 Apr 93 16:21:18 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03662; Tue, 20 Apr 93 16:19:33 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 20 Apr 1993 16:19:30 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from gstws.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03654; Tue, 20 Apr 93 16:19:25 -0400 Received: by gstws.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA16360; Tue, 20 Apr 1993 16:19:21 -0400 Date: Tue, 20 Apr 1993 16:19:21 -0400 From: geist@gstws.epm.ornl.gov (Al Geist) Message-Id: <9304202019.AA16360@gstws.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Postscript of Collective Communication chapter of MPI %!PS-Adobe-2.0 %%Creator: dvips 5.47 Copyright 1986-91 Radical Eye Software %%Title: cc.dvi %%Pages: 17 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: tex.pro /TeXDict 200 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder}N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get}B /ch-height{ch-data dup length 4 sub get}B /ch-xoff{128 ch-data dup length 3 sub get sub}B /ch-yoff{ ch-data dup length 2 sub get 127 sub}B /ch-dx{ch-data dup length 1 sub get}B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image} imagemask restore}B /D{/cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put}for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval(Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{ moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{ S p tail}B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w }B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 400 400 @start /Fa 9 118 400 360 dfs[<7FFFF0FFFFF8FFFFF87F FFF00000000000000000000000007FFFF0FFFFF8FFFFF87FFFF0>21 12 126 148 29 61 D[<1FF0003FFC007FFE00781F00300780000380000380007F8007FF801FFF80 3FC3807C0380F00380E00380E00380E00380F007807C1F803FFFFC1FFDFC07F0FC>22 21 125 148 29 97 D[22 30 127 157 29 I[<00F8FC03FFFE07FFFE0F8F8C0E03801E03C01C01C01C01C01C01C01E03C00E03800F8F80 0FFF001FFE001CF8001C00001C00001E00000FFF801FFFF03FFFF87C00FC70001CF0001EE0000E E0000EE0000EF0001E78003C3F01F81FFFF00FFFE001FF00>23 33 127 148 29 103 D[<01F00007FC001FFF003E0F803C07807803C07001C0E000E0E000E0E000E0E000 E0E000E0E000E0F001E07001C07803C03C07803E0F801FFF0007FC0001F000>19 21 125 148 29 111 D[22 32 127 148 29 I[22 21 126 148 29 114 D[<00C00001C00001C00001C00001C00001C00001C0007FFFE0FFFFE0FF FFE001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C07001C07001 C07001C0F001E1E000FFE0007FC0003F00>20 28 127 155 29 116 D[23 21 127 148 29 I E /Fb 7 118 400 360 dfs[<007C01C2030307070E0F1C0F3C003800780078007800F000F000F000F000F001 70037006301C18380FC0>16 21 123 148 26 99 D[<00007C0000CE00019E00039E00038C0003 00000700000700000700000700000E00000E00000E00000E0001FFF001FFF0001C00001C00001C 00001C00001C0000380000380000380000380000380000700000700000700000700000700000E0 0000E00000E00000E00001C00001C00001C00001C000038000738000F30000F300006600003C00 00>23 45 130 162 17 102 D[<00E000E001E000C00000000000000000000000000000000000 001E00330063806380C380C700C70007000E000E000E001C001C001C40386038C070C070803180 31001E00>11 34 124 161 17 105 D[<1E07803318E063B06063E070C3C070C38070C3807007 00E00700E00700E00700E00E01C00E01C00E03820E03831C03861C07061C070C1C030838031818 01E0>24 21 124 148 31 110 D[<007C0001C6000303000603800E03C01C03C03C03C03803C0 7803C07803C07803C0F00780F00780F00780F00F00F00E00701E00701C003038001860000F8000 >18 21 123 148 28 I[<006000E000E000E000E001C001C001C001C00380FFF8FFF803800700 0700070007000E000E000E000E001C001C001C101C18383038303860186018C00F00>13 31 124 158 19 116 D[<0F003011807021C07061C0E0C1C0E0C380E0C380E00381C00701C007 01C00701C00E03800E03800E03840E03860E070C0C070C0E070C0E0B1806131003E1E0>23 21 124 148 30 I E /Fc 2 61 438 432 dfs[<78FCFCFEFE7A020202020404040810102040> 7 18 123 133 17 59 D[<00000000E000000003E00000000FC00000003F00000000FC00000003 F00000000FC00000003F00000000FC00000003F00000000FC00000003F00000000FC00000003F0 0000000FC00000003F00000000FC00000000F000000000FC000000003F000000000FC000000003 F000000000FC000000003F000000000FC000000003F000000000FC000000003F000000000FC000 000003F000000000FC000000003F000000000FC000000003E000000000E0>35 35 123 159 47 I E /Fd 46 124 400 360 dfs[<000FC0000078300000E0080001803C000380 7C0007007C0007007C0007003800070000000700000007000000070000000700000007000000FF FFFC00FFFFFC0007003C0007001C0007001C0007001C0007001C0007001C0007001C0007001C00 07001C0007001C0007001C0007001C0007001C0007001C0007001C0007001C0007001C007FF1FF C07FF1FFC0>26 35 128 162 31 12 D[<000FC03F00007031E0C000E00B802001803E00F00380 7E01F007007C01F007007C01F007003C00E007001C000007001C000007001C000007001C000007 001C000007001C0000FFFFFFFFF0FFFFFFFFF007001C00F007001C007007001C007007001C0070 07001C007007001C007007001C007007001C007007001C007007001C007007001C007007001C00 7007001C007007001C007007001C007007001C007007001C00707FF1FFC7FF7FF1FFC7FF>40 35 128 162 47 14 D[<701CF83EFC3FFC3F741D04010401040104010802080210041004200840 10>16 15 126 162 28 34 D[<001000200040008001000300060004000C001800180018003000 300030007000600060006000E000E000E000E000E000E000E000E000E000E000E000E000600060 00600070003000300030001800180018000C0004000600030001000080004000200010>12 50 125 164 21 40 D[<800040002000100008000C0006000200030001800180018000C000C000 C000E0006000600060007000700070007000700070007000700070007000700070006000600060 00E000C000C000C00180018001800300020006000C0008001000200040008000>12 50 125 164 21 I[<70F8FCFC7404040404080810102040>6 15 124 132 16 44 D[12 2 127 139 19 I[<70F8F8F870>5 5 124 132 16 I[<70F8F8F870000000000000000000000070F8F8F870>5 21 124 148 16 58 D[<07F000181C00200E00400700F00780F80780F80780F80780700780000F00000E0000 1C0000380000700000600000C00000800000800001800001000001000001000001000001000001 000000000000000000000000000000000003800007C00007C00007C000038000>17 35 125 162 27 63 D[<0007F008003FFC1800FC073801F001B803C000F8078000780F0000381E 0000183E0000183C0000187C0000087C00000878000008F8000000F8000000F8000000F8000000 F8000000F8000000F8000000F8000000780000007C0000087C0000083C0000083E0000081E0000 100F0000100780002003C0004001F0018000FC0700003FFE000007F000>29 34 125 161 40 67 D[32 34 126 161 42 I[28 34 126 161 37 70 D[32 34 126 161 41 72 D[14 34 126 161 20 I[41 34 126 161 51 77 D[32 34 126 161 41 I[<000FF00000781E0000E0070003C003C0078001E00F0000F01E0000781E0000783C00003C3C 00003C7C00003E7800001E7800001EF800001FF800001FF800001FF800001FF800001FF800001F F800001FF800001F7800001E7C00003E7C00003E3C00003C3E00007C1E0000781E0000780F0000 F0078001E003C003C000E0070000781E00000FF000>32 34 125 161 43 I[28 34 126 161 38 I[<7FFFFFFC7FFFFFFC7803C03C6003C00C4003C0044003C004C003C006C003C0068003C00280 03C0028003C0028003C0020003C0000003C0000003C0000003C0000003C0000003C0000003C000 0003C0000003C0000003C0000003C0000003C0000003C0000003C0000003C0000003C0000003C0 000003C0000003C0000003C00001FFFF8001FFFF80>31 34 126 161 40 84 D[32 34 126 161 41 I[<1FF000381C007C06007C07007C0380380380000380000380007F8007C3801E03803C0380 780380780380F00384F00384F00384F00784780B843C11C80FE0F0>22 21 126 148 28 97 D[<0E0000FE0000FE00001E00000E00000E00000E00000E00000E00000E0000 0E00000E00000E00000E00000E1F800E60E00E80300F00380E001C0E001E0E000E0E000F0E000F 0E000F0E000F0E000F0E000F0E000F0E000E0E001E0E001C0F00380C80700C60E0081F80>24 35 127 162 31 I[<01FE000707000C0F801C0F80380F80780700700000F00000F00000F00000 F00000F00000F00000F000007000007800403800401C00800C010007060001F800>18 21 126 148 24 I[<0000700007F00007F00000F0000070000070000070000070000070000070 00007000007000007000007001F8700706700E01701C00F0380070780070700070F00070F00070 F00070F00070F00070F00070F000707000707800703800701C00F00C017807067F01F87F>24 35 126 162 31 I[<01FC000707000C03801C01C03801C07800E07000E0F000E0FFFFE0F00000 F00000F00000F00000F000007000007800203800201C00400E008007030000FC00>19 21 127 148 24 I[<003E0000E30001C780038F80030F80070700070000070000070000070000 070000070000070000070000FFF800FFF800070000070000070000070000070000070000070000 0700000700000700000700000700000700000700000700000700000700007FF8007FF800>17 35 128 162 17 I[<01F078071D9C0E0E1C1C07001C07003C07803C07803C07803C07801C0700 1C07000E0E000F1C0019F0001000001000001800001C00001FFF000FFFE00FFFF03800F8600018 40001CC0000CC0000CC0000C6000186000183800700E01C001FE00>22 32 127 148 28 I[<0E000000FE000000FE0000001E0000000E0000000E0000000E0000000E000000 0E0000000E0000000E0000000E0000000E0000000E0000000E1F80000E60E0000E8070000F0038 000F0038000E0038000E0038000E0038000E0038000E0038000E0038000E0038000E0038000E00 38000E0038000E0038000E0038000E0038000E003800FFE3FF80FFE3FF80>25 35 127 162 31 I[<1C003E003E003E001C00000000000000000000000000000000000E00FE00 FE001E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00FFC0FFC0> 10 34 127 161 16 I[<0E0000FE0000FE00001E00000E00000E00000E00000E00000E00000E00 000E00000E00000E00000E00000E03FC0E03FC0E01E00E01800E02000E04000E08000E10000E38 000EF8000F1C000E1E000E0E000E07000E07800E03C00E01C00E01E00E01F0FFE3FEFFE3FE>23 35 127 162 29 107 D[<0E00FE00FE001E000E000E000E000E000E000E000E000E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00FF E0FFE0>11 35 127 162 16 I[<0E1FC07F00FE60E18380FE807201C01F003C00E00F003C00E0 0E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800 E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E0FFE3FF8FFEFFE3FF 8FFE>39 21 127 148 47 I[<0E1F8000FE60E000FE8070001F0038000F0038000E0038000E00 38000E0038000E0038000E0038000E0038000E0038000E0038000E0038000E0038000E0038000E 0038000E0038000E003800FFE3FF80FFE3FF80>25 21 127 148 31 I[<00FC000703800E01C0 1C00E0380070780078700038F0003CF0003CF0003CF0003CF0003CF0003CF0003C700038780078 3800701C00E00E01C007038000FC00>22 21 127 148 28 I[<0E1F80FE60E0FE80700F00380E 001C0E001E0E001E0E000F0E000F0E000F0E000F0E000F0E000F0E000F0E001E0E001E0E001C0F 00380E80700E60E00E1F800E00000E00000E00000E00000E00000E00000E00000E0000FFE000FF E000>24 31 127 148 31 I[<01F8200704600E02601C01603801E07800E07800E0F000E0F000 E0F000E0F000E0F000E0F000E0F000E07000E07800E03801E01C01E00C02E0070CE001F0E00000 E00000E00000E00000E00000E00000E00000E00000E0000FFE000FFE>23 31 126 148 29 I[<0E1E00FE6300FE87801E87800F03000F00000E00000E00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E0000FFF000FFF000>17 21 127 148 22 I[<0FC4303C600CC00CC004C004E004F0007F803FF00FF800FC001E800E8006 C006C006C004E00CD81887E0>15 21 126 148 22 I[<02000200020002000200060006000600 0E001E003FF8FFF80E000E000E000E000E000E000E000E000E000E000E040E040E040E040E040E 040708030801F0>14 31 127 158 21 I[<0E003800FE03F800FE03F8001E0078000E0038000E 0038000E0038000E0038000E0038000E0038000E0038000E0038000E0038000E0038000E003800 0E0038000E0078000E0078000700BC0003833F8000FC3F80>25 21 127 148 31 I[23 21 127 148 29 I[33 21 127 148 40 I[23 21 127 148 29 I[23 31 127 148 29 I[24 1 128 140 28 123 D E /Fe 29 118 400 360 dfs[<000C0038007000E001C003C0038007800F000F001E001E003E003C003C007C007C007C00 7800F800F800F800F800F800F800F800F800F800F800F80078007C007C007C003C003C003E001E 001E000F000F000780038003C001C000E000700038000C>14 49 124 164 24 40 D[14 49 125 164 24 I[<3C007E00FF00FF00FF80FF807F803D800180018003000300070006000C001C003800 2000>9 18 124 135 18 44 D[<3C7EFFFFFFFF7E3C0000000000003C7EFFFFFFFF7E3C>8 22 124 149 18 58 D[<0001FF0040001FFFC1C0007F80F3C001FC001FC003F0000FC007E00007 C00FC00003C01FC00003C03F800001C03F800001C07F800000C07F000000C07F000000C0FF0000 0000FF00000000FF00000000FF00000000FF00000000FF00000000FF00000000FF000000007F00 0000007F000000C07F800000C03F800000C03F800001C01FC00001800FC000018007E000030003 F000060001FC001C00007F807800001FFFE0000001FF0000>34 34 125 161 46 67 D[38 34 126 161 49 I[33 34 126 161 42 I[<0001FF0020001FFFE0E0007F 8079E001FC001FE003F80007E007E00003E00FC00001E01FC00001E03F800000E03F800000E07F 800000607F000000607F00000060FF00000000FF00000000FF00000000FF00000000FF00000000 FF00000000FF0007FFFEFF0007FFFE7F00000FE07F00000FE07F80000FE03F80000FE03F80000F E01FC0000FE00FE0000FE007E0000FE003F8000FE001FC001FE0007F8073E0001FFFE1E00001FF 8060>39 34 125 161 50 71 D[ 19 34 127 161 24 73 D[47 34 125 161 60 77 D[<0007FE0000003FFFC00000FE07F00003F801FC0007F000FE000FE0007F001FC0003F801F 80001F803F80001FC03F80001FC07F00000FE07F00000FE07F00000FE0FF00000FF0FF00000FF0 FF00000FF0FF00000FF0FF00000FF0FF00000FF0FF00000FF0FF00000FF0FF00000FF07F00000F E07F80001FE07F80001FE03F80001FC01FC0003F801FC0003F800FE0007F0007F000FE0003F801 FC0000FE07F000003FFFC0000007FE0000>36 34 125 161 48 79 D[33 34 126 161 43 I[40 34 126 161 48 82 D[<01FE020007FFCE001F01FE003C007E003C001E0078000E0078000E00F8 000600F8000600FC000600FC000000FF000000FFF000007FFF80003FFFE0003FFFF8001FFFFC00 07FFFE0003FFFF00003FFF000001FF0000003F8000001F8000001F80C0000F80C0000F80C0000F 80E0000F00E0000F00F0001E00FC001C00FF807800E7FFF000807FC000>25 34 125 161 36 I[39 34 126 161 49 85 D[27 35 126 162 36 98 D[<00FF8007FFE00F83F01F03F03E03F07E03F07C01E07C0000FC0000FC00 00FC0000FC0000FC0000FC00007C00007E00007E00003F00301F00600FC0E007FF8000FE00>20 22 126 149 28 I[<00FE0007FF800F83E01F01E03E00F07E00F07C00F8FC00F8FC0078FFFFF8 FFFFF8FC0000FC0000FC0000FC00007E00007E00183E00381F00300F80F003FFC000FF00>21 22 126 149 29 101 D[<001F8000FFE001F1F003E3F007E3F00FC3F00FC1E00FC0000FC0000F C0000FC0000FC0000FC000FFFE00FFFE000FC0000FC0000FC0000FC0000FC0000FC0000FC0000F C0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0007FFC007FFC00> 20 35 126 162 20 I[<00FE0F8003FF9FC00F83E3C01F01F3C01E00F0003E00F8003E00F8003E 00F8003E00F8003E00F8001E00F0001F01F0000F83E0000BFF800008FE00001800000018000000 1C0000001FFFE0001FFFFC000FFFFF0007FFFF001FFFFF807C001FC078000FC0F80007C0F80007 C0F80007C07C000F803E001F001F807E000FFFFC0001FFE000>26 33 127 149 32 I[<0E003F807F807F807F807F803F800E00000000000000000000000000FF80FF801F80 1F801F801F801F801F801F801F801F801F801F801F801F801F801F801F801F801F80FFF0FFF0> 12 36 126 163 17 105 D[12 35 126 162 17 108 D[27 22 125 149 36 110 D[<00FF0007FFE00F81F01F00F83E007C7C003E7C003E7C003EFC003FFC003F FC003FFC003FFC003FFC003FFC003F7C003E7E007E3E007C1F00F80F81F007FFE000FF00>24 22 126 149 32 I[27 32 126 149 36 I[20 22 126 149 27 114 D[<07F9801FFF80380780700380F00180F00180F80000FF0000FFF8007FFE003FFF001FFF8007 FF80003FC0C007C0C003C0E003C0E003C0F00380FC0F00EFFE00C3F800>18 22 126 149 26 I[<00C00000C00000C00000C00001C00001C00003C00007C0000FC0001FC000 FFFF00FFFF000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC000 0FC1800FC1800FC1800FC1800FC18007C18007E30003FE0000FC00>17 32 127 159 24 I[27 22 125 149 36 I E /Ff 65 126 438 432 dfs[<00F0000003F8000003FC000007FC0000071E00000F0E00000E0E00000E 0E00000E0E00000E0E00000E0E00000E1E7FC00E3CFFC00E7CFFC007787FC007F0380007F03800 07E0380007C070000F8070001F8070003FC0E0007DC0E00078E0E00078E1C000F071C000F07B80 00F03B8000F01F0000F01F01C0F00E01C0781F81C0787FC3C03FFBFF803FF1FF801FE0FF000780 3C00>26 37 126 164 31 38 D[<000F001F003E007C00F801F003E007C00F800F001E001E003C 003C003C00780078007800F000F000F000F000F000F000F000F000F000F000F000780078007800 3C003C003C001E001E000F000F8007C003E001F000F8007C003F001F000F>16 47 119 169 31 40 D[<7000F8007C003E001F000F8007C003E001F000F000780078003C003C00 3C001E001E001E000F000F000F000F000F000F000F000F000F000F000F001E001E001E003C003C 003C0078007800F001F003E007C00F801F003E007C00F8007000>16 47 123 169 31 I[<000E0000001F0000001F0000001F0000001F0000001F0000001F0000001F0000 001F0000001F0000001F00007FFFFF80FFFFFFC0FFFFFFC0FFFFFFC07FFFFF80001F0000001F00 00001F0000001F0000001F0000001F0000001F0000001F0000001F0000001F0000000E0000>26 27 126 159 31 43 D[<1C003F007F007F807F803F801F8007800F800F001F007E00FC00F80060 00>9 15 117 134 31 I[<7FFFFEFFFFFFFFFFFFFFFFFF7FFFFE>24 5 125 148 31 I[<387CFEFEFE7C38>7 7 116 134 31 I[<00000E00001F00001F00003F00003E0000 7E00007C00007C0000FC0000F80001F80001F00003F00003E00007E00007C0000FC0000F80000F 80001F80001F00003F00003E00007E00007C0000FC0000F80001F80001F00001F00003F00003E0 0007E00007C0000FC0000F80001F80001F00003F00003E00003E00007E00007C0000FC0000F800 00F80000700000>24 47 125 169 31 I[<007E0001FF8003FFC007FFE00FC3F01F00F81E0078 3E007C3C003C7C003E78001E78001E78001EF0000FF0000FF0000FF0000FF0000FF0000FF0000F F0000FF0000FF0000FF8001F78001E78001E78001E7C003E3C003C3E007C1F00F81F81F80FC3F0 07FFE003FFC001FF80007E00>24 37 125 164 31 I[<00700000700000F00000F00001F00003 F00007F0007FF000FFF000FEF000F8F00000F00000F00000F00000F00000F00000F00000F00000 F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000 F00000F0007FFFE0FFFFF0FFFFF07FFFE0>20 37 122 164 31 I[<00FE0003FFC00FFFE01FFF F83E03FC7C007C78003EF0001EF0000FF8000FF8000F70000F00000F00000F00001E00001E0000 1E00003C00007C0000F80001F00003E00007C0000F80001F00003E00007C0001F80003F00007C0 000F800F1F000F3E000F7FFFFFFFFFFFFFFFFF7FFFFF>24 37 125 164 31 I[<1C3E7F7F7F3E1C0000000000000000000000001C3E7E7F7F3F1F0F1F1E3E7CF8F060>8 34 117 153 31 59 D[<00000E00001F00007F0000FF0003FE0007FC001FF0003FE000FF8001FF 0007FC000FF8003FE0007FC000FF0000FE0000FF00007FC0003FE0000FF80007FC0001FF0000FF 80003FE0001FF00007FC0003FE0000FF00007F00001F00000E>24 31 125 161 31 I[<7FFFFF80FFFFFFC0FFFFFFC0FFFFFFC07FFFFF800000000000000000000000000000 0000000000007FFFFF80FFFFFFC0FFFFFFC0FFFFFFC07FFFFF80>26 15 126 153 31 I[<700000F80000FE0000FF00007FC0003FE0000FF80007FC0001FF0000FF80003F E0001FF00007FC0003FE0000FF00007F0000FF0003FE0007FC001FF0003FE000FF8001FF0007FC 000FF8003FE0007FC000FF0000FE0000F80000700000>24 31 125 161 31 I[<001E0000003F0000003F0000003F00000073800000738000007380000073800000738000 00F3C00000F3C00000F3C00000E1C00001E1E00001E1E00001E1E00001E1E00001E1E00003C0F0 0003C0F00003C0F00003C0F00007C0F80007FFF80007FFF80007FFF80007FFF8000F003C000F00 3C000F003C000F003C000F003C001E001E00FFC0FFC0FFE1FFC0FFE1FFC0FFC0FFC0>26 37 126 164 31 65 D[25 37 126 164 31 I[<001F81C0007FE1C001FFFBC003FFFFC007F03FC00FC01FC01F80 0FC01F0007C03E0007C03C0003C07C0003C0780003C0780003C078000000F0000000F0000000F0 000000F0000000F0000000F0000000F0000000F0000000F00000007800000078000000780003C0 7C0003C03C0003C03E0003C01F0007801F8007800FC00F0007F03F0003FFFE0001FFFC00007FF0 00001FC000>26 37 126 164 31 I[<7FFF8000FFFFE000FFFFF8007FFFFC000F00FE000F003E 000F001F000F000F800F000F800F0007800F0007C00F0003C00F0003C00F0003E00F0001E00F00 01E00F0001E00F0001E00F0001E00F0001E00F0001E00F0001E00F0001E00F0003E00F0003C00F 0003C00F0003C00F0007C00F000F800F000F800F001F000F003E000F00FE007FFFFC00FFFFF800 FFFFF0007FFF8000>27 37 127 164 31 I[27 37 126 164 31 I[26 37 126 164 31 I[<003F070000FFC70001FFEF0007FF FF0007E0FF000F807F001F003F001E001F003E001F003C000F007C000F0078000F0078000F00F8 000000F0000000F0000000F0000000F0000000F0000000F0000000F001FFC0F001FFE0F001FFE0 F801FFC078000F0078000F0078001F003C001F003E001F001E001F001F003F000F807F0007E0FF 0007FFFF0001FFEF0000FFCF00003F0F00>27 37 126 164 31 I[<7FE07FE0FFF0FFF0FFF0FF F07FE07FE00F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F00 0F000F000F000F000F000F000F000FFFFF000FFFFF000FFFFF000FFFFF000F000F000F000F000F 000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F00 0F000F007FE07FE0FFF0FFF0FFF0FFF07FE07FE0>28 37 127 164 31 I[<7FFFF8FFFFFCFFFF FC7FFFF80078000078000078000078000078000078000078000078000078000078000078000078 000078000078000078000078000078000078000078000078000078000078000078000078000078 000078000078000078000078007FFFF8FFFFFCFFFFFC7FFFF8>22 37 124 164 31 I[<7FC07FC0FFE0FFC0FFE0FFC07FC07FC00E001C000E0038000E0078000E00F0000E00 E0000E01C0000E03C0000E0780000E0700000E0E00000E1E00000E3C00000E3E00000E7F00000E F700000EE780000FC380000FC1C0000F81C0000F00E0000E00E0000E0070000E0070000E003800 0E0038000E001C000E001C000E000E000E000E007FC01FE0FFE03FE0FFE03FE07FC01FE0>27 37 127 164 31 75 D[<7FFC0000FFFE0000FFFE00007FFC000007800000078000000780000007 800000078000000780000007800000078000000780000007800000078000000780000007800000 078000000780000007800000078000000780000007800000078000000780000007800000078001 80078003C0078003C0078003C0078003C0078003C0078003C07FFFFFC0FFFFFFC0FFFFFFC07FFF FFC0>26 37 126 164 31 I[28 37 127 164 31 I[<7F00FF80FF81FFC0FF81FFC07FC0FF800EC01C000EC01C 000EE01C000E601C000E601C000E701C000E701C000E301C000E381C000E381C000E381C000E18 1C000E1C1C000E1C1C000E0C1C000E0E1C000E0E1C000E061C000E071C000E071C000E071C000E 031C000E039C000E039C000E019C000E019C000E01DC000E00DC000E00DC007FC0FC00FFE07C00 FFE07C007FC03C00>26 37 126 164 31 I[<03FFC01FFFF83FFFFC3FFFFC7E007E7C003E7800 1E78001EF8001FF0000FF0000FF0000FF0000FF0000FF0000FF0000FF0000FF0000FF0000FF000 0FF0000FF0000FF0000FF0000FF0000FF0000FF0000FF0000FF8001F78001E78001E7C003E7F00 FE3FFFFC3FFFFC1FFFF803FFC0>24 37 125 164 31 I[25 37 126 164 31 I[<7FFF0000FFFFE000FFFFF000 7FFFF8000F01FC000F007E000F001E000F001F000F000F000F000F000F000F000F000F000F001F 000F001E000F007E000F01FC000FFFF8000FFFF0000FFFE0000FFFF0000F01F8000F0078000F00 7C000F003C000F003C000F003C000F003C000F003C000F003C000F003C000F003C780F003C780F 003E787FE01FF0FFF01FF0FFF00FE07FE003C0>29 37 127 164 31 82 D[<01FC1C07FF9C0FFFFC3FFFFC3E03FC7C00FC78007CF0003CF0003CF0003CF0003CF0000078 00007C00003E00003FE0001FFE0007FFC001FFF0001FF80001FC00007C00001E00001E00000F00 000F70000FF0000FF0000FF0001FF8001EFC003EFF00FCFFFFF8FFFFF0E3FFE0E0FF80>24 37 125 164 31 I[<7FFFFFC0FFFFFFC0FFFFFFC0FFFFFFC0F01E03C0F01E03C0F01E03C0F01E 03C0F01E03C0F01E03C0001E0000001E0000001E0000001E0000001E0000001E0000001E000000 1E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000 001E0000001E0000001E0000001E0000001E0000001E000003FFF00007FFF80007FFF80003FFF0 00>26 37 126 164 31 I[28 37 127 164 31 I[<7FC03FE0FFE07FF0FFE07FF07FC03FE00F000F000F000F00 0F000F000F000F0007801E0007801E0007801E0007801E0003C03C0003C03C0003C03C0003C03C 0003E07C0001E0780001E0780001E0780001E0780000F0F00000F0F00000F0F00000F0F0000070 E0000079E0000079E0000079E0000039C0000039C0000039C0000039C000001F8000001F800000 1F8000000F0000>28 37 127 164 31 I[28 37 127 164 31 I[<3FFFFF7FFFFF7FFFFF7FFFFF78001E78003C 78007C7800787800F07801F00001E00003E00003C0000780000F80000F00001E00003E00003C00 007C0000780000F00001F00001E00003C00007C00007800F0F800F0F000F1E000F3E000F3C000F 78000FFFFFFFFFFFFFFFFFFFFFFFFF>24 37 125 164 31 90 D[16 47 116 169 31 I[<700000F80000F80000FC00007C00007E00003E00003E0000 3F00001F00001F80000F80000FC00007C00007E00003E00003F00001F00001F00001F80000F800 00FC00007C00007E00003E00003F00001F00001F80000F80000F80000FC00007C00007E00003E0 0003F00001F00001F80000F80000FC00007C00007C00007E00003E00003F00001F00001F00000E >24 47 125 169 31 I[16 47 126 169 31 I[<7FFFFEFFFFFFFFFFFFFFFFFF7FFFFE>24 5 125 126 31 95 D[<07FC00001FFF00003FFFC0003FFFE0003E03F0001C01F0000000F8000000780000007800 00007800007FF80003FFF8000FFFF8003FE078007E00780078007800F0007800F0007800F00078 00F00078007800F8007E03F8003FFFFFE03FFFFFE00FFE3FE003F00FE0>27 26 125 153 31 97 D[27 37 127 164 31 I[<007FC001FFF007FFF80FFFF81F80F83E00703C00007800007800 00F80000F00000F00000F00000F00000F00000F00000F800007800007C00783E00783F00F81FC1 F00FFFE007FFE001FF80007E00>21 26 123 153 31 I[<0007FC000007FC000007FC000007FC 0000003C0000003C0000003C0000003C0000003C0000003C0000003C0000FC3C0003FF3C0007FF BC000FFFFC001F81FC003E00FC003C007C007C003C0078003C00F8003C00F0003C00F0003C00F0 003C00F0003C00F0003C00F0003C00F8003C0078007C0078007C003C00FC003E01FC001F83FC00 1FFFFFE007FFBFE003FE3FE000F83FE0>27 37 126 164 31 I[<007F0001FFC007FFE00FFFF0 1F81F83F00783C003C7C003C78001E78001EFFFFFEFFFFFEFFFFFEFFFFFEF00000F00000780000 7800007C001E3E001E1F803E1FE07C0FFFF803FFF001FFE0003F80>23 26 125 153 31 I[<0001F80007FC000FFE001FFE003E3E007C1C0078000078000078000078000078 007FFFFCFFFFFCFFFFFCFFFFFC0078000078000078000078000078000078000078000078000078 000078000078000078000078000078000078000078000078000078007FFFF87FFFF87FFFF87FFF F8>23 37 126 164 31 I[<00FC0F8003FF3FC007FFFFE00FFFFFE00F87E1C01F03E0001E01E0 003C00F0003C00F0003C00F0003C00F0003C00F0003C00F0001E01E0001F03E0000F87C0000FFF C0001FFF80001FFF00001CFC00001C0000001C0000000E0000000FFFE0001FFFF8003FFFFE003C 001F007800078070000380E00001C0E00001C0E00001C0E00001C0700003807C000F803F003F00 1FFFFE000FFFFC0003FFF000007F8000>27 40 126 153 31 I[28 37 127 164 31 I[<00300000780000FC00 00FC000078000030000000000000000000000000000000000000007FFC007FFC007FFC007FFC00 003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00 003C00003C00003C00003C00003C007FFFFCFFFFFEFFFFFE7FFFFC>23 38 124 165 31 I[ 27 37 126 164 31 107 D[24 37 125 164 31 I[29 26 128 153 31 I[28 26 127 153 31 I[<00FC0003FF0007FF801FFFE01F87E03E01F07C00F87800787800 78F0003CF0003CF0003CF0003CF0003CF0003CF0003CF8007C7800787C00F87C00F83E01F01F87 E01FFFE007FF8003FF0000FC00>22 26 124 153 31 I[27 39 127 153 31 I[26 26 126 153 31 114 D[<03FC700FFFF03FFFF07FFFF07C03F0F801F0F000F0F000F0F000F07C 00007FE0001FFF0007FFC000FFE00003F00000F870003CF0003CF0003CF8003CFC007CFF01F8FF FFF0FFFFF0E7FFC0E1FE00>22 26 124 153 31 I[<0070000000F0000000F0000000F0000000 F0000000F0000000F000007FFFFE00FFFFFE00FFFFFE00FFFFFE0000F0000000F0000000F00000 00F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F007 8000F0078000F0078000F0078000F80F00007C1F00007FFE00003FFC00001FF8000007E000>25 33 127 160 31 I[28 26 127 153 31 I[<7FE07FE0FFF0FFF0FFF0FFF07FE07FE007000E0007000E0007801E000380 1C0003801C0003C03C0001C0380001C0380001E0780000E0700000E0700000E070000070E00000 70E0000070E0000039C0000039C0000039C000001F8000001F8000001F8000000F0000>28 26 127 153 31 I[<7FF0FFC07FF1FFE07FF1FFE07FF0FFC003C0380001E0700000E0F0000070 E0000079C000003FC000001F8000000F0000000E0000000F0000001F8000003B80000039C00000 70E00000E0F00001E0700001C0380003801C007FE07FE0FFF0FFF0FFF0FFF07FE07FE0>28 26 127 153 31 120 D[<3FFFFF807FFFFF807FFFFF807FFFFF8078003F0078007E007800FC00 7801F8000003F0000007E000000FC000001F8000003F0000007E000000FC000001F8000003F000 0007E007800FC007801F8007803F0007807E000780FFFFFF80FFFFFF80FFFFFF80FFFFFF80>25 26 126 153 31 122 D[<00007F0003FF000FFF001FFF001F00003C00003C00003C00003C0000 3C00003C00003C00003C00003C00003C00003C00003C00003C00003C00007C0000F8007FF800FF E000FFC000FFE0007FF80000F800007C00003C00003C00003C00003C00003C00003C00003C0000 3C00003C00003C00003C00003C00003C00003C00001F00001FFF000FFF0003FF00007F>24 47 125 169 31 I[<7E0000FFC000FFF000FFF80000F800003C00003C00003C00003C00003C00 003C00003C00003C00003C00003C00003C00003C00003C00003C00003E00001F00001FFE0007FF 0003FF0007FF001FFE001F00003E00003C00003C00003C00003C00003C00003C00003C00003C00 003C00003C00003C00003C00003C00003C0000F800FFF800FFF000FFC0007E0000>24 47 125 169 31 125 D E /Fg 47 123 438 432 dfs[<0001FF81FE00001FFFEFFF80007F80FF 87C000FC00FE0FE001F801FE0FE003F801FC0FE007F001FC0FE007F001FC07C007F001FC000007 F001FC000007F001FC000007F001FC000007F001FC000007F001FC000007F001FC0000FFFFFFFF F800FFFFFFFFF800FFFFFFFFF80007F001FC000007F001FC000007F001FC000007F001FC000007 F001FC000007F001FC000007F001FC000007F001FC000007F001FC000007F001FC000007F001FC 000007F001FC000007F001FC000007F001FC000007F001FC000007F001FC000007F001FC000007 F001FC000007F001FC000007F001FC000007F001FC00007FFF1FFFE0007FFF1FFFE0007FFF1FFF E000>43 42 127 169 41 11 D[<00030007000E001C0038007000F001E003C003C007C007800F 800F001F001F003F003E003E007E007E007E007C007C00FC00FC00FC00FC00FC00FC00FC00FC00 FC00FC00FC00FC007C007C007E007E007E003E003E003F001F001F000F000F80078007C003C003 C001E000F000700038001C000E00070003>16 60 122 172 27 40 D[<8000C000E00070003800 1C001E000F000780078007C003C003E001E001F001F001F800F800F800FC00FC00FC007C007C00 7E007E007E007E007E007E007E007E007E007E007E007E007C007C00FC00FC00FC00F800F801F8 01F001F001E003E003C007C0078007800F001E001C0038007000E000C0008000>15 60 123 172 27 I[<1C007F007F00FF80FFC0FFC07FC07FC01CC000C000C00180018001800300 030006000C00180030002000>10 21 123 136 19 44 D[17 6 127 144 23 I[<000E00001E00007E0007FE00FFFE00FFFE00F8FE0000FE0000FE 0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE 0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE 0000FE007FFFFE7FFFFE7FFFFE>23 39 123 166 34 49 D[<000003800000000007C000000000 07C0000000000FE0000000000FE0000000000FE0000000001FF0000000001FF0000000003FF800 0000003FF8000000003FF80000000073FC0000000073FC00000000F3FE00000000E1FE00000000 E1FE00000001C0FF00000001C0FF00000003C0FF80000003807F80000007807FC0000007003FC0 000007003FC000000E003FE000000E001FE000001E001FF000001C000FF000001FFFFFF000003F FFFFF800003FFFFFF80000780007FC0000700003FC0000700003FC0000E00001FE0000E00001FE 0001E00001FF0001C00000FF0001C00000FF00FFFE001FFFFEFFFE001FFFFEFFFE001FFFFE>47 41 126 168 53 65 D[41 41 125 168 50 I[<00003FF001800003FFFE0380000FFFFF8780003FF007DF8000FF8001FF8001FE00007F80 03FC00003F8007F000001F800FF000000F801FE0000007801FE0000007803FC0000007803FC000 0003807FC0000003807F80000003807F8000000000FF8000000000FF8000000000FF8000000000 FF8000000000FF8000000000FF8000000000FF8000000000FF8000000000FF80000000007F8000 0000007F80000000007FC0000003803FC0000003803FC0000003801FE0000003801FE000000700 0FF00000070007F000000E0003FC00001E0001FE00003C0000FF8000F800003FF007E000000FFF FFC0000003FFFF000000003FF80000>41 41 124 168 51 I[ 45 41 125 168 54 I[39 41 125 168 46 I[36 41 125 168 44 I[<00007FE003000003FFFC0700001FFFFF0F00003FF00FFF0000FF 8001FF0001FE0000FF0003F800003F0007F000003F000FF000001F001FE000000F001FE000000F 003FC000000F003FC0000007007FC0000007007F80000007007F8000000000FF8000000000FF80 00000000FF8000000000FF8000000000FF8000000000FF8000000000FF8000000000FF80000000 00FF8001FFFFF87F8001FFFFF87F8001FFFFF87FC00000FF003FC00000FF003FC00000FF001FE0 0000FF001FE00000FF000FF00000FF0007F00000FF0003F80000FF0001FE0000FF0000FF8001FF 00003FF007BF00001FFFFF1F000003FFFE0F0000007FF00300>45 41 124 168 55 I[47 41 125 168 55 I[22 41 126 168 26 I[47 41 125 168 55 75 D[34 41 125 168 42 I[58 41 125 168 66 I[47 41 125 168 55 I[<0000FFE000000007FFFC0000003FC07F8000007F001FC00001FC0007F00003F80003 F80007F00001FC000FF00001FE001FE00000FF001FE00000FF003FC000007F803FC000007F807F C000007FC07F8000003FC07F8000003FC07F8000003FC0FF8000003FE0FF8000003FE0FF800000 3FE0FF8000003FE0FF8000003FE0FF8000003FE0FF8000003FE0FF8000003FE0FF8000003FE0FF 8000003FE07F8000003FC07FC000007FC07FC000007FC03FC000007F803FC000007F801FE00000 FF001FE00000FF000FF00001FE0007F00001FC0003F80003F80001FC0007F00000FF001FE00000 3FC07F8000000FFFFE00000000FFE00000>43 41 124 168 53 I[ 39 41 125 168 48 I[48 41 125 168 53 82 D[<007F806003FFF0E007FFF9E00F807FE01F001FE03E0007E07C0003E07C0001E0FC00 01E0FC0001E0FC0000E0FE0000E0FE0000E0FF000000FFC000007FFE00007FFFE0003FFFFC001F FFFE000FFFFF8007FFFFC003FFFFE000FFFFE00007FFF000007FF000000FF8000007F8000003F8 600001F8E00001F8E00001F8E00001F8F00001F0F00001F0F80003F0FC0003E0FF0007C0FFE01F 80F3FFFF00E0FFFE00C01FF000>29 41 124 168 39 I[<7FFFFFFFFFC07FFFFFFFFFC07FFFFF FFFFC07F803FC03FC07E003FC007C078003FC003C078003FC003C070003FC001C0F0003FC001E0 F0003FC001E0E0003FC000E0E0003FC000E0E0003FC000E0E0003FC000E0E0003FC000E000003F C0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC00000 00003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003F C0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC00000 00003FC0000000003FC00000007FFFFFE000007FFFFFE000007FFFFFE000>43 40 126 167 49 I[46 41 125 168 54 I[<7FFFF81FFFF07FFFF81FFFF07FFFF81FFFF001FF0000780000FF8000F000007FC001E00000 7FC001C000003FE003C000001FF0078000000FF80F0000000FF80E00000007FC1E00000003FE3C 00000003FE7800000001FF7000000000FFF0000000007FE0000000007FC0000000003FE0000000 001FF0000000001FF0000000001FF8000000001FFC000000003FFE000000007BFE00000000F1FF 00000000E0FF80000001E0FFC0000003C07FC0000007803FE0000007001FF000000F001FF00000 1E000FF800003C0007FC0000380003FE0000780003FE0000F00001FF0000E00000FF80FFFF801F FFFEFFFF801FFFFEFFFF801FFFFE>47 41 126 168 53 88 D[<3FFFFFFF803FFFFFFF803FFFFF FF803FF000FF003F8001FE003F0003FE003E0003FC003C0007F8007C000FF80078000FF0007800 1FE00070003FE00070003FC00070007F80007000FF80000000FF00000001FE00000003FE000000 03FC00000007F80000000FF80000000FF00000001FE00000003FE00000003FC001C0007F8001C0 00FF8001C000FF0001C001FE0001C003FE0003C003FC0003C007F80003C00FF80007800FF00007 801FE0000F803FE0001F803FC0007F807F8001FF80FFFFFFFF80FFFFFFFF80FFFFFFFF80>34 41 124 168 43 90 D[<01FF800007FFF0000F81F8001FC07E001FC07E001FC03F000F803F8007 003F8000003F8000003F8000003F80000FFF8000FFFF8007FC3F800FE03F803F803F803F003F80 7F003F80FE003F80FE003F80FE003F80FE003F807E007F807F00DF803F839FFC0FFF0FFC01FC03 FC>30 27 126 154 33 97 D[33 42 126 169 39 I[<001FF80000FFFE0003F01F0007E03F800FC03F801F803F803F801F007F80 0E007F0000007F000000FF000000FF000000FF000000FF000000FF000000FF000000FF0000007F 0000007F0000007F8000003F8001C01F8001C00FC0038007E0070003F01E0000FFFC00001FE000 >26 27 126 154 31 I[<00003FF80000003FF80000003FF800000003F800000003F800000003 F800000003F800000003F800000003F800000003F800000003F800000003F800000003F8000000 03F800000003F800001FE3F80000FFFBF80003F03FF80007E00FF8000FC007F8001F8003F8003F 8003F8007F0003F8007F0003F8007F0003F800FF0003F800FF0003F800FF0003F800FF0003F800 FF0003F800FF0003F800FF0003F8007F0003F8007F0003F8007F0003F8003F8003F8001F8003F8 000F8007F80007C00FF80003F03BFF8000FFF3FF80003FC3FF80>33 42 126 169 39 I[<003FE00001FFF80003F07E0007C01F000F801F801F800F803F800FC07F000FC0 7F0007C07F0007E0FF0007E0FF0007E0FFFFFFE0FFFFFFE0FF000000FF000000FF0000007F0000 007F0000007F0000003F8000E01F8000E00FC001C007E0038003F81F0000FFFE00001FF000>27 27 126 154 32 I[<0007F0003FFC00FE3E01F87F03F87F03F07F07F07F07F03E07F00007F000 07F00007F00007F00007F00007F000FFFFC0FFFFC0FFFFC007F00007F00007F00007F00007F000 07F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F000 07F00007F00007F0007FFF807FFF807FFF80>24 42 126 169 21 I[<00FF81F003FFE7F80FC1 FE7C1F80FC7C1F007C383F007E107F007F007F007F007F007F007F007F007F007F007F007F003F 007E001F007C001F80FC000FC1F8001FFFE00018FF800038000000380000003C0000003E000000 3FFFF8001FFFFF001FFFFF800FFFFFC007FFFFE01FFFFFF03E0007F07C0001F8F80000F8F80000 F8F80000F8F80000F87C0001F03C0001E01F0007C00FC01F8003FFFE00007FF000>30 40 126 154 34 I[33 42 125 169 39 I[<07000FC01FE03FE03FE03FE01FE00FC007000000000000000000000000000000FFE0 FFE0FFE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00F E00FE00FE00FE0FFFEFFFEFFFE>15 43 125 170 20 I[31 42 126 169 37 107 D[15 42 125 169 20 I[33 27 125 154 39 110 D[<003FE00001FFFC0003F07E000FC01F801F800F C03F800FE03F0007E07F0007F07F0007F07F0007F0FF0007F8FF0007F8FF0007F8FF0007F8FF00 07F8FF0007F8FF0007F8FF0007F87F0007F07F0007F03F800FE03F800FE01F800FC00FC01F8007 F07F0001FFFC00003FE000>29 27 126 154 34 I[33 39 126 154 39 I[24 27 126 154 28 114 D[<03FE300FFFF01E03F03800F0700070F00070F000 70F80070FC0000FFE0007FFE007FFF803FFFE01FFFF007FFF800FFF80003FC0000FC60007CE000 3CF0003CF00038F80038FC0070FF01E0F7FFC0C1FF00>22 27 126 154 27 I[<00700000700000700000700000F00000F00000F00001F00003F00003F00007F0001FFFF0 FFFFF0FFFFF007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F000 07F00007F00007F03807F03807F03807F03807F03807F03803F03803F87001F86000FFC0001F80 >21 38 127 165 27 I[33 27 125 154 39 I[33 39 127 154 37 121 D[<3FFFFF803FFFFF803F007F003C00FE0038 01FE007803FC007803F8007007F800700FF000700FE000001FC000003FC000007F8000007F0000 00FF000001FE038001FC038003F8038007F803800FF007800FE007801FE007003FC00F003F801F 007F007F00FFFFFF00FFFFFF00>25 27 126 154 31 I E /Fh 25 87 438 432 dfs[<387C7EFC7C38>7 6 123 133 17 46 D[<0000C0000180000780007F8007F7800787 80000780000F00000F00000F00000F00000F00000F00001E00001E00001E00001E00001E00001E 00003C00003C00003C00003C00003C00003C0000780000780000780000780000780000780000F0 0000F00000F00000F00000F00000F00001F000FFFFE0FFFFE0>19 40 122 167 30 49 D[<0003FC00000FFF00003C0F80006007C0008003E001E003E001F003E003F003E0 03F003E001E003E000C007E0000007C0000007C000000F8000001F0000001E00000038000000E0 00001FE0000000780000001C0000001E0000000F0000000F8000000F8000000FC000000FC00000 0FC018000FC07C000FC07C000F80FC001F80FC001F80F8001F0040003E0040003E0020007C0038 00F8001E03E00007FFC00000FE0000>27 41 125 167 30 51 D[<00C000C000F0078000FFFF00 00FFFE0001FFF80001FFE000011F00000100000001000000010000000200000002000000020000 000200000002000000021F800004E0E000050070000600380004003C0000003C0000001E000000 1E0000001E0000001E0000003E0000003E0070003E00F8003E00F8003E00F8007C00F0007C0080 0078008000F8008000F000C001E0004003C00060078000381F00000FFC000007F00000>26 41 123 167 30 53 D[<0000FC000007FF00000F0180003C01C0007003C000E007C001C007C003 C0078007800000070000000F0000001F0000001E0000003E0000003E0000003C1F80007C606000 7C8038007D001C00FA001C00FA001E00FC000F00FC000F00F8000F00F8000F00F8001F00F0001F 00F0001F00F0001F00F0001F00F0001E00F0003E00F0003C00F0003C0070007800700070003800 E0003C01C0001E0780000FFE000003F80000>26 41 123 167 30 I[<080000000C0000001FFF FFC01FFFFFC01FFFFF803FFFFF8020000100600002004000040040000800800010008000200000 004000000040000000800000010000000200000006000000040000000800000018000000380000 00300000007000000060000000E0000000E0000001E0000001C0000003C0000003C0000007C000 0007800000078000000F8000000F8000000F8000000F8000001F0000001F0000001F0000000E00 0000>26 42 120 168 30 I[<0000003000000000300000000070000000007000000000F00000 0000F800000001F800000001F800000002F800000002F800000004FC0000000C7C000000087C00 0000107C000000107C000000207C000000203E000000403E000000403E000000803E000000803E 000001003F000001001F000002001F000002001F000004001F000004001F80000FFFFF80000FFF FF800010000F800010000F800020000F8000200007C000400007C000C00007C000800007C00180 0007C001000007E003800003E00FC00007E0FFF000FFFFFFF000FFFF>40 42 126 169 46 65 D[<01FFFFFF0003FFFFFFC0000FC003F0000F8000F8000F8000FC000F8000 7C000F80007E000F80007E001F00003E001F00003E001F00007E001F00007E001F00007C001F00 00FC003E0000F8003E0001F0003E0003E0003E0007C0003E001F00003FFFFC00007C000F80007C 0007C0007C0003F0007C0001F0007C0000F8007C0000F800F80000FC00F80000FC00F80000FC00 F80000FC00F80000FC00F80000F801F00001F801F00001F801F00003F001F00007E001F0000FC0 01F0001F8003F0007F00FFFFFFFC00FFFFFFE000>39 41 126 168 43 I[<00000FF00100007F FE030001FC07070007E0018E001F80005E003E00003E007C00003E00F800001E01F000001E03E0 00000C07C000000C07C000000C0F8000000C1F8000000C1F0000000C3F000000083F000000007E 000000007E000000007E000000007E00000000FC00000000FC00000000FC00000000FC00000000 FC00000000FC00000000FC00000000FC000000207C000000207C000000207C000000403E000000 403E000000801E000000801F000001000F8000020007C000040003E000180001F0003000007E01 E000003FFF80000007FC0000>40 43 122 169 44 I[<01FFFFFF800003FFFFFFE000000FC003 F800000F80007C00000F80003E00000F80001F00000F80000F80000F80000780001F000007C000 1F000007C0001F000003E0001F000003E0001F000003E0001F000003E0003E000003F0003E0000 03F0003E000003F0003E000003F0003E000003F0003E000003F0007C000003E0007C000007E000 7C000007E0007C000007E0007C000007C0007C000007C000F800000FC000F800000F8000F80000 1F8000F800001F0000F800001F0000F800003E0001F000007C0001F00000F80001F00001F00001 F00003E00001F00007C00001F0001F800003F000FE0000FFFFFFF80000FFFFFFC00000>44 41 126 168 47 I[<01FFFFFFFF03FFFFFFFF000FC0007F000F80000F000F800007000F800007 000F800003000F800003001F000003001F000003001F000001001F000001001F000801001F0008 01003E001000003E001000003E001000003E003000003E00F000003FFFF000007FFFE000007C00 E000007C006000007C006000007C002000007C00200200F800400400F800400400F800000400F8 00000800F800000800F800001801F000001001F000003001F000003001F000007001F00000E001 F00003E003F0000FE0FFFFFFFFC0FFFFFFFFC0>40 41 126 168 42 I[<01FFFFFFFE03FFFFFF FE000FC0007E000F80001E000F80000E000F800006000F800006000F800006001F000006001F00 0006001F000002001F000002001F001002001F001002003E002000003E002000003E002000003E 006000003E01E000003FFFE000007FFFC000007C01C000007C00C000007C00C000007C00400000 7C00400000F800800000F800800000F800000000F800000000F800000000F800000001F0000000 01F000000001F000000001F000000001F000000001F000000003F0000000FFFFE00000FFFFE000 00>39 41 126 168 40 I[<00000FF00100007FFE030001FC07070007E0018E001F80005E003E 00003E007C00003E00F800001E01F000001E03E000000C07C000000C07C000000C0F8000000C1F 8000000C1F0000000C3F000000083F000000007E000000007E000000007E000000007E00000000 FC00000000FC00000000FC00000000FC00000000FC0001FFFFFC0001FFFFFC000003F0FC000003 E07C000003E07C000003E07C000003E03E000003E03E000007C01F000007C01F000007C00F8000 07C007C0000FC003E00013C001F8002380007E01C180003FFF00800007FC0000>40 43 122 169 48 I[<01FFFF03FFFE03FFFF07FFFE000FC0001F80000F80001F00000F80001F00 000F80001F00000F80001F00000F80001F00001F00003E00001F00003E00001F00003E00001F00 003E00001F00003E00001F00003E00003E00007C00003E00007C00003E00007C00003E00007C00 003E00007C00003FFFFFFC00007FFFFFF800007C0000F800007C0000F800007C0000F800007C00 00F800007C0000F80000F80001F00000F80001F00000F80001F00000F80001F00000F80001F000 00F80001F00001F00003E00001F00003E00001F00003E00001F00003E00001F00003E00001F000 03E00003F00007E000FFFF81FFFF00FFFF81FFFF00>47 41 126 168 46 I[<03FFFF03FFFF000FC0000F80000F80000F80000F80000F80001F00001F00001F00001F0000 1F00001F00003E00003E00003E00003E00003E00003E00007C00007C00007C00007C00007C0000 7C0000F80000F80000F80000F80000F80000F80001F00001F00001F00001F00001F00001F00003 F000FFFFC0FFFFC0>24 41 126 168 22 I[<01FFFF800003FFFF8000000FC00000000F800000 000F800000000F800000000F800000000F800000001F000000001F000000001F000000001F0000 00001F000000001F000000003E000000003E000000003E000000003E000000003E000000003E00 0000007C000000007C000000007C000000007C000000007C000000007C00002000F800004000F8 00004000F800004000F800008000F800008000F800018001F000018001F000030001F000030001 F000070001F0000E0001F0003E0003F001FE00FFFFFFFC00FFFFFFFC00>35 41 126 168 38 76 D[<01FF80000007FF8003FFC000000BFF80000BC000000FE000000BC00000 17C000000BC0000017C0000009E0000027C0000009E0000027C0000009E0000047C0000011E000 004F80000011E000008F80000010F000010F80000010F000010F80000010F000020F80000010F0 00020F800000207800041F000000207800041F000000207800081F000000207800081F00000020 7800101F000000203C00201F000000403C00203E000000403C00403E000000403C00403E000000 401E00803E000000401E00803E000000401E01003E000000801E01007C000000801E02007C0000 00800F04007C000000800F04007C000000800F08007C000000800F08007C00000100079000F800 000100079000F80000010007A000F80000010007A000F80000010007C000F800000300038000F8 00000FC0038001F80000FFF803007FFFC000FFF803007FFFC000>57 41 126 168 56 I[<01FFC0003FFE03FFC0007FFE000FE00007E0000BF00003800009F00001000009 F80001000008F80001000008FC00010000107C00020000107E00020000103E00020000103F0002 0000101F00020000101F80020000200F80040000200FC00400002007C00400002007E004000020 03E00400002003F00400004001F00800004001F80800004000F80800004000FC08000040007C08 000040007E08000080003E10000080003F10000080001F10000080001F90000080000F90000080 0007D00001000007E00001000003E00001000003E00001000001E00001000001E00003000000E0 000FC00000C000FFF800004000FFF800004000>47 41 126 168 46 I[<00001FF0000000F03C 000003C00700000F0003C0001E0001E0003C0000F000F80000F001F000007801E000007C03C000 007C07C000003C0F8000003E0F8000003E1F0000003E1F0000003E3F0000003F3E0000003F7E00 00003F7E0000003F7E0000003F7E0000003FFC0000007EFC0000007EFC0000007EFC0000007EFC 0000007CFC000000FCFC000000FCFC000000F8FC000001F87C000001F07C000003F07C000003E0 3E000007C03E00000F801E00000F001F00001E000F80003C00078000780003C000F00000F003C0 00007C0F0000000FF80000>40 43 122 169 47 I[<01FFFFFF0003FFFFFFE0000FC003F0000F 8000F8000F80007C000F80003E000F80003F000F80003F001F00003F001F00003F001F00003F00 1F00003F001F00003F001F00007E003E00007E003E00007C003E0000F8003E0001F0003E0007E0 003E001F80007FFFFC00007C000000007C000000007C000000007C000000007C00000000F80000 0000F800000000F800000000F800000000F800000000F800000001F000000001F000000001F000 000001F000000001F000000001F000000003F0000000FFFF800000FFFF800000>40 41 126 168 42 I[<01FFFFFC000003FFFFFF8000000FC00FE000000F8001F000000F8000F800 000F80007C00000F80007C00000F80007E00001F00007E00001F00007E00001F00007E00001F00 007E00001F00007E00001F0000FC00003E0000F800003E0001F800003E0001E000003E0003C000 003E000F0000003E007C0000007FFFE00000007C00700000007C003C0000007C001E0000007C00 0F0000007C000F000000F8000F800000F8000F800000F8000F800000F8000F800000F8000F8000 00F8000F800001F0001F800001F0001F800001F0001F800001F0001F804001F0001F804001F000 1F808003F0000FC080FFFF8007C100FFFF8003C20000000000FC00>42 42 126 168 45 82 D[<0001FC020007FF06001E038E003800DC0070007C00E0003C01E0001C03C0 001C03C0001C0380000807800008078000080780000807C0000807C0000007E0000003F0000003 FE000001FFE00001FFFE0000FFFF00003FFF80000FFFC00000FFE000000FE0000003F0000001F0 000001F0000001F0200000F0200000F0200000F0200000E0600001E0600001E0700001C0700003 C0780007807C000700E6001E00E3C07C00C1FFF000803FC000>31 43 125 169 33 I[<1FFFFFFFFE3FFFFFFFFE3F003F007E38003E001E30003E000C20003E000460003E00 0440003E000440007C000440007C000480007C000480007C000480007C000480007C00040000F8 00000000F800000000F800000000F800000000F800000000F800000001F000000001F000000001 F000000001F000000001F000000001F000000003E000000003E000000003E000000003E0000000 03E000000003E000000007C000000007C000000007C000000007C000000007C000000007C00000 000FC000000FFFFFC0001FFFFFC000>39 41 121 168 44 I[<7FFFC00FFF80FFFFC01FFF8003 F00001F80003E00000E00003E00000400003E00000400003E00000400003E00000400007C00000 800007C00000800007C00000800007C00000800007C00000800007C0000080000F80000100000F 80000100000F80000100000F80000100000F80000100000F80000100001F00000200001F000002 00001F00000200001F00000200001F00000200001F00000200003E00000400003E00000400003E 00000400003E00000400003E00000800003E00000800003E00001800001E00001000001E000020 00000F00006000000F0000C0000007800180000003C00700000001F01C000000007FF800000000 1FC0000000>41 42 120 168 46 I[42 42 120 168 46 I E /Fi 3 21 438 432 dfs[35 3 123 143 47 0 D[<007C0001FF0007FFC00FFFE01FFFF03FFFF83FFFF87FFFFC7FFFFCFFFFFE FFFFFEFFFFFEFFFFFEFFFFFEFFFFFEFFFFFE7FFFFC7FFFFC3FFFF83FFFF81FFFF00FFFE007FFC0 01FF00007C00>23 25 125 154 30 15 D[<000000006000000001E000000007E00000001F8000 00007E00000001F800000007E00000001F800000007E00000001F800000007E00000001F800000 007E00000001F800000007E00000001F800000007E00000000F800000000F8000000007E000000 001F8000000007E000000001F8000000007E000000001F8000000007E000000001F8000000007E 000000001F8000000007E000000001F8000000007E000000001F8000000007C000000001E00000 0000E0000000000000000000000000000000000000000000000000000000000000000000000000 0000000000000000007FFFFFFFC0FFFFFFFFE0FFFFFFFFE0>35 48 123 165 47 20 D E /Fj 35 123 576 432 dfs[<1C003E007F00FF80FF80FF807F003E001C00>9 9 123 136 25 46 D[<000E00001E00007E0007FE00FFFE00FFFE00F8FE0000FE0000FE0000FE 0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE 0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE 007FFFFE7FFFFE7FFFFE>23 39 123 166 45 49 D[<00FF800003FFF0000FFFFC001F03FE0038 00FF007C007F80FE003FC0FF003FC0FF003FE0FF001FE0FF001FE07E001FE03C003FE000003FE0 00003FC000003FC000007F8000007F000000FE000000FC000001F8000003F0000003E000000780 00000F0000001E0000003C00E0007000E000E000E001C001C0038001C0070001C00FFFFFC01FFF FFC03FFFFFC07FFFFFC0FFFFFF80FFFFFF80FFFFFF80>27 39 125 166 45 I[<007F800003FFF00007FFFC000F81FE001F00FF003F80FF003F807F803F807F803F807F80 1F807F800F007F800000FF000000FF000000FE000001FC000001F8000007F00000FFC00000FFF0 000001FC0000007E0000007F0000007F8000003FC000003FC000003FE000003FE03C003FE07E00 3FE0FF003FE0FF003FE0FF003FC0FF007FC07E007F807C007F003F01FE001FFFFC0007FFF00000 FF8000>27 39 125 166 45 I[<00000E0000001E0000003E0000007E000000FE000000FE0000 01FE000003FE0000077E00000E7E00000E7E00001C7E0000387E0000707E0000E07E0000E07E00 01C07E0003807E0007007E000E007E000E007E001C007E0038007E0070007E00E0007E00FFFFFF F8FFFFFFF8FFFFFFF80000FE000000FE000000FE000000FE000000FE000000FE000000FE000000 FE00007FFFF8007FFFF8007FFFF8>29 39 126 166 45 I[<0C0003000F803F000FFFFE000FFF FC000FFFF8000FFFF0000FFFE0000FFFC0000FFE00000E0000000E0000000E0000000E0000000E 0000000E0000000E7FC0000FFFF8000F80FC000E003E000C003F0000001F8000001FC000001FC0 00001FE000001FE018001FE07C001FE0FE001FE0FE001FE0FE001FE0FE001FC0FC001FC078003F 8078003F803C007F001F01FE000FFFF80003FFF00000FF8000>27 39 125 166 45 I[<0007F000003FFC0000FFFE0001FC0F0003F01F8007E03F800FC03F801FC03F801F80 3F803F801F003F8000007F0000007F0000007F000000FF000000FF0FC000FF3FF800FF707C00FF C03E00FFC03F00FF801F80FF801FC0FF001FC0FF001FE0FF001FE0FF001FE07F001FE07F001FE0 7F001FE07F001FE03F001FE03F001FC01F801FC01F803F800FC03F0007E07E0003FFFC0000FFF0 00003FC000>27 39 125 166 45 I[<380000003E0000003FFFFFF03FFFFFF03FFFFFF07FFFFF E07FFFFFC07FFFFF807FFFFF0070000E0070000E0070001C00E0003800E0007000E000E0000000 E0000001C000000380000007800000078000000F0000000F0000001F0000001F0000003F000000 3E0000003E0000007E0000007E0000007E0000007E000000FE000000FE000000FE000000FE0000 00FE000000FE000000FE000000FE0000007C000000380000>28 41 124 168 45 I[<00003FF001800003FFFE0380000FFFFF8780003FF007DF8000FF8001FF8001FE0000 7F8003FC00003F8007F000001F800FF000000F801FE0000007801FE0000007803FC0000007803F C0000003807FC0000003807F80000003807F8000000000FF8000000000FF8000000000FF800000 0000FF8000000000FF8000000000FF8000000000FF8000000000FF8000000000FF80000000007F 80000000007F80000000007FC0000003803FC0000003803FC0000003801FE0000003801FE00000 07000FF00000070007F000000E0003FC00001E0001FE00003C0000FF8000F800003FF007E00000 0FFFFFC0000003FFFF000000003FF80000>41 41 124 168 67 67 D[45 41 125 168 71 I[36 41 125 168 57 70 D[<00007FE003000003FFFC0700001FFFFF0F00003FF00FFF0000FF8001FF0001FE0000 FF0003F800003F0007F000003F000FF000001F001FE000000F001FE000000F003FC000000F003F C0000007007FC0000007007F80000007007F8000000000FF8000000000FF8000000000FF800000 0000FF8000000000FF8000000000FF8000000000FF8000000000FF8000000000FF8001FFFFF87F 8001FFFFF87F8001FFFFF87FC00000FF003FC00000FF003FC00000FF001FE00000FF001FE00000 FF000FF00000FF0007F00000FF0003F80000FF0001FE0000FF0000FF8001FF00003FF007BF0000 1FFFFF1F000003FFFE0F0000007FF00300>45 41 124 168 72 I[22 41 126 168 35 73 D[<0000FFE000000007FFFC0000003FC07F8000007F001FC00001FC0007F0 0003F80003F80007F00001FC000FF00001FE001FE00000FF001FE00000FF003FC000007F803FC0 00007F807FC000007FC07F8000003FC07F8000003FC07F8000003FC0FF8000003FE0FF8000003F E0FF8000003FE0FF8000003FE0FF8000003FE0FF8000003FE0FF8000003FE0FF8000003FE0FF80 00003FE0FF8000003FE07F8000003FC07FC000007FC07FC000007FC03FC000007F803FC000007F 801FE00000FF001FE00000FF000FF00001FE0007F00001FC0003F80003F80001FC0007F00000FF 001FE000003FC07F8000000FFFFE00000000FFE00000>43 41 124 168 69 79 D[<007F806003FFF0E007FFF9E00F807FE01F001FE03E0007E07C0003E07C0001E0FC00 01E0FC0001E0FC0000E0FE0000E0FE0000E0FF000000FFC000007FFE00007FFFE0003FFFFC001F FFFE000FFFFF8007FFFFC003FFFFE000FFFFE00007FFF000007FF000000FF8000007F8000003F8 600001F8E00001F8E00001F8E00001F8F00001F0F00001F0F80003F0FC0003E0FF0007C0FFE01F 80F3FFFF00E0FFFE00C01FF000>29 41 124 168 51 83 D[<01FF800007FFF0000F81F8001FC0 7E001FC07E001FC03F000F803F8007003F8000003F8000003F8000003F80000FFF8000FFFF8007 FC3F800FE03F803F803F803F003F807F003F80FE003F80FE003F80FE003F80FE003F807E007F80 7F00DF803F839FFC0FFF0FFC01FC03FC>30 27 126 154 44 97 D[33 42 126 169 51 I[<001FF80000FFFE0003F01F0007E03F800FC03F801F803F80 3F801F007F800E007F0000007F000000FF000000FF000000FF000000FF000000FF000000FF0000 00FF0000007F0000007F0000007F8000003F8001C01F8001C00FC0038007E0070003F01E0000FF FC00001FE000>26 27 126 154 41 I[<00003FF80000003FF80000003FF800000003F8000000 03F800000003F800000003F800000003F800000003F800000003F800000003F800000003F80000 0003F800000003F800000003F800001FE3F80000FFFBF80003F03FF80007E00FF8000FC007F800 1F8003F8003F8003F8007F0003F8007F0003F8007F0003F800FF0003F800FF0003F800FF0003F8 00FF0003F800FF0003F800FF0003F800FF0003F8007F0003F8007F0003F8007F0003F8003F8003 F8001F8003F8000F8007F80007C00FF80003F03BFF8000FFF3FF80003FC3FF80>33 42 126 169 51 I[<003FE00001FFF80003F07E0007C01F000F801F801F800F803F800FC07F00 0FC07F0007C07F0007E0FF0007E0FF0007E0FFFFFFE0FFFFFFE0FF000000FF000000FF0000007F 0000007F0000007F0000003F8000E01F8000E00FC001C007E0038003F81F0000FFFE00001FF000 >27 27 126 154 43 I[<0007F0003FFC00FE3E01F87F03F87F03F07F07F07F07F03E07F00007 F00007F00007F00007F00007F00007F000FFFFC0FFFFC0FFFFC007F00007F00007F00007F00007 F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007 F00007F00007F00007F0007FFF807FFF807FFF80>24 42 126 169 28 I[33 42 125 169 51 104 D[<07000FC01FE03FE03FE03FE01FE00FC0070000 00000000000000000000000000FFE0FFE0FFE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE0 0FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE0FFFEFFFEFFFE>15 43 125 170 27 I[15 42 125 169 27 108 D[53 27 125 154 80 I[33 27 125 154 51 I[<003FE00001FFFC0003F07E000FC01F801F800FC03F800FE03F0007E07F0007F07F0007F07F 0007F0FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F87F0007F0 7F0007F03F800FE03F800FE01F800FC00FC01F8007F07F0001FFFC00003FE000>29 27 126 154 45 I[33 39 126 154 51 I[24 27 126 154 37 114 D[<03FE300FFFF01E03F03800F0700070F00070F00070F80070FC0000FFE0007FFE007F FF803FFFE01FFFF007FFF800FFF80003FC0000FC60007CE0003CF0003CF00038F80038FC0070FF 01E0F7FFC0C1FF00>22 27 126 154 36 I[<00700000700000700000700000F00000F00000F0 0001F00003F00003F00007F0001FFFF0FFFFF0FFFFF007F00007F00007F00007F00007F00007F0 0007F00007F00007F00007F00007F00007F00007F00007F03807F03807F03807F03807F03807F0 3803F03803F87001F86000FFC0001F80>21 38 127 165 36 I[33 27 125 154 51 I[33 27 127 154 48 I[33 39 127 154 48 121 D[<3FFFFF803FFFFF803F007F003C00FE0038 01FE007803FC007803F8007007F800700FF000700FE000001FC000003FC000007F8000007F0000 00FF000001FE038001FC038003F8038007F803800FF007800FE007801FE007003FC00F003F801F 007F007F00FFFFFF00FFFFFF00>25 27 126 154 41 I E /Fk 70 124 438 432 dfs[<0007F81F80003C067060007003E0F000E007C1F001C00FC1F003C00F80E00780 078040078007800007800780000780078000078007800007800780000780078000078007800007 800780000780078000FFFFFFFF00FFFFFFFF000780078000078007800007800780000780078000 078007800007800780000780078000078007800007800780000780078000078007800007800780 000780078000078007800007800780000780078000078007800007800780000780078000078007 80000780078000078007C000FFF87FFE00FFF87FFE00>36 42 127 169 35 11 D[<0007F800003C06000070010000E0070001C00F8003C00F8007800F80078007000780 000007800000078000000780000007800000078000000780000007800000FFFFFF80FFFFFF8007 800F80078007800780078007800780078007800780078007800780078007800780078007800780 078007800780078007800780078007800780078007800780078007800780078007800780078007 800780078007800780FFF87FFCFFF87FFC>30 42 127 169 33 I[<0007F980003C0780007007 8000E00F8001C00F8003C00F800780078007800780078007800780078007800780078007800780 0780078007800780078007800780FFFFFF80FFFFFF800780078007800780078007800780078007 800780078007800780078007800780078007800780078007800780078007800780078007800780 0780078007800780078007800780078007800780078007800780078007800780FFFCFFFCFFFCFF FC>30 42 127 169 33 I[<0003F803FC00001E061E030000700138008000E003F0038001C007 E007C003C007E007C0078007C007C0078003C00380078003C00000078003C00000078003C00000 078003C00000078003C00000078003C00000078003C00000078003C00000FFFFFFFFFFC0FFFFFF FFFFC0078003C007C0078003C003C0078003C003C0078003C003C0078003C003C0078003C003C0 078003C003C0078003C003C0078003C003C0078003C003C0078003C003C0078003C003C0078003 C003C0078003C003C0078003C003C0078003C003C0078003C003C0078003C003C0078003C003C0 078003C003C0078003C003C0078003C003C0FFFC7FFE7FFEFFFC7FFE7FFE>47 42 127 169 51 I[<78FCFCFEFE7A020202020404040810102040>7 18 123 169 17 39 D[<0004000800100020004000C0018003000300060006000E000C001C001800 38003800380030007000700070007000F000F000E000E000E000E000E000E000E000E000E000E0 00E000F000F0007000700070007000300038003800380018001C000C000E000600060003000300 018000C000400020001000080004>14 61 123 172 23 I[<800040002000100008000C000600 030003000180018001C000C000E0006000700070007000300038003800380038003C003C001C00 1C001C001C001C001C001C001C001C001C001C003C003C00380038003800380030007000700070 006000E000C001C0018001800300030006000C0008001000200040008000>14 61 125 172 23 I[<000038000000003800000000380000000038000000003800000000380000 000038000000003800000000380000000038000000003800000000380000000038000000003800 0000003800000000380000000038000000003800000000380000FFFFFFFFFEFFFFFFFFFEFFFFFF FFFE00003800000000380000000038000000003800000000380000000038000000003800000000 380000000038000000003800000000380000000038000000003800000000380000000038000000 00380000000038000000003800000000380000>39 41 125 162 47 43 D[<78FCFCFEFE7A020202020404040810102040>7 18 123 133 17 I[15 3 127 142 20 I[<78FCFCFCFC78>6 6 123 133 17 I[<00000600000E00000E00001C00001C 00001C0000380000380000380000700000700000E00000E00000E00001C00001C00001C0000380 000380000380000700000700000700000E00000E00000E00001C00001C00001C00003800003800 00700000700000700000E00000E00000E00001C00001C00001C000038000038000038000070000 0700000700000E00000E00000E00001C00001C0000380000380000380000700000700000700000 E00000E00000C00000>23 60 125 172 30 I[<007F000001C1C0000780F0000F0078000E0038 001C001C003C001E003C001E003C001E0078000F0078000F0078000F0078000F00F8000F80F800 0F80F8000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8 000F80F8000F80F8000F80F8000F8078000F0078000F0078000F0078000F003C001E003C001E00 3C001E001C001C000E0038000F0078000780F00001C1C000007F0000>25 41 126 167 30 I[<00100000700001F0000FF000FEF000F0F00000F00000F00000F00000F000 00F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F000 00F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F000 00F00001F8007FFFE07FFFE0>19 40 123 167 30 I[<00FE0007FF800E07E01803F02001F820 00F840007C40007CF8007EFC007EFC003EFC003EFC003E78007E00007E00007C00007C0000F800 00F80001F00001E00003C0000780000700000E00001C0000380000700000600000C00001800203 00020600040C000418000410000C3FFFFC7FFFF8FFFFF8FFFFF8>23 40 125 167 30 I[<007F000003FFC0000701F0000C00F80010007C001C007C003E007E003E003E00 3E003E001E003E000C007E0000007C0000007C00000078000000F0000000E0000001C000000700 0000FF00000001E0000000F0000000780000003C0000003E0000001F0000001F0000001F800000 1F8030001F8078001F80FC001F80FC001F80FC001F00F8001F0040003F0040003E0030007C0018 00F8000F01F00003FFC000007F0000>25 41 126 167 30 I[<00006000000060000000E00000 01E0000001E0000003E0000003E0000005E0000009E0000009E0000011E0000021E0000021E000 0041E0000081E0000081E0000101E0000201E0000201E0000401E0000801E0000801E0001001E0 003001E0002001E0004001E000C001E000FFFFFF80FFFFFF800001E0000001E0000001E0000001 E0000001E0000001E0000001E0000001E0000003F000007FFF80007FFF80>25 40 126 167 30 I[<1800181F00F01FFFE01FFFC01FFF801FFF0011F800100000100000100000 100000100000100000100000100000107E001183801600C01800E010007000007800003C00003C 00003C00003E00003E00003E70003EF8003EF8003EF8003EF8003C80003C40007C400078200078 3000F01801E00E07C007FF0001FC00>23 41 125 167 30 I[<000FE000003FF80000F81C0001 E00C0003801E0007803E000F003E000E001C001E0000001C0000003C0000003C0000007C000000 7800000078000000F83F0000F840E000F9807000F9003800FA001C00FC001E00FC001E00FC000F 00F8000F00F8000F80F8000F80F8000F80F8000F8078000F8078000F8078000F807C000F803C00 0F003C000F001C001E001E001E000E003C000700780003C0F00001FFC000007F0000>25 41 126 167 30 I[<20000000380000003FFFFF803FFFFF803FFFFF007FFFFF00600002004000 040040000400400008008000100080002000000020000000400000008000000080000001000000 030000000200000006000000060000000C0000000C0000001C0000001C0000001C000000380000 00380000003800000078000000780000007800000078000000F8000000F8000000F8000000F800 0000F8000000F8000000F8000000F8000000700000>25 42 125 168 30 I[<007F000001FFC0000381F000060078000C003C001C001C0018000E0038000E0038000E0038 000E003C000E003C000E003E001C001F8018001FC038000FF0600007F8C00003FF800001FF0000 007FC00000FFE000030FF8000603FC001C01FE0038007E0030003F0070000F0070000780E00007 80E0000380E0000380E0000380E0000380F0000300700007007800060038000C001E0038000F80 F00003FFE000007F0000>25 41 126 167 30 I[<007F000001FFC00007C1E0000F0070001E00 38001C003C003C001C0078001E0078001E00F8000F00F8000F00F8000F00F8000F00F8000F80F8 000F80F8000F80F8000F8078000F8078001F803C001F803C001F801C002F800E004F800700CF80 03810F80007E0F8000000F0000000F0000000F0000001E0000001E0000001E0000003C001C003C 003E0078003E0070003C00E0001801C0001C0780000FFE000003F80000>25 41 126 167 30 I[<78FCFCFCFC78000000000000000000000000000078FCFCFCFC78>6 26 123 153 17 I[<78FCFCFCFC78000000000000000000000000000070F8FCFCFC7C04040404 0808081010202040>6 38 123 153 17 I[39 15 125 149 47 61 D[<000018000000001800000000 18000000003C000000003C000000003C000000007E000000007E00000000FF000000009F000000 009F000000011F800000010F800000010F8000000207C000000207C000000207C000000403E000 000403E000000403E000000801F000000801F000001801F800001000F800001000F800002000FC 000020007C00003FFFFC00007FFFFE000040003E000040003E000080001F000080001F00008000 1F000100000F800100000F800100000F8002000007C007000007C01F80000FE0FFF000FFFFFFF0 00FFFF>40 42 126 169 46 65 D[36 41 126 168 43 I[<0000FF00100007FFE030001FC07830003E000C7000F80006F001F00003F003E00001 F007C00000F00F800000700F800000701F000000303F000000303E000000303E000000107E0000 00107E000000107C00000000FC00000000FC00000000FC00000000FC00000000FC00000000FC00 000000FC00000000FC00000000FC000000007C000000007E000000007E000000103E000000103E 000000103F000000101F000000200F800000200F8000006007C000004003E000008001F0000180 00F8000300003E000E00001FC038000007FFE0000000FF8000>36 43 125 169 44 I[36 41 126 168 42 69 D[34 41 126 168 40 I[<0000FF00100007FFE030001FC07830003E000C7000F80006F001F00003F0 03E00001F007C00000F00F800000700F800000701F000000303F000000303E000000303E000000 107E000000107E000000107C00000000FC00000000FC00000000FC00000000FC00000000FC0000 0000FC00000000FC00000000FC00000000FC0000FFFF7C0000FFFF7E000003F07E000001F03E00 0001F03E000001F03F000001F01F000001F00F800001F00F800001F007C00001F003E00001F001 F00002F000F80002F0003E000C70001FC038300007FFE0100000FF8000>40 43 125 169 48 I[40 41 126 168 46 I[17 41 126 168 22 I[50 41 126 168 56 77 D[40 41 126 168 46 I[<0001FF0000000F01E000003C00 78000078003C0000E0000E0001E0000F0003C000078007800003C00F800003E01F000001F01F00 0001F03E000000F83E000000F87E000000FC7E000000FC7C0000007C7C0000007CFC0000007EFC 0000007EFC0000007EFC0000007EFC0000007EFC0000007EFC0000007EFC0000007EFC0000007E 7C0000007C7E000000FC7E000000FC7E000000FC3E000000F83F000001F81F000001F01F000001 F00F800003E007800003C007C00007C003E0000F8000F0001E000078003C00003C007800000F01 E0000001FF0000>39 43 125 169 47 I[35 41 126 168 42 I[41 42 126 168 45 82 D[<00FE010003FF83000F81E3001E0037003C001F0038000F00780007007000 0700F0000300F0000300F0000300F0000100F8000100F8000100FC0000007C0000007F0000003F E000001FFF00000FFFE00007FFF80003FFFC00007FFE000007FF0000007F0000001F8000000F80 000007C0000007C0800003C0800003C0800003C0800003C0C00003C0C0000380C0000380E00007 80F0000700F8000E00EE001C00C3C07800C1FFF000803FC000>26 43 125 169 33 I[<7FFFFFFFF87FFFFFFFF87C007C00F870007C003860007C001840007C000840007C00 08C0007C000CC0007C000C80007C000480007C000480007C000480007C000480007C000400007C 000000007C000000007C000000007C000000007C000000007C000000007C000000007C00000000 7C000000007C000000007C000000007C000000007C000000007C000000007C000000007C000000 007C000000007C000000007C000000007C000000007C000000007C000000007C000000007C0000 0000FE000000FFFFFE0000FFFFFE00>38 41 126 168 44 I[58 42 127 168 62 87 D[9 60 122 172 17 91 D[9 60 127 172 17 93 D[<01FC00000E0780001001C0003C00E0003E 00F0003E0078001C00780008007800000078000000780000007800007FF80003E078000F807800 1F0078003E0078007C00780078007820F8007820F8007820F8007820F800F8207C00F8203C013C 401F063FC007F80F00>27 26 126 153 30 97 D[<07800000FF800000FF8000000F8000000780 000007800000078000000780000007800000078000000780000007800000078000000780000007 800000078000000783F000078C1C0007B0070007A0038007C003C0078001E0078001E0078000F0 078000F0078000F8078000F8078000F8078000F8078000F8078000F8078000F8078000F0078000 F0078001F0078001E0078001C007C003C00740078007200E0006181C000407E000>29 42 127 169 33 I[<007F8001C0700780080F003C1E007C3C007C3C00387C0010780000F80000 F80000F80000F80000F80000F80000F80000F800007800007C00003C00043C00041E00080F0010 07802001C0C0007F00>22 26 126 153 27 I[<00000F000001FF000001FF0000001F0000000F 0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F000000 0F0000000F00003F0F0001C0CF0003802F000F001F001E001F001C000F003C000F007C000F0078 000F0078000F00F8000F00F8000F00F8000F00F8000F00F8000F00F8000F00F8000F0078000F00 78000F003C000F003C000F001E001F000E002F0007004F8001C18FF8007E0FF8>29 42 126 169 33 I[<007E0003C3800700E00E00F01C00703C00783C003878003C78003CF8003C F8003CFFFFFCF80000F80000F80000F80000F800007800007C00003C00043C00041E00080E0010 07002001C0C0007F00>22 26 126 153 27 I[<001F000070C000E1E001C3E003C3E00381C007 8080078000078000078000078000078000078000078000078000078000FFFE00FFFE0007800007 800007800007800007800007800007800007800007800007800007800007800007800007800007 800007800007800007800007800007800007800007C000FFFE00FFFE00>19 42 127 169 18 I[<0000078001FC1840070721C00E03C1C01E03C0803C01E0003C01E0007C01 F0007C01F0007C01F0007C01F0007C01F0003C01E0003C01E0001E03C0000E0380001707000011 FC0000300000003000000030000000380000001C0000001FFFC0000FFFF80007FFFC001C003E00 30000F007000070060000380E0000380E0000380E0000380E0000380700007007000070038000E 000C0018000780F00000FF8000>26 40 126 154 30 I[<07800000FF800000FF8000000F8000 000780000007800000078000000780000007800000078000000780000007800000078000000780 000007800000078000000783F800078C1C0007900E0007A0070007A0078007C0078007C0078007 800780078007800780078007800780078007800780078007800780078007800780078007800780 07800780078007800780078007800780078007800780078007800780FFFCFFFCFFFCFFFC>30 42 127 169 33 I[<07000F801F801F800F800700000000000000000000000000000000000000 07807F807F800F8007800780078007800780078007800780078007800780078007800780078007 800780078007800780FFF8FFF8>13 41 127 168 17 I[<007800FC00FC00FC00FC0078000000 000000000000000000000000000000007C07FC07FC007C003C003C003C003C003C003C003C003C 003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C00 3C7038F838F870F07060C01F80>14 53 130 168 18 I[<07800000FF800000FF8000000F8000 000780000007800000078000000780000007800000078000000780000007800000078000000780 0000078000000780000007807FF007807FF007801F8007801C0007801800078020000780400007 808000078100000782000007870000079F800007A7800007C7C0000783E0000781E0000781F000 0780F8000780780007807C0007803E0007801E0007801F0007801F80FFFC7FF8FFFC7FF8>29 42 127 169 32 I[<0780FF80FF800F8007800780078007800780078007800780078007800780 078007800780078007800780078007800780078007800780078007800780078007800780078007 8007800780078007800780FFFCFFFC>14 42 127 169 17 I[<0781F800FC00FF860E030700FF 98070C03800FA0079003C007A003D001E007C003E001E007C003E001E0078003C001E0078003C0 01E0078003C001E0078003C001E0078003C001E0078003C001E0078003C001E0078003C001E007 8003C001E0078003C001E0078003C001E0078003C001E0078003C001E0078003C001E0078003C0 01E0078003C001E0078003C001E0FFFC7FFE3FFFFFFC7FFE3FFF>48 26 127 153 52 I[<0783F800FF8C1C00FF900E000FA0070007A0078007C0078007C0078007800780 078007800780078007800780078007800780078007800780078007800780078007800780078007 80078007800780078007800780078007800780078007800780FFFCFFFCFFFCFFFC>30 26 127 153 33 I[<007F000001C1C000070070000E0038001C001C003C001E003C001E007800 0F0078000F00F8000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8000F8078 000F0078000F003C001E003C001E001E003C000E0038000700700001C1C000007F0000>25 26 126 153 30 I[<0783F000FF8C1C00FFB00F0007A0078007C003C0078003E0078001E00780 01F0078001F0078000F8078000F8078000F8078000F8078000F8078000F8078000F8078000F007 8001F0078001F0078001E0078003C007C003C007C0078007A00E0007983C000787E00007800000 078000000780000007800000078000000780000007800000078000000780000007800000FFFC00 00FFFC0000>29 38 127 153 33 I[<003F008001E0C180038021800F0013801E000B801E000F 803C0007807C0007807C00078078000780F8000780F8000780F8000780F8000780F8000780F800 0780F80007807C0007807C0007803C0007803E000F801E000F800F0017800780278001E0C78000 7F0780000007800000078000000780000007800000078000000780000007800000078000000780 000007800000FFFC0000FFFC>30 38 126 153 32 I[<0787C0FF98E0FF91F00FA1F007C1F007 C0E007C00007800007800007800007800007800007800007800007800007800007800007800007 800007800007800007800007800007C000FFFE00FFFE00>20 26 127 153 23 I[<07F8401C06C03001C06000C06000C0E00040E00040F00040F800007E00007FF0003FFE00 0FFF0003FF80003FC00007C08001E08001E0C000E0C000E0C000E0E000C0F001C0F80180C40700 83F800>19 26 126 153 24 I[<00800000800000800000800001800001800001800003800003 80000780000F80001FFF80FFFF8007800007800007800007800007800007800007800007800007 800007800007800007800007800007804007804007804007804007804007804007804003C08001 C08000E100003E00>18 37 127 164 23 I[<07800780FF80FF80FF80FF800F800F8007800780 078007800780078007800780078007800780078007800780078007800780078007800780078007 80078007800780078007800780078007800780078007800F8007800F800380178001C027C000E0 47FC003F87FC>30 26 127 153 33 I[29 26 127 153 32 I[40 26 127 153 44 I[29 26 127 153 32 I[29 38 127 153 32 I[<7FFFF87800F06001F04001E04003C0C007C0800780800F 00801F00001E00003C00007C0000780000F00001F00001E00403C00407C0040780040F000C1F00 081E00083C00187C00387800F8FFFFF8>22 26 126 153 27 I[29 1 128 144 30 I E /Fl 12 119 995 432 dfs[<00003FF001800003FFFE0380000FFFFF8780 003FF007DF8000FF8001FF8001FE00007F8003FC00003F8007F000001F800FF000000F801FE000 0007801FE0000007803FC0000007803FC0000003807FC0000003807F80000003807F8000000000 FF8000000000FF8000000000FF8000000000FF8000000000FF8000000000FF8000000000FF8000 000000FF8000000000FF80000000007F80000000007F80000000007FC0000003803FC000000380 3FC0000003801FE0000003801FE0000007000FF00000070007F000000E0003FC00001E0001FE00 003C0000FF8000F800003FF007E000000FFFFFC0000003FFFF000000003FF80000>41 41 124 168 115 67 D[<01FF800007FFF0000F81F8001FC07E001FC07E001FC03F000F803F80 07003F8000003F8000003F8000003F80000FFF8000FFFF8007FC3F800FE03F803F803F803F003F 807F003F80FE003F80FE003F80FE003F80FE003F807E007F807F00DF803F839FFC0FFF0FFC01FC 03FC>30 27 126 154 76 97 D[<001FF80000FFFE0003F01F0007E03F800FC03F801F803F803F 801F007F800E007F0000007F000000FF000000FF000000FF000000FF000000FF000000FF000000 FF0000007F0000007F0000007F8000003F8001C01F8001C00FC0038007E0070003F01E0000FFFC 00001FE000>26 27 126 154 71 99 D[<003FE00001FFF80003F07E0007C01F000F801F801F80 0F803F800FC07F000FC07F0007C07F0007E0FF0007E0FF0007E0FFFFFFE0FFFFFFE0FF000000FF 000000FF0000007F0000007F0000007F0000003F8000E01F8000E00FC001C007E0038003F81F00 00FFFE00001FF000>27 27 126 154 74 101 D[<07000FC01FE03FE03FE03FE01FE00FC00700 0000000000000000000000000000FFE0FFE0FFE00FE00FE00FE00FE00FE00FE00FE00FE00FE00F E00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE0FFFEFFFEFFFE>15 43 125 170 46 105 D[15 42 125 169 46 108 D[53 27 125 154 138 I[33 27 125 154 88 I[<003FE00001FFFC0003F07E000FC01F801F800FC03F800FE03F0007E07F0007F07F00 07F07F0007F0FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F87F 0007F07F0007F03F800FE03F800FE01F800FC00FC01F8007F07F0001FFFC00003FE000>29 27 126 154 78 I[<00700000700000700000700000F00000F00000F00001F00003F00003F000 07F0001FFFF0FFFFF0FFFFF007F00007F00007F00007F00007F00007F00007F00007F00007F000 07F00007F00007F00007F00007F03807F03807F03807F03807F03807F03803F03803F87001F860 00FFC0001F80>21 38 127 165 62 116 D[33 27 125 154 88 I[33 27 127 154 83 I E /Fm 8 117 829 432 dfs[<000E00001E00007E0007FE00FFFE00FFFE00F8FE 0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE 0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE 0000FE0000FE0000FE007FFFFE7FFFFE7FFFFE>23 39 123 166 65 49 D[<00003FF001800003FFFE0380000FFFFF8780003FF007DF8000FF8001FF8001FE00007F8003 FC00003F8007F000001F800FF000000F801FE0000007801FE0000007803FC0000007803FC00000 03807FC0000003807F80000003807F8000000000FF8000000000FF8000000000FF8000000000FF 8000000000FF8000000000FF8000000000FF8000000000FF8000000000FF80000000007F800000 00007F80000000007FC0000003803FC0000003803FC0000003801FE0000003801FE0000007000F F00000070007F000000E0003FC00001E0001FE00003C0000FF8000F800003FF007E000000FFFFF C0000003FFFF000000003FF80000>41 41 124 168 96 67 D[<01FF800007FFF0000F81F8001F C07E001FC07E001FC03F000F803F8007003F8000003F8000003F8000003F80000FFF8000FFFF80 07FC3F800FE03F803F803F803F003F807F003F80FE003F80FE003F80FE003F80FE003F807E007F 807F00DF803F839FFC0FFF0FFC01FC03FC>30 27 126 154 63 97 D[<003FE00001FFF80003F0 7E0007C01F000F801F801F800F803F800FC07F000FC07F0007C07F0007E0FF0007E0FF0007E0FF FFFFE0FFFFFFE0FF000000FF000000FF0000007F0000007F0000007F0000003F8000E01F8000E0 0FC001C007E0038003F81F0000FFFE00001FF000>27 27 126 154 61 101 D[33 42 125 169 73 104 D[33 39 126 154 73 112 D[24 27 126 154 54 114 D[<007000007000007000007000 00F00000F00000F00001F00003F00003F00007F0001FFFF0FFFFF0FFFFF007F00007F00007F000 07F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F03807F03807F038 07F03807F03807F03803F03803F87001F86000FFC0001F80>21 38 127 165 52 116 D E /Fn 36 119 400 300 dfs[<0007800000001C600380003010078000601007 8000E010038000C010010001C07001000380F002000380F0020003806004000700000800070000 10000700006000073C00A00007E20310000781041000070108080007011008000F022008001F84 2008001C782C08003C003E080038003E080078001C0800780000080078000008007000001000F0 00001000700000200070000020007800004000380000800038000100001C000200000E000C0000 0380700000007F800000>33 37 123 163 49 38 D[<70F8F8F0E0>5 5 122 132 20 46 D[<0000030000000300000007000000070000000F0000000F0000001F000000 2F0000002F0000004F0000004F8000008780000087800001078000020780000207800004078000 0407800008078000080780001007800030078000200780007FFF80004007C0008007C0008003C0 010003C0030003C0020003C0040003C0040003C00C0003C03C0007C0FF003FFC>30 35 125 162 48 65 D[<00FFFFE0000F0038000F001C000F001E001E000E001E000F001E000F00 1E000F003C000E003C001E003C001E003C003C00780078007800F0007801E00078078000FFFF80 00F001E000F000F000F0007801E0007801E0003801E0003C01E0003C03C0007803C0007803C000 7803C000F0078000F0078001E0078003C0078007000F801E00FFFFF000>32 34 125 161 45 I[<00FFFFF000000F003C00000F000E00000F000700001E000380001E000380 001E0001C0001E0001C0003C0001C0003C0001E0003C0001E0003C0001E000780001E000780001 E000780001E000780001E000F00003C000F00003C000F00003C000F00003C001E000078001E000 078001E000070001E0000F0003C0000E0003C0001C0003C0003C0003C000380007800070000780 00E000078001C00007800700000F801C0000FFFFF00000>35 34 125 161 49 68 D[<00007F00800003808100000E00630000380027000070001F0000E0000E0001C0000E 000380000E000700000E000F000004000E000004001E000004003C000004003C00000800780000 000078000000007800000000F000000000F000000000F000000000F000000000F0003FFC00E000 01E000E00001E000E00001E000E00003C000E00003C000F00003C000700003C000700007800038 0007800018000F80001C0013800006002300000381C1000000FE000000>33 36 121 162 51 71 D[<00FFF8000F00000F00000F00001E00001E00001E00001E00003C00003C 00003C00003C0000780000780000780000780000F00000F00000F00000F00001E00001E00001E0 0001E00003C00003C00003C00003C0000780000780000780000780000F8000FFF800>21 34 125 161 25 73 D[<0007FFC000003C0000003C0000003C0000007800000078000000780000 0078000000F0000000F0000000F0000000F0000001E0000001E0000001E0000001E0000003C000 0003C0000003C0000003C00000078000000780000007800000078000000F0000000F0000380F00 00780F0000F81E0000F81E0000F03C0000403800004070000021E000001F800000>26 35 124 161 35 I[<00FFF807FC000F0001E0000F0001C0000F000100001E000200001E000400 001E001000001E002000003C004000003C008000003C010000003C040000007808000000781800 00007838000000787C000000F0BC000000F23C000000F41E000000F81E000001F01F000001E00F 000001E00F000001E00F800003C007800003C007800003C003C00003C003C000078003C0000780 01E000078001E000078001E0000F8001F000FFF80FFE00>38 34 125 161 49 I[<00FFFC00000F8000000F0000000F0000001E0000001E0000001E0000001E0000003C0000 003C0000003C0000003C00000078000000780000007800000078000000F0000000F0000000F000 0000F0000001E0000001E0000001E0002001E0002003C0004003C0004003C0008003C000800780 0180078001000780030007800F000F803E00FFFFFE00>27 34 125 161 41 I[<00FF800007FC000F80000F80000F80001780000F80001780001780002F000013C0002F00 0013C0004F000013C0008F000023C0009E000023C0011E000023C0011E000023C0021E000043C0 043C000043C0043C000043C0083C000041E0083C000081E01078000081E02078000081E0207800 0081E04078000101E040F0000101E080F0000101E100F0000101E100F0000200F201E0000200F2 01E0000200F401E0000200F801E0000400F803C0000400F003C0000400F003C0000C00E003C000 1E00C007C000FFC0C07FFC00>46 34 125 161 59 I[<00FF000FFC000F8001E0000F80018000 0FC000800013C001000013C001000011E001000011E001000021E002000020F002000020F00200 0020F0020000407804000040780400004078040000403C040000803C080000803E080000801E08 0000801E080001001F100001000F100001000F10000100079000020007A000020007A000020003 E000020003E000040003C000040001C000040001C0000C0001C0001E00008000FFC0008000>38 34 125 161 48 I[<0000FE0000078380000C00E0003800700070003800E0003801C0001C0380 001C0700001C0F00001E1E00001E1C00001E3C00001E3C00001E7800001E7800001E7800001EF0 00003CF000003CF000003CF0000078F0000078E0000078E00000F0E00000F0E00001E0E00001C0 F00003C0F00007807000070078000E0038001C001C0038000E00E0000703800001FC0000>31 36 121 162 49 I[<00FFFFC0000F0070000F003C000F001C001E000E001E000E001E000F001E 000F003C001E003C001E003C001E003C003C0078003800780070007801E00078078000FFFC0000 F00E0000F0070000F0038001E003C001E003C001E003C001E003C003C0078003C0078003C00780 03C0078007800F0007800F0107800F01078007020F800702FFF8038C000000F0>32 35 125 161 48 82 D[<0001F020000E0C40001802C0003001C0006001C000E0018000C0018001 C0018001C0018003C0010003C0010003C0000003C0000003E0000001F8000001FF000000FFE000 007FF000001FF8000003FC0000007C0000003C0000001E0000001E0000001E0020001C0020001C 0020001C00200018006000380060003000700060007000C000C8018000C607000081FC0000>27 36 125 162 36 I[<1FFFFFF81E03C0381803C0183003C0182007801820078018400780104007 8010400F0010800F0010800F0010000F0000001E0000001E0000001E0000001E0000003C000000 3C0000003C0000003C00000078000000780000007800000078000000F0000000F0000000F00000 00F0000001E0000001E0000001E0000001E0000003E00000FFFF0000>29 34 119 161 47 I[<3FFE03FF03C0007803C0006003C000200780004007800040078000400780 00400F0000800F0000800F0000800F0000801E0001001E0001001E0001001E0001003C0002003C 0002003C0002003C0002007800040078000400780004007800040070000800F0000800F0001000 7000100070002000700040003000400038018000180200000E0C000003F00000>32 35 119 161 48 I[45 35 118 161 65 87 D[32 34 118 161 48 89 D[<00F8C00185C00705C00E03800E03801C03803C0380380700780700780700 780700F00E00F00E00F00E00F00E10F01C20701C20703C20305C40308C400F0780>20 21 123 148 33 97 D[<007E0001C1000301800703800E07801C07803C00003800007800007800 00780000F00000F00000F00000F00000F00100700100700200300C001830000FC000>17 21 123 148 29 99 D[<00003C0003F80000380000380000380000700000700000700000700000 E00000E00000E00000E00001C000F9C00185C00705C00E03800E03801C03803C03803807007807 00780700780700F00E00F00E00F00E00F00E10F01C20701C20703C20305C40308C400F0780>22 35 123 162 33 I[<00F803840E021C023C0238027804F018FFE0F000F000E000E000E000E000 E002E0026004701830600F80>15 21 122 148 29 I[<00003E0000470000CF00018F00018600 0380000380000380000700000700000700000700000700000E0000FFF0000E00000E00000E0000 1C00001C00001C00001C00001C0000380000380000380000380000380000700000700000700000 700000700000E00000E00000E00000E00000C00001C00001C000718000F18000F300006200003C 0000>24 45 130 162 20 I[<001F180030B800E0B801C07001C0700380700780700700E00F00 E00F00E00F00E01E01C01E01C01E01C01E01C01E03800E03800E0780060B8006170001E7000007 00000700000E00000E00000E00701C00F01800F0300060E0003F8000>21 31 126 148 29 I[<00C001E001C001C0000000000000000000000000000000001C0023004300 43008700870087000E000E001C001C001C00380038003840708070807080710032001C00>11 33 123 160 20 105 D[<00F0000FE00000E00000E00000E00001C00001C00001C00001C00003 80000380000380000380000700000701E0070210070C700E10F00E10F00E20600E40001D80001E 00001FC0001C7000383800383800381C00381C20703840703840703840701880E01880600F00> 20 35 125 162 29 107 D[<01E01FC001C001C001C0038003800380038007000700070007000E 000E000E000E001C001C001C001C0038003800380038007000700070007100E200E200E200E200 64003800>11 35 124 162 16 I[<1C0F002631C04740C08780E08780E08700E08700E00E01C0 0E01C00E01C00E01C01C03801C03801C03801C0704380708380E08380E103806107006203003C0 >22 21 123 148 36 110 D[<007E0001C3000381800701C00E01C01C01E03C01E03801E07801 E07801E07801E0F003C0F003C0F00380F00780700700700E00700C0030180018700007C000>19 21 123 148 33 I[<01C1F002621804741C08780C08700E08700E08701E00E01E00E01E00E01E 00E01E01C03C01C03C01C03C01C07803807003807003C0E003C1C0072380071E00070000070000 0E00000E00000E00000E00001C00001C00001C0000FFC000>23 31 127 148 33 I[<1C1F002620804741C08783C08703C08701808700000E00000E00000E00000E00001C 00001C00001C00001C0000380000380000380000380000700000300000>18 21 123 148 28 114 D[<00FC000183000200800401800C03800C03000C00000F00000FF00007 FC0003FE00003E00000F00000700700700F00600F00600E004004008002030001FC000>17 21 125 148 27 I[<00C001C001C001C001C003800380038003800700FFF8070007000E000E00 0E000E001C001C001C001C003800380038003810702070207040708031001E00>13 31 124 158 21 I[<1E00602300E04380E04381C08381C08701C08701C00703800E03800E0380 0E03801C07001C07001C07001C07081C0E10180E101C0E101C1E200C262007C3C0>21 21 123 148 35 I[<1E03802307C04387C04383C08381C08700C08700C00700800E00800E0080 0E00801C01001C01001C01001C02001C02001C04001C08001C08000C300003C000>18 21 123 148 29 I E /Fo 54 122 400 300 dfs[<0000C018000000C018000000C01800000180 300000018030000001803000000180300000030060000003006000000300600000030060000003 006000000600C000000600C000000600C000000600C000000C018000FFFFFFFFC0FFFFFFFFC000 180300000018030000001803000000180300000030060000003006000000300600000030060000 FFFFFFFFC0FFFFFFFFC000600C000000C018000000C018000000C018000000C018000001803000 0001803000000180300000018030000003006000000300600000030060000003006000000600C0 00000600C000000600C00000>34 45 125 162 55 35 D[<70F8FCFC7404040404080810102040 >6 15 124 132 19 44 D[11 2 127 139 21 I[<70F8F8F870>5 5 124 132 19 I[<01F000071C000C06001803003803803803807001C07001C07001C07001C0F0 01E0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F0 01E07001C07001C07001C07803C03803803803801C07000C0600071C0001F000>19 34 126 160 32 48 D[<008003800F80F380038003800380038003800380038003800380038003 80038003800380038003800380038003800380038003800380038003800380038007C0FFFE>15 33 124 160 32 I[<03F0000C1C001007002007804003C04003C08003E0F003E0F801E0F801E0 F801E02003E00003E00003C00003C0000780000700000E00001C0000180000300000600000C000 0180000100000200200400200800201800603000403FFFC07FFFC0FFFFC0>19 33 126 160 32 I[<03F8000C1E001007002007804007C07807C07803C07807C03807C0000780 000780000700000F00000E0000380003F000001C00000F000007800007800003C00003C00003E0 2003E07003E0F803E0F803E0F003C04003C0400780200780100F000C1C0003F000>19 34 126 160 32 I[<1000801E07001FFF001FFE001FF80013E000100000100000100000100000 10000010000010F800130E001407001803801003800001C00001C00001E00001E00001E00001E0 7001E0F001E0F001E0E001C08001C04003C04003802007001006000C1C0003F000>19 34 126 160 32 53 D[<007E0001C1000300800601C00E03C01C03C01801803800003800007800 00700000700000F0F800F30C00F40600F40300F80380F801C0F001C0F001E0F001E0F001E0F001 E0F001E07001E07001E07001E03801C03801C01803801C03000C0600070C0001F000>19 34 126 160 32 I[<01F800060E000803001001802001802000C06000C06000C06000C07000C0 7801803E01003F02001FC4000FF80003F80003FC00067F00083F80100F803007C06001C06000E0 C000E0C00060C00060C00060C000606000406000C03000801803000E0E0003F000>19 34 126 160 32 56 D[<01F000060C000C0600180700380380700380700380F001C0F001C0F001 C0F001E0F001E0F001E0F001E0F001E07001E07003E03803E01805E00C05E00619E003E1E00001 C00001C00001C0000380000380300300780700780600700C002018001030000FC000>19 34 126 160 32 I[<0001800000018000000180000003C0000003C0000003C0000005E0000005 E000000DF0000008F0000008F0000010F800001078000010780000203C0000203C0000203C0000 401E0000401E0000401E0000800F0000800F0000FFFF000100078001000780030007C0020003C0 020003C0040003E0040001E0040001E00C0000F00C0000F03E0001F8FF800FFF>32 35 126 162 49 65 D[29 34 126 161 47 I[<0007E0100038183000E0063001C00170038000F0070000F00E0000701E00 00701C0000303C0000303C0000307C0000107800001078000010F8000000F8000000F8000000F8 000000F8000000F8000000F8000000F800000078000000780000107C0000103C0000103C000010 1C0000201E0000200E000040070000400380008001C0010000E0020000381C000007E000>28 36 125 162 47 I[31 34 126 161 49 I[27 34 126 161 43 70 D[<0007F008003C0C1800E0021801C001B8038000F8070000780F0000381E 0000381E0000183C0000183C0000187C0000087800000878000008F8000000F8000000F8000000 F8000000F8000000F8000000F8000000F8001FFF780000F8780000787C0000783C0000783C0000 781E0000781E0000780F00007807000078038000B801C000B800E00318003C0C080007F000>32 36 125 162 51 I[32 34 126 161 49 I[<03FFF0001F00000F00000F00000F00000F00000F00000F00000F00000F00 000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00 000F00000F00000F00700F00F80F00F80F00F80E00F01E00401C0020380018700007C000>20 35 126 161 33 74 D[33 34 126 161 51 I[25 34 126 161 40 I[40 34 126 161 60 I[32 34 126 161 49 I[<000FE00000783C0000E00E0003C007800780 03C00F0001E00E0000E01E0000F03C0000783C0000787C00007C7C00007C7800003C7800003CF8 00003EF800003EF800003EF800003EF800003EF800003EF800003EF800003EF800003E7800003C 7C00007C7C00007C3C0000783E0000F81E0000F00F0001E00F0001E0078003C003C0078000E00E 0000783C00000FE000>31 36 125 162 51 I[27 34 126 161 44 I[33 35 126 161 48 82 D[<03F0200C0C601802603001E07000E0600060E00060E00060E00020E000 20E00020F00000F000007800007F00003FF0001FFE000FFF0003FF80003FC00007E00001E00000 F00000F0000070800070800070800070800070C00060C00060E000C0F000C0C80180C6070081FC 00>20 36 125 162 36 I[<7FFFFFF87807807860078018400780084007800840078008C00780 0C8007800480078004800780048007800400078000000780000007800000078000000780000007 800000078000000780000007800000078000000780000007800000078000000780000007800000 0780000007800000078000000780000007800000078000000FC00003FFFF00>30 34 126 161 47 I[47 35 127 161 67 87 D[<7FFFFE7E003E78003C7000786000784000F0C000F0C001E08003C08003C08007 80000780000F00001F00001E00003C00003C0000780000780000F00001F00001E00103C00103C0 010780010780030F00031E00021E00023C00063C000E78001EF8007EFFFFFE>24 34 125 161 40 90 D[<0FE0001838003C0C003C0E0018070000070000070000070000FF0007C7 001E07003C0700780700700700F00708F00708F00708F00F087817083C23900FC1E0>21 21 126 148 32 97 D[<0E0000FE00001E00000E00000E00000E00000E00000E00000E00000E00 000E00000E00000E00000E00000E1F000E61C00E80600F00300E00380E003C0E001C0E001E0E00 1E0E001E0E001E0E001E0E001E0E001E0E001C0E003C0E00380F00700C80600C41C0083F00>23 35 127 162 36 I[<01FE000703000C07801C0780380300780000700000F00000F00000F00000 F00000F00000F00000F000007000007800403800401C00800C010007060001F800>18 21 126 148 29 I[<0000E0000FE00001E00000E00000E00000E00000E00000E00000E00000E0 0000E00000E00000E00000E001F8E00704E00C02E01C01E03800E07800E07000E0F000E0F000E0 F000E0F000E0F000E0F000E0F000E07000E07800E03800E01801E00C02E0070CF001F0FE>23 35 126 162 36 I[<01FC000707000C03801C01C03801C07801E07000E0F000E0FFFFE0F00000 F00000F00000F00000F000007000007800203800201C00400E008007030000FC00>19 21 127 148 29 I[<003C00C6018F038F030F070007000700070007000700070007000700FFF8 07000700070007000700070007000700070007000700070007000700070007000700070007807F F8>16 35 128 162 20 I[<00007001F198071E180E0E181C07001C07003C07803C07803C0780 3C07801C07001C07000E0E000F1C0019F0001000001000001800001800001FFE000FFFC00FFFE0 3800F0600030400018C00018C00018C000186000306000303800E00E038003FE00>21 33 127 149 32 I[<0E0000FE00001E00000E00000E00000E00000E00000E00000E00000E0000 0E00000E00000E00000E00000E1F800E60C00E80E00F00700F00700E00700E00700E00700E0070 0E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E0070FFE7FF>24 35 127 162 36 I[<1C001E003E001E001C00000000000000000000000000000000000E00FE00 1E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00FFC0> 10 34 127 161 19 I[<01C003E003E003E001C00000000000000000000000000000000001E00F E001E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E0 00E000E000E000E000E060E0F0C0F18061803E00>11 44 130 161 20 I[<0E0000FE00001E00 000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E03FC0E01 F00E01C00E01800E02000E04000E08000E10000E38000EF8000F1C000E1E000E0E000E07000E07 800E03C00E01C00E01E00E00F00E00F8FFE3FE>23 35 127 162 35 I[<0E00FE001E000E000E 000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00 0E000E000E000E000E000E000E000E000E000E00FFE0>11 35 127 162 19 I[<0E1FC07F00FE60E183801E807201C00F003C00E00F003C00E00E003800E00E003800E00E 003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E0 0E003800E00E003800E00E003800E00E003800E00E003800E0FFE3FF8FFE>39 21 127 148 56 I[<0E1F80FE60C01E80E00F00700F00700E00700E00700E00700E00700E0070 0E00700E00700E00700E00700E00700E00700E00700E00700E00700E0070FFE7FF>24 21 127 148 36 I[<01FC000707000C01801800C03800E0700070700070F00078F00078F00078 F00078F00078F00078F000787000707800F03800E01C01C00E038007070001FC00>21 21 127 148 32 I[<0E1F00FE61C00E80600F00700E00380E003C0E001C0E001E0E001E0E001E 0E001E0E001E0E001E0E001E0E003C0E003C0E00380F00700E80E00E41C00E3F000E00000E0000 0E00000E00000E00000E00000E00000E00000E0000FFE000>23 31 127 148 36 I[<0E3CFE461E8F0F0F0F060F000E000E000E000E000E000E000E000E000E000E000E00 0E000E000F00FFF0>16 21 127 148 25 114 D[<0F8830786018C018C008C008E008F0007F80 3FE00FF001F8003C801C800C800CC00CC008E018D0308FC0>14 21 126 148 25 I[<02000200020002000600060006000E001E003E00FFF80E000E000E000E000E000E00 0E000E000E000E000E000E040E040E040E040E040E040708030801F0>14 31 127 158 25 I[<0E0070FE07F01E00F00E00700E00700E00700E00700E00700E00700E0070 0E00700E00700E00700E00700E00700E00700E00F00E00F006017003827800FC7F>24 21 127 148 36 I[23 21 127 148 35 I[32 21 127 148 47 I[23 31 127 148 35 121 D E /Fp 20 118 400 360 dfs[21 4 127 148 29 45 D[53 59 124 186 68 68 D[20 59 124 186 31 73 D[65 59 124 186 81 77 D[46 59 124 186 60 80 D[<000FF00080007FFE018001F00F81 8003C001C380070000E3800E000037801C00003F803C00001F803800000F807800000F80700000 07807000000780F000000380F000000380F000000380F000000380F000000180F800000180F800 000180FC000001807C000000007E000000003F000000003FC00000001FF00000000FFF0000000F FFF0000007FFFF000001FFFFC00000FFFFF000003FFFFC000003FFFE0000007FFF00000007FF00 0000007F800000001FC00000000FC000000007E000000003E000000003E000000001F0C0000001 F0C0000001F0C0000000F0C0000000F0C0000000F0E0000000F0E0000000F0E0000000E0F00000 00E0F0000001E0F8000001C0F8000001C0FC00000380FE00000780F700000700E1C0001E00E0F0 003C00C07E00F000C00FFFE0008001FF0000>36 61 124 187 49 83 D[<003F80000001C0F000 00030038000004001C00000C001E000018000F00001C000F80003E000780003F0007C0003F0007 C0003F0007C0001E0007C000000007C000000007C000000007C00000003FC000000FE7C000007E 07C00001F007C00007E007C0000F8007C0001F0007C0003F0007C0003E0007C0007E0007C0007C 0007C060FC0007C060FC0007C060FC0007C060FC000FC060FC000FC0607C000FC0607E0017C060 3E0023E0C01F0041F18007C180FF0000FE003E00>35 37 124 164 43 97 D[<0007F800003C0E0000F0018001E000C003C00060078000300F0000701F0000F81F0001F83E 0001F83E0001F87E0000F07C0000007C000000FC000000FC000000FC000000FC000000FC000000 FC000000FC000000FC000000FC0000007C0000007C0000007E0000003E0000003E00000C1F0000 0C1F0000180F8000180780003003C0006001E000C000F00180003C0E000007F800>30 37 125 164 39 99 D[<0000000700000000FF00000007FF00000007FF000000003F000000001F 000000001F000000001F000000001F000000001F000000001F000000001F000000001F00000000 1F000000001F000000001F000000001F000000001F000000001F000000001F000000001F000000 001F000000001F000003F81F00001E061F000070019F0001E000DF0003C0007F000780003F000F 80003F000F00001F001F00001F003E00001F003E00001F007E00001F007C00001F007C00001F00 FC00001F00FC00001F00FC00001F00FC00001F00FC00001F00FC00001F00FC00001F00FC00001F 00FC00001F007C00001F007C00001F007E00001F003E00001F003E00001F001E00001F001F0000 1F000F00003F000780007F0003C0005F0001E0009F0000F0031F80003C0E1FFC0007F01FFC>38 60 125 187 49 I[<000FF00000383C0000E00F0001C00780038003C0078001E00F0001F01F00 00F01E0000F83E0000F83E0000F87C00007C7C00007C7C00007CFC00007CFC00007CFFFFFFFCFC 000000FC000000FC000000FC000000FC000000FC0000007C0000007C0000007E0000003E000000 3E00000C1E00000C1F0000180F0000180780003003C0006001E000C000F00180003C0E000007F8 00>30 37 125 164 39 I[<0000FC000003830000070380000E07C0001E0FC0003C0FC0007C0F C0007C07800078000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8 000000F8000000F8000000F8000000F8000000F8000000F80000FFFFFC00FFFFFC0000F8000000 F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F80000 00F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F800 0000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8 000000F8000001FC00003FFFF0003FFFF000>26 60 127 187 27 I[<00000007C0000FE01860 003838207000F01E40F001E00F80F003C007806007C007C0000F8003E0000F8003E0000F8003E0 001F8003F0001F8003F0001F8003F0001F8003F0001F8003F0001F8003F0000F8003E0000F8003 E0000F8003E00007C007C00003C007800001E00F000003F01E00000238380000060FE000000400 00000004000000000C000000000C000000000E000000000E000000000700000000078000000003 FFFF000003FFFFF00001FFFFFC0000FFFFFE00078000FF000E00001F801C000007C038000003C0 78000003C070000001E0F0000001E0F0000001E0F0000001E0F0000001E0F0000001E078000003 C038000003803C000007801E00000F000700001C0003C0007800007803C000000FFE0000>36 56 126 165 43 I[<038007C00FE00FE00FE007C0038000000000000000000000000000000000 0000000000000000000000E01FE0FFE0FFE007E003E003E003E003E003E003E003E003E003E003 E003E003E003E003E003E003E003E003E003E003E003E003E003E003E003E003E003E003E003E0 07F0FFFFFFFF>16 57 126 184 23 105 D[<00E01FC0007F00001FE060780181E000FFE1803C 0600F000FFE2001E0800780007E4001F10007C0003E4001F10007C0003E8000F20003C0003F000 0FC0003E0003F0000FC0003E0003F0000FC0003E0003E0000F80003E0003E0000F80003E0003E0 000F80003E0003E0000F80003E0003E0000F80003E0003E0000F80003E0003E0000F80003E0003 E0000F80003E0003E0000F80003E0003E0000F80003E0003E0000F80003E0003E0000F80003E00 03E0000F80003E0003E0000F80003E0003E0000F80003E0003E0000F80003E0003E0000F80003E 0003E0000F80003E0003E0000F80003E0003E0000F80003E0003E0000F80003E0003E0000F8000 3E0003E0000F80003E0003E0000F80003E0007F0001FC0007F00FFFF83FFFE0FFFF8FFFF83FFFE 0FFFF8>61 37 125 164 74 109 D[<00E03FC0001FE0C0F000FFE1007800FFE2003C0007E400 3E0003E8001E0003E8001E0003F0001F0003F0001F0003F0001F0003E0001F0003E0001F0003E0 001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003 E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F00 03E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0007F0003F80FFFF87FF FCFFFF87FFFC>38 37 125 164 49 I[<0007F00000003C1E000000F007800001C001C0000380 00E000078000F0000F000078001E00003C001E00003C003E00003E003E00003E007C00001F007C 00001F007C00001F00FC00001F80FC00001F80FC00001F80FC00001F80FC00001F80FC00001F80 FC00001F80FC00001F80FC00001F807C00001F007C00001F007C00001F003E00003E003E00003E 001E00003C001F00007C000F00007800078000F00003C001E00001C001C00000F0078000003C1E 00000007F00000>33 37 125 164 43 I[<00E0FC001FE10600FFE20F00FFE41F8007E81F8003 E81F8003F00F0003F0060003F0000003F0000003E0000003E0000003E0000003E0000003E00000 03E0000003E0000003E0000003E0000003E0000003E0000003E0000003E0000003E0000003E000 0003E0000003E0000003E0000003E0000003E0000003E0000003E0000003E0000003E0000007F0 0000FFFFC000FFFFC000>25 37 125 164 33 114 D[<00FF02000700C6000C002E0010001E00 30001E0060000E0060000E00E0000600E0000600E0000600F0000600F8000600FC0000007F0000 003FF000003FFF80001FFFE00007FFF00001FFFC00003FFE000001FE0000003F00C0001F00C000 0F80C0000780E0000380E0000380E0000380E0000380F0000300F0000300F8000700F8000600E4 000C00E2001800C1807000807F8000>25 37 125 164 34 I[<00180000001800000018000000 1800000018000000380000003800000038000000380000007800000078000000F8000000F80000 01F8000003F8000007F800001FFFFE00FFFFFE0000F8000000F8000000F8000000F8000000F800 0000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8 000000F8000000F8000000F8000000F8000000F8018000F8018000F8018000F8018000F8018000 F8018000F8018000F8018000F801800078018000780300007C0300003C0200001E0600000F0C00 0003F000>25 53 127 180 33 I[<00E00007001FE000FF00FFE007FF00FFE007FF0007E0003F 0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E000 1F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0 001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003 E0003F0003E0003F0001E0003F0001E0005F0001F0009F0000F0009F000078011F80001C061FFC 0007F81FFC>38 37 125 164 49 I E /Fq 5 85 691 432 dfs[<0000001C000000001C000000 003C000000003C000000007C00000000FC00000000FC00000001FC00000001FC000000037C0000 00037C000000067C0000000E7C0000000C7C000000187C000000187C000000307C000000307E00 0000603E000000603E000000C03E000001C03E000001803E000003003E000003003E000006003E 000006003E00000FFFFE00000FFFFE000018003E000038003E000030003E000060003E00006000 3E0000C0003F0000C0001F000180001F000380001F000380001F000FC0003F00FFF007FFF8FFF0 07FFF0>37 42 124 169 70 65 D[<003FFFFF8000003FFFFFF0000001F801F8000001F0007C00 0001F0003E000001F0001F000001F0000F000003E0000F800003E0000F800003E0000F800003E0 0007800007C00007C00007C00007C00007C00007C00007C0000FC0000F80000FC0000F80000FC0 000F80000FC0000F80000FC0001F00000F80001F00001F80001F00001F80001F00001F80003E00 001F00003E00003F00003E00003E00003E00003E00007C00007C00007C00007C00007C0000F800 007C0000F80000F80001F00000F80003E00000F80003C00000F80007800001F0000F000001F000 3E000001F0007C000003F003F000007FFFFFC00000FFFFFE000000>42 41 124 168 72 68 D[<003FFFFFFF003FFFFFFF0001F8003F0001F0000F0001F0000E0001F0000E 0001F000060003E000060003E000060003E0000E0003E0000C0007C0000C0007C0180C0007C018 0C0007C01800000F803000000F803000000F807000000F80F000001FFFE000001FFFE000001F01 E000001F00E000003E00C000003E00C000003E00C000003E00C000007C018000007C000000007C 000000007C00000000F800000000F800000000F800000000F800000001F000000001F000000001 F000000003F00000007FFFC00000FFFFC00000>40 41 124 168 62 70 D[<003FFFFE00003FFFFFC00001F803F00001F001F80001F000F80001F0007C0001F0007C0003 E0007E0003E0007E0003E0007E0003E0007E0007C000FC0007C000FC0007C000F80007C001F800 0F8001F0000F8003E0000F8007C0000F800F00001F007C00001FFFE000001F00E000001F007800 003E003C00003E003C00003E003E00003E003E00007C003E00007C003E00007C003E00007C003E 0000F8007E0000F8007E0000F8007E0000F8007E0001F0007E0601F0007E0601F0007E0603F000 7E0C7FFF803E08FFFF801E1000000007E0>39 42 124 168 70 82 D[<0FFFFFFFF80FFFFFFFF8 1F803E01F81E007C00781C007C003038007C003038007C00303000F800303000F800306000F800 306000F800606001F00060C001F00060C001F000600001F000000003E000000003E000000003E0 00000003E000000007C000000007C000000007C000000007C00000000F800000000F800000000F 800000000F800000001F000000001F000000001F000000001F000000003E000000003E00000000 3E000000003E000000007C000000007C000000007C00000000FC000000FFFFF80000FFFFF80000 >37 41 117 168 69 84 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 400 TeXDict begin %%EndSetup %%Page: 0 1 bop 1060 927 a Fq(D)34 b(R)g(A)h(F)g(T)299 1049 y Fp(Do)r(cumen)n(t)30 b(for)g(a)g(Standard)h(Message-P)n(assi)q(ng)i(In)n(terface)913 1308 y Fo(Scott)23 b(Berryman,)c Fn(Y)-5 b(ale)25 b(Univ)933 1386 y Fo(James)c(Co)n(wnie,)g Fn(Meiko)j(Ltd)632 1463 y Fo(Jac)n(k)e (Dongarra,)i Fn(Univ.)32 b(of)23 b(T)-5 b(ennesse)m(e)26 b(and)f(ORNL)1068 1541 y Fo(Al)c(Geist,)f Fn(ORNL)1060 1618 y Fo(Bill)e(Gropp,)k Fn(ANL)1023 1696 y Fo(Rolf)f(Hemp)r(el,)c Fn(GMD)1016 1773 y Fo(Bob)22 b(Knigh)n(ten,)e Fn(Intel)1049 1851 y Fo(Rust)n(y)i(Lusk,)f Fn(ANL)818 1928 y Fo(Stev)n(e)h(Otto,)f Fn(Or)m(e)m(gon)k(Gr)m(aduate)f(Inst) 771 2006 y Fo(T)-5 b(on)n(y)21 b(Skjellum,)c Fn(Missisippi)k(State)26 b(Univ)871 2083 y Fo(Marc)c(Snir,)e Fn(IBM)j(T.)g(J.)f(Watson)992 2161 y Fo(Da)n(vid)g(W)-5 b(alk)n(er,)19 b Fn(ORNL)835 2238 y Fo(Stev)n(e)i(Zenith,)f Fn(Kuck)26 b(&)e(Asso)m(ciates)1099 2404 y Fo(April)c(20,)h(1993)116 2482 y(This)g(w)n(ork)h(w)n(as)h(supp)r (orted)h(b)n(y)d(ARP)-5 b(A)21 b(and)i(NSF)e(under)h(con)n(tract)i(n)n(um)n (b)r(er)c(###,)f(b)n(y)i(the)256 2559 y(National)h(Science)e(F)-5 b(oundation)23 b(Science)d(and)j(T)-5 b(ec)n(hnology)21 b(Cen)n(ter)h(Co)r (op)r(erativ)n(e)867 2636 y(Agreemen)n(t)f(No.)28 b(CCR-8809615.)p eop %%Page: 1 2 bop 100 477 a Fm(Chapter)44 b(1)100 756 y Fl(Colle)o(c)o(ti)n(v)l(e)k(Com)n (m)-10 b(uni)n(c)o(ati)o(on)1189 1054 y Fk(Al)20 b(Geist)1168 1129 y(Marc)g(Snir)100 1331 y Fj(1.1)94 b(In)m(tro)s(duction)100 1469 y Fk(This)28 b(section)h(is)f(a)g(draft)j(of)e(the)g(curren)n(t)i(prop)r (osal)g(for)e(collectiv)n(e)f(comm)n(unication.)51 b(Collectiv)n(e)100 1545 y(comm)n(unication)15 b(is)f(de\014ned)k(to)d(b)r(e)g(comm)n(unication)g (that)i(in)n(v)n(olv)n(es)f(a)f(group)i(of)f(pro)r(cesses.)26 b(Examples)100 1620 y(are)20 b(broadcast)j(and)e(global)f(sum.)26 b(A)18 b(collectiv)n(e)i(op)r(eration)i(is)d(executed)i(b)n(y)f(ha)n(ving)h (all)e(pro)r(cesses)h(in)100 1695 y(the)k(group)i(call)e(the)g(comm)n (unication)g(routine,)i(with)e(matc)n(hing)h(parameters.)38 b(Routines)24 b(can)h(\(but)100 1770 y(are)f(not)g(required)i(to\))e(return)j (as)c(so)r(on)h(as)f(their)h(participation)k(in)23 b(the)h(collectiv)n(e)g (comm)n(unication)100 1846 y(is)i(complete.)43 b(The)27 b(completion)f(of)h (a)f(call)g(indicates)i(that)g(the)f(caller)f(is)g(no)n(w)h(free)g(to)g (access)f(the)100 1921 y(lo)r(cations)j(in)g(the)g(comm)n(unication)f (bu\013er,)k(or)d(an)n(y)g(other)i(lo)r(cation)e(that)h(can)f(b)r(e)f (referenced)j(b)n(y)100 1996 y(the)26 b(collectiv)n(e)e(op)r(eration.)43 b(Ho)n(w)n(ev)n(er,)26 b(it)f(do)r(es)g(not)h(indicate)g(that)h(other)f(pro)r (cesses)g(in)f(the)g(group)100 2072 y(ha)n(v)n(e)c(started)i(the)d(op)r (eration)j(\(unless)e(otherwise)g(indicated)i(in)d(the)h(description)h(of)f (the)g(op)r(eration\).)100 2147 y(Ho)n(w)n(ev)n(er,)i(the)h(successful)f (completion)g(of)g(a)f(collectiv)n(e)h(comm)n(unication)f(call)g(ma)n(y)f (dep)r(end)k(on)e(the)100 2222 y(execution)f(of)f(a)g(matc)n(hing)g(call)f (at)h(all)f(pro)r(cesses)h(in)g(the)g(group.)221 2299 y(The)g(syn)n(tax)h (and)f(seman)n(tics)g(of)g(the)g(collectiv)n(e)f(op)r(erations)k(is)19 b(de\014ned)k(so)e(as)f(to)h(b)r(e)f(consisten)n(t)100 2374 y(with)h(the)g(syn)n(tax)h(and)g(seman)n(tics)e(of)h(the)h(p)r(oin)n(t)g(to)f (p)r(oin)n(t)h(op)r(erations.)221 2452 y(The)e(reader)h(is)e(referred)j(to)e (the)g(p)r(oin)n(t-to-p)r(oin)o(t)k(comm)n(unication)19 b(section)i(of)e(the) i(curren)n(t)h(MPI)100 2527 y(draft)g(for)e(information)i(concerning)g(comm)n (unication)d(bu\013ers)j(and)e(their)h(manipulations.)28 b(The)20 b(con-)100 2602 y(text)d(section)h(describ)r(es)f(the)h(formation,)g (manipulation,)h(and)f(query)g(functions)h(\(suc)n(h)f(as)f(group)i(size\)) 100 2677 y(that)j(are)f(a)n(v)m(ailable)h(for)f(groups)i(and)f(group)h(ob)s (jects.)221 2754 y(The)31 b(collectiv)n(e)g(comm)n(unication)g(routines)j (are)d(built)i(ab)r(o)n(v)n(e)f(the)g(p)r(oin)n(t-to-p)r(oin)n(t)k(routines.) 100 2830 y(While)17 b(v)n(endors)h(ma)n(y)e(optimize)f(certain)j(collectiv)n (e)f(routines)i(for)e(their)h(arc)n(hitectures,)j(a)16 b(complete)g(li-)100 2905 y(brary)22 b(of)e(the)h(collectiv)n(e)e(comm)n(unication)h(routines)i (can)e(b)r(e)g(written)i(en)n(tirely)f(using)g(p)r(oin)n(t-to-p)r(oin)n(t)100 2980 y(comm)n(unication)28 b(functions.)51 b(W)-5 b(e)27 b(are)h(using)h (naiv)n(e)f(implemen)n(tations)g(of)g(the)h(collectiv)n(e)e(calls)g(in)100 3056 y(terms)f(of)i(p)r(oin)n(t)h(to)e(p)r(oin)n(t)i(op)r(erations)h(in)d (order)i(to)f(pro)n(vide)h(an)f(op)r(erational)i(de\014nition)g(of)d(their) 100 3131 y(seman)n(tics.)221 3208 y(The)21 b(follo)n(wing)h(comm)n(unication) e(functions)k(are)d(prop)r(osed.)191 3340 y Fi(\017)31 b Fk(Broadcast)22 b(from)f(one)g(mem)n(b)r(er)d(to)j(all)f(mem)n(b)r(ers)f(of)i(a)f(group.)191 3472 y Fi(\017)31 b Fk(Barrier)22 b(across)f(all)f(group)j(mem)n(b)r(ers)191 3605 y Fi(\017)31 b Fk(Gather)21 b(data)i(from)d(all)g(group)j(mem)n(b)r(ers) 18 b(to)j(one)g(mem)n(b)r(er.)1285 3771 y(1)p eop %%Page: 2 3 bop 100 -134 a Fk(2)968 b Fh(CHAPTER)17 b(1.)48 b(COLLECTIVE)17 b(COMMUNICA)-5 b(TION)191 60 y Fi(\017)31 b Fk(Scatter)22 b(data)g(from)f (one)g(mem)n(b)r(er)d(to)j(all)g(mem)n(b)r(ers)d(of)j(a)g(group.)191 183 y Fi(\017)31 b Fk(Global)20 b(op)r(erations)j(suc)n(h)e(as)f(sum,)f(max,) g(min,)g(etc.,)h(w)n(ere)g(the)i(result)f(is)f(kno)n(wn)h(b)n(y)g(all)f (group)252 258 y(mem)n(b)r(ers)d(and)k(a)f(v)m(ariation)i(where)e(the)h (result)g(is)f(kno)n(wn)h(b)n(y)f(only)h(one)f(mem)n(b)r(er.)25 b(The)20 b(abilit)n(y)252 333 y(to)h(ha)n(v)n(e)g(user)h(de\014ned)h(global)e (op)r(erations.)191 456 y Fi(\017)31 b Fk(Sim)n(ultaneous)20 b(shift)f(of)f(data)i(around)h(the)e(group,)i(the)e(simplest)e(example)g(b)r (eing)i(all)f(mem)n(b)r(ers)252 531 y(sending)k(their)g(data)g(to)f (\(rank+1\))i(with)e(wrap)h(around.)191 653 y Fi(\017)31 b Fk(Scan)21 b(across)h(all)e(mem)n(b)r(ers)e(of)j(a)g(group)i(\(also)e(called) f(parallel)i(pre\014x\).)191 776 y Fi(\017)31 b Fk(Broadcast)22 b(from)f(all)f(mem)n(b)r(ers)e(to)j(all)f(mem)n(b)r(ers)f(of)i(a)f(group.)191 899 y Fi(\017)31 b Fk(Scatter)c(data)f(from)f(all)f(mem)n(b)r(ers)f(to)i(all) f(mem)n(b)r(ers)f(of)i(a)g(group)i(\(also)f(called)f(complete)f(ex-)252 974 y(c)n(hange)e(or)f(index\).)221 1092 y(T)-5 b(o)21 b(simplify)f(the)h (collectiv)n(e)g(comm)n(unication)g(in)n(terface)j(it)d(is)f(designed)j(with) e(t)n(w)n(o)h(la)n(y)n(ers.)30 b(The)100 1168 y(lo)n(w)c(lev)n(el)f(routines) k(ha)n(v)n(e)e(all)e(the)i(generalit)n(y)h(of,)g(and)f(mak)n(e)e(use)h(of,)i (the)f(comm)n(unication)f(bu\013er)100 1243 y(routines)c(of)f(the)g(p)r(oin)n (t-to-p)r(oin)n(t)k(section)20 b(whic)n(h)h(allo)n(ws)f(arbitrarily)j (complex)c(messages)f(to)j(b)r(e)f(con-)100 1318 y(structed.)29 b(The)20 b(second)g(lev)n(el)f(routines)j(are)d(similar)f(to)i(the)g(upp)r (er)i(lev)n(el)d(p)r(oin)n(t-to-p)r(oin)n(t)24 b(routines)d(in)100 1394 y(that)h(they)g(send)f(only)g(a)g(con)n(tiguous)i(bu\013er.)100 1583 y Fj(1.2)94 b(Group)32 b(F)-8 b(unctions)100 1719 y Fk(A)22 b(full)h(description)i(of)e(the)g(group)i(formation)f(and)f(manipulation)i (functions)g(can)e(b)r(e)f(found)j(in)e(the)100 1794 y(con)n(text)j(c)n (hapter)g(of)f(the)f(MPI)e(do)r(cumen)n(t.)38 b(Here)24 b(w)n(e)f(describ)r (e)i(only)f(those)h(group)h(functions)h(that)100 1869 y(are)21 b(used)h(in)e(the)i(seman)n(tic)e(description)j(of)e(the)g(collectiv)n(e)g (comm)n(unication)g(routines.)221 1944 y(An)h(initial)g(group)j(con)n (taining)g(all)c(pro)r(cesses)i(is)e(supplied)j(b)n(y)f(default)h(in)e(MPI.)e (MPI)h(pro)n(vides)100 2020 y(a)f(pro)r(cedure)j(that)f(returns)h(the)e (handle)h(to)e(this)h(initial)f(group.)30 b(The)20 b(pro)r(cesses)g(in)h(the) g(inital)f(group)100 2095 y(eac)n(h)30 b(ha)n(v)n(e)g(a)g(unique)g(rank)h(in) e(the)h(group)i(represen)n(ted)h(b)n(y)d(in)n(tegers)h(\(0,)h(1,)f(2,)g(...,) f([n)n(um)n(b)r(er-of-)100 2170 y(pro)r(cesses)21 b(-)g(1].)221 2293 y Fg(MPI)p 365 2293 21 3 v 26 w(ALLGR)n(OUP\(group\))f Fk(Return)k(the)f(descriptor)h(of)f(the)f(inital)h(group)h(con)n(taining)h (all)100 2415 y(pro)r(cesses.)100 2534 y Fg(OUT)d(group)32 b Fk(handle)22 b(to)f(descriptor)j(ob)s(ject)e(of)f(initial)g(group.)221 2700 y Fg(MPI)p 365 2700 V 26 w(RANK\(group,)i(rank\))e Fk(Return)h(the)g (rank)f(of)g(the)g(calling)g(pro)r(cess)g(within)h(the)f(sp)r(eci-)100 2822 y(\014ed)g(group.)100 2941 y Fg(IN)i(group)32 b Fk(group)23 b(handle)100 3063 y Fg(OUT)f(rank)31 b Fk(in)n(teger)221 3229 y Fg(MPI)p 365 3229 V 26 w(GSIZE\(group,)25 b(size\))20 b Fk(Return)j(the)e (n)n(um)n(b)r(er)h(of)g(pro)r(cesses)f(that)i(b)r(elong)f(to)f(the)h(sp)r (ec-)100 3352 y(i\014ed)f(group.)100 3470 y Fg(IN)i(group)32 b Fk(group)23 b(handle)100 3593 y Fg(OUT)f(size)30 b Fk(in)n(teger)p eop %%Page: 3 4 bop 100 -134 a Fh(1.3.)47 b(COMMUNICA)-5 b(TION)17 b(FUNCTIONS)1283 b Fk(3)100 60 y Fj(1.3)94 b(Comm)-6 b(uni)n(cati)o(on)27 b(F)-8 b(unctions)100 196 y Fk(The)15 b(prop)r(osed)i(comm)n(unication)e(functions)j (are)d(divided)h(in)n(to)g(t)n(w)n(o)g(la)n(y)n(ers.)26 b(The)15 b(lo)n(w)n(est)h(lev)n(el)e(uses)h(the)100 271 y(same)j(bu\013er)j (descriptor)h(ob)s(jects)f(a)n(v)m(ailable)f(in)f(p)r(oin)n(t-to-p)r(oin)o(t) 24 b(to)19 b(create)i(noncon)n(tiguou)q(s,)i(m)n(ultiple)100 346 y(data)k(t)n(yp)r(e)g(messages.)40 b(The)26 b(second)g(lev)n(el)g(is)f (similar)f(to)i(the)g(blo)r(c)n(k)g(send/receiv)n(e)i(p)r(oin)n(t-to-p)r(oin) n(t)100 422 y(op)r(erations)h(in)e(that)h(it)f(supp)r(orts)i(only)e(con)n (tiguous)j(bu\013ers)e(of)f(data.)47 b(F)-5 b(or)26 b(eac)n(h)i(comm)n (unication)100 497 y(op)r(eration,)23 b(w)n(e)d(list)h(these)g(t)n(w)n(o)g (lev)n(el)f(of)h(calls)f(together.)100 689 y Fj(1.4)94 b(Sync)m(hronization) 100 825 y Fg(Barrier)25 b(sync)n(hronization)100 987 y(MPI)p 244 987 21 3 v 25 w(BARRIER\()c(group)k(\))221 1110 y Fk(MPI)p 345 1110 19 3 v 21 w(BARRIER)18 b(blo)r(c)n(ks)k(the)h(calling)e(pro)r(cess)i (un)n(til)g(all)e(group)j(mem)n(b)r(ers)19 b(ha)n(v)n(e)k(called)f(it;)g(the) 100 1185 y(call)e(returns)k(at)d(an)n(y)g(pro)r(cess)g(only)g(after)i(all)d (group)j(mem)n(b)r(ers)18 b(ha)n(v)n(e)k(en)n(tered)h(the)e(call.)100 1311 y Fg(IN)i(group)32 b Fk(group)23 b(handle)221 1437 y Ff(MPI)p 318 1437 20 3 v 26 w(BARRIER)q(\()37 b(group)f(\))21 b Fk(is)100 1564 y Ff(MPI_CR)q(E)q(A)q(TE)q(_)q(B)q(U)q(FF)q(E)q(R)q(\()q(b)q(uf)q(f)q(e) q(r)q(_h)q(a)q(n)q(d)q(l)q(e,)37 b(MPI_BU)q(F)q(F)q(ER)q(,)g(MPI_PE)q(R)q(SI) q(S)q(T)q(E)q(NT)q(\))q(;)100 1639 y(MPI_GS)q(I)q(Z)q(E\()g(group,)g(&size)f (\);)100 1714 y(MPI_RA)q(N)q(K)q(\()g(group,)h(&rank)f(\);)100 1789 y(if)d(\(rank)q(==)q(0)q(\))100 1865 y({)195 1940 y(for)i(\(i=1;)h(i)c (<)h(size;)j(i++\))291 2015 y(MPI_RE)q(C)q(V\()q(b)q(u)q(f)q(f)q(er)q(_)q(h)q (a)q(nd)q(l)q(e)q(,)h(i,)c(tag,)i(group,)i(return)q(_)q(ha)q(n)q(d)q(l)q(e\)) q(;)195 2091 y(for)e(\(i=1;)h(i)c(<)h(size;)j(i++\))291 2166 y(MPI_SE)q(N)q(D\()q(b)q(u)q(f)q(f)q(er)q(_)q(h)q(a)q(nd)q(l)q(e)q(,)h(i,)c (tag,)i(group\))q(;)100 2241 y(})100 2316 y(else)100 2392 y({)195 2467 y(MPI_S)q(EN)q(D)q(\()q(b)q(uf)q(f)q(e)q(r)q(_)q(ha)q(n)q(d)q(l)q(e,)i (0,)d(tag,)h(group\))q(;)195 2542 y(MPI_R)q(EC)q(V)q(\()q(b)q(uf)q(f)q(e)q(r) q(_)q(ha)q(n)q(d)q(l)q(e,)i(0,)d(tag,)h(group,)h(return)q(_)q(h)q(a)q(n)q(dl) q(e)q(\))q(;)100 2617 y(})100 2693 y(MPI_FR)q(E)q(E)q(\(b)q(u)q(f)q(f)q(er)q (_)q(h)q(a)q(n)q(dl)q(e)q(\))q(;)221 2929 y Fe(Discussion:)45 b Fd(since)17 b(tag)g(w)n(as)g(v)n(oted)h(out)f(in)f(the)i(last)f(meeting,)f (a)h("safe")f(tag)h(m)n(ust)g(b)r(e)g(selected)i(to)e(use)100 3004 y(in)h(the)h(ab)r(o)n(v)n(e)g(description)100 3307 y Fj(1.5)94 b(Data)30 b(mo)m(v)m(e)e(functions)100 3443 y Fg(Broadcast)100 3605 y(MPI)p 244 3605 21 3 v 25 w(BCAST\()21 b(bu\013er)p 738 3605 V 25 w(handle,)i(group,)j(ro)r(ot)f(\))p eop %%Page: 4 5 bop 100 -134 a Fk(4)968 b Fh(CHAPTER)17 b(1.)48 b(COLLECTIVE)17 b(COMMUNICA)-5 b(TION)221 60 y Ff(MPI)p 318 60 20 3 v 26 w(BCAST)20 b Fk(broadcasts)g(a)d(message)f(from)h(the)g(pro)r(cess)h(with)g(rank)g Ff(root)i Fk(to)e(all)e(other)j(pro)r(cesses)100 135 y(of)25 b(the)h(group.)41 b(It)25 b(is)f(called)h(b)n(y)g(all)f(mem)n(b)r(ers)f(of)i (group)i(using)f(the)f(same)e(argumen)n(ts)j(for)g Ff(group,)100 211 y(and)34 b(root)p Fk(.)c(On)21 b(return)j(the)d(con)n(ten)n(ts)i(of)e (the)h(bu\013er)g(of)f(the)g(pro)r(cess)h(with)f(rank)g Ff(root)j Fk(is)c(con)n(tained)100 286 y(in)h(bu\013er)h(of)f(all)f(group)j(mem)n(b)r (ers.)100 422 y Fg(INOUT)f(bu\013er)p 542 422 21 3 v 25 w(handle)29 b Fk(Handle)21 b(for)h(bu\013er)h(where)e(from)f(message)f(is)h(sen)n(t)i(or) f(receiv)n(ed.)100 558 y Fg(IN)i(group)32 b Fk(group)23 b(handle)100 694 y Fg(IN)g(ro)r(ot)33 b Fk(rank)22 b(of)f(broadcast)i(ro)r(ot)f(\(in)n (teger\))221 877 y Fg(MPI)p 365 877 V 26 w(BCASTC\()e(buf,)j(len,)g(group,)i (ro)r(ot)h(\))221 1002 y Ff(MPI)p 318 1002 20 3 v 26 w(BCASTC)33 b Fk(b)r(eha)n(v)n(es)c(lik)n(e)f(broadcast,)34 b(restricted)d(to)e(a)f(blo)r (c)n(k)h(bu\013er.)53 b(It)28 b(is)g(called)h(b)n(y)g(all)100 1078 y(pro)r(cesses)21 b(with)g(the)h(same)c(argumen)n(ts)23 b(for)e Ff(len,)35 b(group)25 b Fk(and)d Ff(root)p Fk(.)100 1214 y Fg(INOUT)g(bu\013er)30 b Fk(Starting)24 b(address)e(of)f(bu\013er)h (\(c)n(hoice)g(t)n(yp)r(e\))100 1350 y Fg(IN)h(len)30 b Fk(Num)n(b)r(er)20 b(of)h(w)n(ords)h(in)e(bu\013er)j(\(in)n(teger\))100 1486 y Fg(IN)g(group)32 b Fk(group)23 b(handle)100 1622 y Fg(in)f(ro)r(ot)33 b Fk(rank)22 b(of)f(broadcast)j(ro)r(ot)d(\(in)n(teger\))221 1758 y Ff(MPI)p 318 1758 V 26 w(BCAST\()36 b(buffer)p 753 1758 V 28 w(handle)q(,)h(group,)f(root)f(\))21 b Fk(is)100 1896 y Ff(MPI_GS)q(I)q(Z)q(E\()37 b(group,)g(&size)f(\);)100 1972 y(MPI_RA)q(N)q(K)q(\()g(group,)h(&rank)f(\);)100 2047 y(MPI_IR)q(E)q(C)q(V\() q(h)q(a)q(n)q(dl)q(e)q(,)h(buffer)q(_h)q(a)q(n)q(d)q(l)q(e,)g(root,)f(tag,)f (group,)i(return)q(_h)q(a)q(n)q(d)q(l)q(e\))q(;)100 2122 y(if)c(\(rank)q(==)q (r)q(o)q(o)q(t\))195 2197 y(for)i(\(i=0;)h(i)c(<)h(size;)j(i++\))291 2273 y(MPI_SE)q(N)q(D\()q(b)q(u)q(f)q(f)q(er)q(_)q(h)q(a)q(nd)q(l)q(e)q(,)h (i,)c(tag,)i(group\))q(;)100 2348 y(MPI_WA)q(I)q(T)q(\(h)q(a)q(n)q(d)q(le)q (\))100 2524 y Fg(Circular)23 b(shift)100 2690 y(MPI)p 244 2690 21 3 v 25 w(CSHIFT\()f(in)n(buf,)h(outbuf,)h(group,)h(shift\))221 2816 y Fk(Pro)r(cess)30 b(with)h(rank)h Ff(i)f Fk(sends)g(the)g(data)h(in)f (its)f(input)j(bu\013er)f(to)f(pro)r(cess)h(with)f(rank)g(\()p Ff(i)22 b Fk(+)100 2891 y Ff(shift)p Fk(\))f(mo)r(d)16 b Ff(group)p 592 2891 20 3 v 27 w(size)p Fk(,)22 b(who)c(receiv)n(es)h(the)g(data)g(in)g (its)f(output)j(bu\013er.)28 b(All)17 b(pro)r(cesses)i(mak)n(e)e(the)100 2966 y(call)k(with)g(the)h(same)e(v)m(alues)h(for)h Ff(group)p Fk(,)j(and)d Ff(shift)p Fk(.)33 b(The)21 b Ff(shift)k Fk(v)m(alue)c(can)g(b)r (e)h(p)r(ositiv)n(e,)f(zero,)h(or)100 3042 y(negativ)n(e.)100 3197 y Fg(IN)h(in)n(buf)29 b Fk(handle)23 b(to)e(input)h(bu\013er)h (descriptor)100 3333 y Fg(OUT)f(outbuf)30 b Fk(handle)23 b(to)e(output)i (bu\013er)g(descriptor)100 3469 y Fg(IN)g(group)32 b Fk(handle)23 b(to)e(group)100 3605 y Fg(IN)i(shift)30 b Fk(in)n(teger)p eop %%Page: 5 6 bop 100 -134 a Fh(1.5.)47 b(D)n(A)-5 b(T)g(A)19 b(MO)n(VE)g(FUNCTIONS)1463 b Fk(5)221 60 y Fg(MPI)p 365 60 21 3 v 26 w(CSHIFT1\()22 b(buf,)h(group,)j (shift\))221 189 y Fk(Pro)r(cess)19 b(with)i(rank)g Ff(i)f Fk(sends)h(the)g(data)g(in)f(its)g(bu\013er)i(to)e(pro)r(cess)h(with)f(rank)i (\()p Ff(i)13 b Fk(+)f Ff(shift)p Fk(\))22 b(mo)r(d)100 264 y Ff(group)p 259 264 20 3 v 27 w(size)p Fk(,)g(who)e(receiv)n(es)f(the)h (data)g(in)g(the)f(same)f(bu\013er.)28 b(All)18 b(pro)r(cesses)i(mak)n(e)e (the)i(call)e(with)i(the)100 339 y(same)f(v)m(alues)h(for)i Ff(group)p Fk(,)i(and)e Ff(shift)p Fk(.)31 b(The)20 b Ff(shift)25 b Fk(v)m(alue)20 b(can)h(b)r(e)g(p)r(ositiv)n(e,)g(zero,)g(or)g(negativ)n(e.) 100 512 y Fg(INOUT)h(buf)30 b Fk(handle)22 b(to)f(bu\013er)i(descriptor)100 662 y Fg(IN)g(group)32 b Fk(handle)23 b(to)e(group)100 813 y Fg(IN)i(shift)30 b Fk(in)n(teger)221 1033 y Fg(MPI)p 365 1033 21 3 v 26 w(CSHIFTC\()21 b(in)n(buf,)h(outbuf,)i(len,)f(group,)i (shift\))221 1161 y Fk(Beha)n(v)n(es)j(lik)n(e)e Ff(MPI)p 675 1161 20 3 v 26 w(CSHIFT)q Fk(,)k(with)d(bu\013ers)i(restricted)g(to)f(b)r(e)e (blo)r(c)n(ks)h(of)h(n)n(umeric)f(units.)47 b(All)100 1237 y(pro)r(cesses)21 b(mak)n(e)e(the)j(call)e(with)h(the)g(same)e(v)m(alues)i (for)g Ff(len,)35 b(group)q Fk(,)24 b(and)d Ff(shift)q Fk(.)100 1387 y Fg(IN)i(in)n(buf)29 b Fk(initial)21 b(lo)r(cation)h(of)f(input)i (bu\013er)100 1537 y Fg(OUT)f(outbuf)30 b Fk(initial)21 b(lo)r(cation)h(of)f (output)j(bu\013er)100 1687 y Fg(IN)f(len)30 b Fk(n)n(um)n(b)r(er)21 b(of)g(en)n(tries)h(in)f(input)i(\(and)f(output\))i(bu\013ers)100 1837 y Fg(IN)f(group)32 b Fk(handle)23 b(to)e(group)100 1987 y Fg(IN)i(shift)30 b Fk(in)n(teger)221 2184 y Fg(MPI)p 365 2184 21 3 v 26 w(CSHIFTC1\()21 b(buf,)j(len,)e(group,)k(shift\))221 2313 y Fk(Beha)n(v)n(es)g(lik)n(e)e Ff(MPI)p 671 2313 20 3 v 25 w(CSHIFT)q(1)q Fk(,)29 b(with)c(bu\013ers)h(restricted)h(to)e(b)r(e)f (blo)r(c)n(ks)h(of)g(n)n(umeric)g(units.)40 b(All)100 2389 y(pro)r(cesses)21 b(mak)n(e)e(the)j(call)e(with)h(the)g(same)e(v)m(alues)i (for)g Ff(len,)35 b(group)q Fk(,)24 b(and)d Ff(shift)q Fk(.)100 2539 y Fg(INOUT)h(buf)30 b Fk(initial)21 b(lo)r(cation)g(of)g(bu\013er)100 2689 y Fg(IN)i(len)30 b Fk(n)n(um)n(b)r(er)21 b(of)g(en)n(tries)h(in)f(input) i(\(and)f(output\))i(bu\013ers)100 2839 y Fg(IN)f(group)32 b Fk(handle)23 b(to)e(group)100 2989 y Fg(IN)i(shift)30 b Fk(in)n(teger)221 3139 y Ff(MPI)p 318 3139 V 26 w(CSHIFT\()37 b(inbuf,)g(outbuf)q(,)f(group,)h (shift\))25 b Fk(is)100 3295 y Ff(MPI_GS)q(I)q(Z)q(E\()37 b(group,)g(&size)f (\);)100 3371 y(MPI_RA)q(N)q(K)q(\()g(group,)h(&rank)f(\);)100 3446 y(MPI_IS)q(E)q(N)q(D\()h(handle)q(,)g(inbuf,)f(mod\(ra)q(n)q(k)q(+)q(sh) q(i)q(f)q(t)q(,)g(size\),)h(tag,)e(group\))q(;)100 3521 y(MPI_RE)q(C)q(V)q (\()h(outbuf)q(,)h(mod\(ra)q(nk)q(-)q(s)q(h)q(i)q(ft)q(,)q(s)q(i)q(ze)q(\))q (,)g(tag,)e(group,)i(return_)q(h)q(a)q(n)q(d)q(le)q(\))100 3596 y(MPI_WA)q(I)q(T)q(\(h)q(a)q(n)q(d)q(le)q(\))q(;)p eop %%Page: 6 7 bop 100 -134 a Fk(6)968 b Fh(CHAPTER)17 b(1.)48 b(COLLECTIVE)17 b(COMMUNICA)-5 b(TION)100 60 y Fg(End-o\013)23 b(shift)100 222 y(MPI)p 244 222 21 3 v 25 w(EOSHIFT\()f(in)n(buf,)g(outbuf,)i(group,)i (shift\))221 344 y Fk(Pro)r(cess)21 b(with)h(rank)h Ff(i)p Fk(,)f(max)o(\()p Ff(0)p Fc(;)10 b Fi(\000)p Ff(shif)q(t)q Fk(\))23 b Fi(\024)18 b Ff(i)i Fc(<)e Ff(min)p Fk(\()p Ff(si)q(z)q(e)q Fc(;)10 b Ff(s)q(iz)q(e)19 b Fi(\000)14 b Ff(shift)q Fk(\),)25 b(sends)e(the)f(data)100 420 y(in)17 b(its)f(input)j(bu\013er)g(to)e(pro)r (cess)g(with)g(rank)h Ff(i+)34 b(shift)p Fk(,)20 b(who)d(receiv)n(es)g(the)h (data)g(in)f(its)f(output)k(bu\013er.)100 495 y(The)j(output)j(bu\013er)f(of) e(pro)r(cesses)g(whic)n(h)h(do)g(not)g(receiv)n(e)f(data)h(is)e(left)i(unc)n (hanged.)37 b(All)22 b(pro)r(cesses)100 570 y(mak)n(e)d(the)j(call)e(with)h (the)g(same)e(v)m(alues)h(for)i Ff(group)p Fk(,)i(and)e Ff(shift)p Fk(.)100 707 y Fg(IN)h(in)n(buf)29 b Fk(handle)23 b(to)e(input)h(bu\013er)h (descriptor)100 831 y Fg(OUT)f(outbuf)30 b Fk(handle)23 b(to)e(output)i (bu\013er)g(descriptor)100 954 y Fg(IN)g(group)32 b Fk(handle)23 b(to)e(group)100 1078 y Fg(IN)i(shift)30 b Fk(in)n(teger)221 1262 y Fg(MPI)p 365 1262 V 26 w(EOSHIFT1\()22 b(buf,)h(group,)j(shift\))221 1385 y Fk(Pro)r(cess)21 b(with)h(rank)h Ff(i)p Fk(,)f(max)o(\()p Ff(0)p Fc(;)10 b Fi(\000)p Ff(shif)q(t)q Fk(\))23 b Fi(\024)18 b Ff(i)i Fc(<)e Ff(min)p Fk(\()p Ff(si)q(z)q(e)q Fc(;)10 b Ff(s)q(iz)q(e)19 b Fi(\000)14 b Ff(shift)q Fk(\),)25 b(sends)e(the)f(data)100 1460 y(in)e(its)h(bu\013er)h(to)e(pro)r(cess)h(with)g(rank)h Ff(i+)33 b(shift)p Fk(,)24 b(who)d(receiv)n(es)f(the)h(data)h(in)e(the)h (same)e(bu\013er.)29 b(The)100 1535 y(output)24 b(bu\013er)f(of)e(pro)r (cesses)g(whic)n(h)h(do)f(not)h(receiv)n(e)f(data)h(is)e(left)i(unc)n (hanged.)31 b(All)20 b(pro)r(cesses)h(mak)n(e)100 1610 y(the)g(call)f(with)i (the)f(same)e(v)m(alues)h(for)i Ff(group)p Fk(,)i(and)e Ff(shift)p Fk(.)100 1747 y Fg(IN)h(in)n(buf)29 b Fk(handle)23 b(to)e(input)h(bu\013er)h (descriptor)100 1871 y Fg(OUT)f(outbuf)30 b Fk(handle)23 b(to)e(output)i (bu\013er)g(descriptor)100 1995 y Fg(IN)g(group)32 b Fk(handle)23 b(to)e(group)100 2118 y Fg(IN)i(shift)30 b Fk(in)n(teger)221 2302 y Fg(MPI)p 365 2302 V 26 w(EOSHIFTC\()20 b(in)n(buf,)j(outbuf,)h(len,)e (group,)k(shift\))221 2425 y Fk(Beha)n(v)n(es)g(lik)n(e)e Ff(MPI)p 671 2425 20 3 v 25 w(EOSHIF)q(T)q Fk(,)k(with)d(bu\013ers)h(restricted)h(to)e (b)r(e)g(blo)r(c)n(ks)g(of)g(n)n(umeric)g(units.)40 b(All)100 2500 y(pro)r(cesses)21 b(mak)n(e)e(the)j(call)e(with)h(the)g(same)e(v)m (alues)i(for)g Ff(len,)35 b(group)q Fk(,)24 b(and)d Ff(shift)q Fk(.)100 2622 y Fg(IN)i(in)n(buf)29 b Fk(initial)21 b(lo)r(cation)h(of)f (input)i(bu\013er)100 2745 y Fg(OUT)f(outbuf)30 b Fk(initial)21 b(lo)r(cation)h(of)f(output)j(bu\013er)100 2869 y Fg(IN)f(len)30 b Fk(n)n(um)n(b)r(er)21 b(of)g(en)n(tries)h(in)f(input)i(\(and)f(output\))i (bu\013ers)100 2993 y Fg(IN)f(group)32 b Fk(handle)23 b(to)e(group)100 3117 y Fg(IN)i(shift)30 b Fk(in)n(teger)221 3285 y Fg(MPI)p 365 3285 21 3 v 26 w(EOSHIFTC1\()21 b(buf,)j(len,)e(group,)k(shift\))221 3408 y Fk(Beha)n(v)n(es)f(lik)n(e)f Ff(MPI)p 670 3408 20 3 v 25 w(EOSHIF)q(T)q(1)p Fk(,)29 b(with)24 b(bu\013er)i(restricted)g(to)f(b)r (e)f(blo)r(c)n(ks)g(of)g(n)n(umeric)g(units.)39 b(All)100 3483 y(pro)r(cesses)21 b(mak)n(e)e(the)j(call)e(with)h(the)g(same)e(v)m(alues)i (for)g Ff(len,)35 b(group)q Fk(,)24 b(and)d Ff(shift)q Fk(.)100 3605 y Fg(INOUT)h(buf)30 b Fk(initial)21 b(lo)r(cation)g(of)g(bu\013er)p eop %%Page: 7 8 bop 100 -134 a Fh(1.5.)47 b(D)n(A)-5 b(T)g(A)19 b(MO)n(VE)g(FUNCTIONS)1463 b Fk(7)100 60 y Fg(IN)23 b(len)30 b Fk(n)n(um)n(b)r(er)21 b(of)g(en)n(tries)h (in)f(input)i(\(and)f(output\))i(bu\013ers)100 182 y Fg(IN)f(group)32 b Fk(handle)23 b(to)e(group)100 303 y Fg(IN)i(shift)30 b Fk(in)n(teger)100 461 y Fg(Gather)100 623 y(MPI)p 244 623 21 3 v 25 w(GA)-6 b(THER\()21 b(in)n(buf,)i(outbuf,)h(group,)h(ro)r(ot,)h(return)p 1643 623 V 26 w(status\))221 746 y Fk(Eac)n(h)d(pro)r(cess)h(\(including)h(the)f(ro)r (ot)g(pro)r(cess\))g(sends)g(the)g(con)n(ten)n(t)h(of)f(its)f(input)i (bu\013er)f(to)g(the)100 821 y(ro)r(ot)f(pro)r(cess.)32 b(The)22 b(ro)r(ot)h(pro)r(cess)f(concatenates)j(all)c(the)i(incoming)f(messages)e(in) i(the)h(order)g(of)g(the)100 896 y(senders')18 b(rank)h(and)f(places)f(the)h (results)g(in)f(its)g(output)j(bu\013er.)28 b(It)17 b(is)g(called)g(b)n(y)g (all)g(mem)n(b)r(ers)e(of)i(group)100 971 y(using)k(the)f(same)e(argumen)n (ts)j(for)f Ff(group)p Fk(,)k(and)c Ff(root)p Fk(.)30 b(The)20 b(input)h(bu\013er)h(of)e(eac)n(h)g(pro)r(cess)g(ma)n(y)e(ha)n(v)n(e)100 1047 y(a)j(di\013eren)n(t)i(length.)100 1163 y Fg(IN)g(in)n(buf)29 b Fk(handle)23 b(to)e(input)h(bu\013er)h(descriptor)100 1284 y Fg(OUT)f(outbuf)30 b Fk(handle)23 b(to)e(output)i(bu\013er)g(descriptor)g ({)e(signi\014can)n(t)i(only)e(at)g(ro)r(ot)h(\(c)n(hoice\))100 1406 y Fg(IN)h(group)32 b Fk(group)23 b(handle)100 1527 y Fg(IN)g(ro)r(ot)33 b Fk(rank)22 b(of)f(receiving)g(pro)r(cess)h(\(in)n(teger\))100 1649 y Fg(OUT)g(return)p 475 1649 V 26 w(status)32 b Fk(return)23 b(status)f(handle)221 1812 y Fg(MPI)p 365 1812 V 26 w(GA)-6 b(THER)n(C\()20 b(in)n(buf,)j(inlen,)e(outbuf,)j(group,)i(ro)r(ot\))221 1934 y Ff(MPI)p 318 1934 20 3 v 26 w(GATHERC)j Fk(b)r(eha)n(v)n(es)c(lik)n(e) f Ff(MPI)p 1027 1934 V 25 w(GATHER)k Fk(restricted)f(to)d(blo)r(c)n(k)g (bu\013ers,)j(and)e(with)g(the)f(addi-)100 2010 y(tional)g(restriction)h (that)g(all)d(input)j(bu\013ers)g(should)f(ha)n(v)n(e)g(the)g(same)d(length.) 35 b(All)22 b(pro)r(cesses)h(should)100 2085 y(pro)n(vided)g(the)f(same)d(v)m (alues)h(for)i Ff(inlen,)37 b(group)p Fk(,)24 b(and)d Ff(root)j Fk(.)100 2201 y Fg(IN)f(in)n(buf)29 b Fk(\014rst)22 b(v)m(ariable)g(of)f (input)i(bu\013er)f(\(c)n(hoice\))100 2322 y Fg(IN)h(inlen)29 b Fk(Num)n(b)r(er)20 b(of)h(\(w)n(ord\))i(v)m(ariables)e(in)g(input)i (bu\013er)f(\(in)n(teger\))100 2444 y Fg(OUT)g(outbuf)30 b Fk(\014rst)22 b(v)m(ariable)g(of)f(output)j(bu\013er)e({)f(signi\014can)n(t)i (only)e(at)g(ro)r(ot)h(\(c)n(hoice\))100 2565 y Fg(IN)h(group)32 b Fk(group)23 b(handle)100 2687 y Fg(IN)g(ro)r(ot)33 b Fk(rank)22 b(of)f(receiving)g(pro)r(cess)h(\(in)n(teger\))221 2803 y Ff(MPI)p 318 2803 V 26 w(GATHERC)q(\()37 b(inbuf,)g(inlen,)f(outbuf)q(,)h(tag,)e (group,)i(root\))56 b Fk(is)100 2919 y Ff(MPI_GS)q(I)q(Z)q(E\()37 b(&size,)g(group\))q(;)100 2994 y(MPI_RA)q(N)q(K)q(\()f(&rank,)h(group\))q(;) 100 3069 y(MPI_IS)q(E)q(N)q(DC)q(\()q(h)q(a)q(nd)q(l)q(e)q(,)g(inbuf,)f (inlen,)h(root,)f(tag,)f(group\))q(;)100 3145 y(if)e(\(rank)q(==)q(r)q(o)q(o) q(t\))195 3220 y(for)i(\(i=0;)h(i)c(<)h(size;)j(i++\))195 3295 y({)291 3371 y(MPI_RE)q(C)q(VC)q(\()q(o)q(u)q(t)q(bu)q(f)q(,)h(inlen,)f(i,)e (tag,)h(group,)i(return)q(\\_)q(s)q(t)q(a)q(tu)q(s)q(\))q(;)291 3446 y(outbuf)g(+=)c(inlen;)195 3521 y(})100 3596 y(MPI_WA)q(I)q(T)q(\(h)q(a) q(n)q(d)q(le)q(\))q(;)p eop %%Page: 8 9 bop 100 -134 a Fk(8)975 b Fh(CHAPTER)17 b(1.)41 b(COLLECTIVE)17 b(COMMUNICA)-5 b(TION)100 60 y Fg(Scatter)100 222 y(MPI)p 244 222 21 3 v 25 w(SCA)f(TTER\()21 b(list)p 745 222 V 24 w(of)p 824 222 V 26 w(in)n(bufs,)i(outbuf,)h(group,)i(ro)r(ot,)g(return)p 1899 222 V 26 w(status\))221 345 y Fk(The)21 b(ro)r(ot)g(pro)r(cess)g(sends)g (the)g(con)n(ten)n(t)j(of)d(its)f Ff(i)p Fk(-th)i(input)g(bu\013er)h(to)e (the)g(pro)r(cess)g(with)g(rank)g Ff(i)p Fk(;)100 420 y(eac)n(h)h(pro)r(cess) f(\(including)j(the)d(ro)r(ot)i(pro)r(cess\))f(stores)g(the)f(incoming)g (message)e(in)i(its)g(output)j(bu\013er.)100 495 y(The)19 b(routine)h(is)e (called)h(b)n(y)g(all)f(mem)n(b)r(ers)e(of)j(the)h(group)h(using)e(the)g (same)e(argumen)n(ts)j(for)g Ff(group)p Fk(,)i(and)100 571 y Ff(root)p Fk(.)100 696 y Fg(IN)h(list)p 302 696 V 25 w(of)p 382 696 V 26 w(in)n(bufs)30 b Fk(list)20 b(of)h(bu\013er)h(descriptor)i (handles)100 822 y Fg(OUT)e(outbuf)30 b Fk(bu\013er)23 b(descriptor)g(handle) 100 948 y Fg(IN)g(group)32 b Fk(handle)100 1073 y Fg(IN)23 b(ro)r(ot)33 b Fk(rank)22 b(of)f(sending)h(pro)r(cess)f(\(in)n(teger\))100 1199 y Fg(OUT)h(return)p 475 1199 V 26 w(status)32 b Fk(return)23 b(status)f(handle)221 1325 y Ff(MPI)p 318 1325 20 3 v 26 w(SCATTER)q(\()37 b(list)p 754 1325 V 26 w(of)p 842 1325 V 25 w(inbufs)q(,)f(outbuf)q(,)h (group,)f(root,)g(return)p 1976 1325 V 28 w(status)q(\))25 b Fk(is)100 1451 y Ff(MPI_GS)q(I)q(Z)q(E\()37 b(group,)g(&size)f(\);)100 1526 y(MPI_RA)q(N)q(K)q(\()g(group,)h(&rank)f(\);)100 1601 y(MPI_IR)q(E)q(C)q(V\()q(h)q(a)q(n)q(dl)q(e)q(,)h(outbuf)q(,)f(root,)g(tag,)f (group\))q(;)100 1676 y(if)e(\(rank)q(=r)q(o)q(o)q(t)q(\))195 1752 y(for)i(\(i=0;)h(i)c(<)h(size;)j(i++\))291 1827 y(MPI_SE)q(N)q(D\()q(i)q (n)q(b)q(u)q(f[)q(i)q(])q(,)g(i,)e(tag,)h(group\))q(;)100 1902 y(MPI_WA)q(I)q(T)q(\(h)q(a)q(n)q(d)q(le)q(,)i(return)q(\\)q(_s)q(t)q(a)q(t)q (u)q(s\))q(;)221 2075 y Fg(MPI)p 365 2075 21 3 v 26 w(SCA)-6 b(TTER)n(C\()19 b(in)n(buf,)k(outbuf,)h(len,)e(group,)k(ro)r(ot\))221 2198 y Ff(MPI)p 318 2198 20 3 v 26 w(SCATTER)q(C)i Fk(b)r(eha)n(v)n(es)c(lik) n(e)e Ff(MPI)p 1055 2198 V 25 w(SCATTE)q(R)27 b Fk(restricted)f(to)d(blo)r(c) n(k)g(bu\013ers,)i(and)f(with)f(the)h(ad-)100 2273 y(ditional)j(restriction)h (that)f(all)e(output)k(bu\013ers)e(ha)n(v)n(e)f(the)g(same)e(length.)43 b(The)26 b(input)h(bu\013er)g(blo)r(c)n(k)100 2348 y(of)d(the)h(ro)r(ot)f (pro)r(cess)h(is)e(partitioned)k(in)n(to)e Ff(n)f Fk(consecutiv)n(e)h(blo)r (c)n(ks,)g(eac)n(h)f(consisting)i(of)e Ff(len)h Fk(w)n(ords.)100 2424 y(The)19 b Ff(i)p Fk(-th)i(blo)r(c)n(k)f(is)f(sen)n(t)h(to)g(the)g Ff(i)p Fk(-th)h(pro)r(cess)f(in)f(the)h(group)i(and)e(stored)h(in)e(its)h (output)i(bu\013er.)29 b(The)100 2499 y(routine)e(is)c(called)h(b)n(y)h(all)f (mem)n(b)r(ers)e(of)j(the)g(group)h(using)f(the)g(same)e(argumen)n(ts)j(for)f Ff(group,)37 b(len)p Fk(,)100 2574 y(and)22 b Ff(root)p Fk(.)100 2700 y Fg(IN)h(in)n(buf)29 b Fk(\014rst)22 b(en)n(try)h(in)e(input)h (bu\013er)h({)d(signi\014can)n(t)k(only)d(at)g(ro)r(ot)g(\(c)n(hoice\).)100 2826 y Fg(OUT)h(outbuf)30 b Fk(\014rst)22 b(en)n(try)h(in)e(output)i (bu\013er)g(\(c)n(hoice\).)100 2951 y Fg(IN)g(len)30 b Fk(n)n(um)n(b)r(er)21 b(of)g(en)n(tries)h(to)f(b)r(e)g(stored)h(in)f(output)j(bu\013er)e(\(in)n (teger\))100 3077 y Fg(IN)h(group)32 b Fk(handle)100 3203 y Fg(IN)23 b(ro)r(ot)33 b Fk(rank)22 b(of)f(sending)h(pro)r(cess)f(\(in)n (teger\))221 3328 y Ff(MPI)p 318 3328 V 26 w(SCATTER)q(C)q(\()37 b(inbuf,)f(outbu)q(f,)h(outlen)q(,)g(group,)f(root\))56 b Fk(is)100 3454 y Ff(MPI_GS)q(I)q(Z)q(E\()37 b(&size,)g(group\))q(;)100 3530 y(MPI_RA)q(N)q(K)q(\()f(&rank,)h(group\))q(;)100 3605 y(MPI_IR)q(E)q(C)q(VC)q(\()g(handle)q(,)f(outbuf)q(,)h(outlen)q(,)f(root,)g (tag,)f(group)q(\);)p eop %%Page: 9 10 bop 100 -134 a Fh(1.5.)41 b(D)n(A)-5 b(T)g(A)18 b(MO)n(VE)h(FUNCTIONS)1470 b Fk(9)100 60 y Ff(if)33 b(\(rank)q(=r)q(o)q(o)q(t)q(\))195 135 y(for)i(\(i=0;)h(i)c(<)h(size;)j(i++\))195 211 y({)291 286 y(MPI_SE)q(N)q(DC)q(\()q(i)q(n)q(b)q(uf)q(,)h(outlen)q(,)f(i,)e(tag,)h (group,)i(return)q(\\_)q(s)q(t)q(a)q(tu)q(s)q(\))q(;)291 361 y(inbuf)f(+=)d(outlen)q(;)195 437 y(})100 512 y(MPI_WA)q(I)q(T)q(\(h)q(a)q(n) q(d)q(le)q(\))q(;)100 671 y Fg(All-to-all)23 b(scatter)100 833 y(MPI)p 244 833 21 3 v 25 w(ALLSCA)-6 b(TTER\()20 b(list)p 881 833 V 25 w(of)p 961 833 V 26 w(in)n(bufs,)j(outbuf,)h(group,)h(return)p 1865 833 V 26 w(status\))221 955 y Fk(Eac)n(h)g(pro)r(cess)g(in)f(the)h (group)h(sends)f(its)g Ff(i)p Fk(-th)h(bu\013er)g(in)e(its)h(input)h (bu\013er)g(list)e(to)h(the)g(pro)r(cess)100 1031 y(with)18 b(rank)g Ff(i)g Fk(\(itself)g(included\);)j(eac)n(h)d(pro)r(cess)g (concatenates)h(the)g(incoming)d(messages)g(in)h(its)h(output)100 1106 y(bu\013er,)j(in)e(the)g(order)i(of)e(the)g(senders')h(ranks.)28 b(The)19 b(routine)i(is)d(called)h(b)n(y)g(all)f(mem)n(b)r(ers)f(of)i(the)g (group)100 1181 y(using)j(the)f(same)e(argumen)n(ts)j(for)g Ff(group)p Fk(.)100 1303 y Fg(IN)h(list)p 302 1303 V 25 w(of)p 382 1303 V 26 w(in)n(bufs)30 b Fk(list)20 b(of)h(bu\013er)h(descriptor)i (handles)100 1427 y Fg(OUT)e(outbuf)30 b Fk(bu\013er)23 b(descriptor)g (handle)100 1550 y Fg(IN)g(group)32 b Fk(handle)100 1674 y Fg(OUT)22 b(return)p 475 1674 V 26 w(status)32 b Fk(return)23 b(status)f(handle)221 1843 y Fg(MPI)p 365 1843 V 26 w(ALLSCA)-6 b(TTER)n(C)o(\()20 b(in)n(buf,)i(outbuf,)i(len,)f(group\))221 1965 y Ff(MPI)p 318 1965 20 3 v 26 w(ALLSCAT)q(T)q(E)q(R)q(C)29 b Fk(b)r(eha)n(v)n(es)e(lik)n(e)e Ff(MPI)p 1158 1965 V 25 w(ALLSCA)q(T)q(TE)q (R)30 b Fk(restricted)e(to)d(blo)r(c)n(k)h(bu\013ers,)i(and)e(with)100 2041 y(the)f(additional)i(restriction)h(that)e(all)e(blo)r(c)n(ks)h(sen)n(t)g (from)g(one)g(pro)r(cess)g(to)g(another)i(ha)n(v)n(e)f(the)f(same)100 2116 y(length.)j(The)18 b(input)i(bu\013er)h(blo)r(c)n(k)d(of)h(eac)n(h)g (pro)r(cess)g(is)e(partitioned)22 b(in)n(to)e Ff(n)e Fk(consecutiv)n(e)i(blo) r(c)n(ks,)f(eac)n(h)100 2191 y(consisting)26 b(of)g Ff(len)g Fk(w)n(ords.)42 b(The)24 b Ff(i)p Fk(-th)j(blo)r(c)n(k)e(is)f(sen)n(t)i(to)f (the)h Ff(it)p Fk(-th)h(pro)r(cess)f(in)f(the)g(group.)43 b(Eac)n(h)100 2267 y(pro)r(cess)24 b(concatenates)h(the)f(incoming)e(messages,)g(in)h(the)g (order)i(of)e(the)h(senders')g(ranks,)g(and)h(store)100 2342 y(them)18 b(in)g(its)g(output)k(bu\013er.)28 b(The)18 b(routine)j(is)d (called)g(b)n(y)h(all)f(mem)n(b)r(ers)e(of)j(the)g(group)i(using)e(the)g (same)100 2417 y(argumen)n(ts)j(for)g Ff(group)p Fk(,)i(and)e Ff(len)p Fk(.)100 2539 y Fg(IN)h(in)n(buf)29 b Fk(\014rst)22 b(en)n(try)h(in)e(input)h(bu\013er)h(\(c)n(hoice\).)29 b(ro)r(ot)21 b(\(in)n(teger\))100 2662 y Fg(OUT)h(outbuf)30 b Fk(\014rst)22 b(en)n(try)h(in)e(output)i(bu\013er)g(\(c)n(hoice\).)100 2786 y Fg(IN)g(len)30 b Fk(n)n(um)n(b)r(er)21 b(of)g(en)n(tries)h(sen)n(t)g(from)e (eac)n(h)i(pro)r(cess)f(to)g(eac)n(h)g(other)i(\(in)n(teger\).)100 2910 y Fg(IN)g(group)32 b Fk(handle)221 3031 y Ff(MPI)p 318 3031 V 26 w(ALLSCAT)q(T)q(E)q(R)q(C)q(\()k(inbuf,)h(outbuf)q(,)f(len,)g (group\))25 b Fk(is)100 3153 y Ff(MPI_GS)q(I)q(Z)q(E\()37 b(group,)g(&size)f (\);)100 3228 y(MPI_RA)q(N)q(K)q(\()g(group,)h(&rank)f(\);)100 3304 y(for)e(\(i=0;)i(i)d(<)f(rank;)k(i++\))195 3379 y({)227 3454 y(MPI_IR)q(E)q(C)q(V)q(C\()q(r)q(e)q(c)q(v)q(_h)q(a)q(n)q(d)q(le)q(s)q ([)q(i)q(])q(,)g(outbuf)q(,)h(len,)e(tag,)g(group\))q(;)227 3530 y(outbuf)i(+=)c(len;)195 3605 y(})p eop %%Page: 10 11 bop 100 -134 a Fk(10)945 b Fh(CHAPTER)17 b(1.)41 b(COLLECTIVE)17 b(COMMUNICA)-5 b(TION)100 60 y Ff(for)34 b(\(i=0;)i(i)d(<)f(size;)k(i++\))195 135 y({)227 211 y(MPI_IS)q(E)q(N)q(D)q(C\()q(s)q(e)q(n)q(d)q(_h)q(a)q(n)q(d)q (le)q([)q(i)q(])q(,)g(inbuf,)h(len,)e(i,)f(tag,)h(group\))q(;)227 286 y(inbuf)h(+=)e(len;)195 361 y(})100 437 y(MPI_WA)q(I)q(T)q(AL)q(L)q(\()q (s)q(en)q(d)q(_)q(h)q(a)q(nd)q(l)q(e)q(\))q(;)100 512 y(MPI_WA)q(I)q(T)q(AL)q (L)q(\()q(r)q(ec)q(v)q(_)q(h)q(a)q(nd)q(l)q(e)q(\))q(;)100 671 y Fg(All-to-all)23 b(broadcast)100 833 y(MPI)p 244 833 21 3 v 25 w(ALLCAST\()e(in)n(buf,)i(outbuf,)h(group,)h(return)p 1497 833 V 26 w(status\))221 955 y Fk(Eac)n(h)19 b(pro)r(cess)g(in)f(the)h (group)i(broadcasts)g(its)e(input)h(bu\013er)h(to)d(all)g(pro)r(cesses)h (\(including)i(itself)5 b(\);)100 1031 y(eac)n(h)24 b(pro)r(cess)h (concatenates)h(the)e(incoming)g(messages)e(in)h(its)h(output)j(bu\013er,)f (in)e(the)g(order)i(of)e(the)100 1106 y(senders')19 b(ranks.)27 b(The)18 b(routine)i(is)d(called)g(b)n(y)h(all)g(mem)n(b)r(ers)d(of)j(the)g (group)i(using)f(the)f(same)e(argumen)n(ts)100 1181 y(for)22 b Ff(group)p Fk(.)100 1303 y Fg(IN)h(in)n(buf)29 b Fk(bu\013er)23 b(descriptor)g(handle)g(for)f(input)g(bu\013er)100 1427 y Fg(OUT)g(outbuf)30 b Fk(bu\013er)23 b(descriptor)g(handle)g(for)f(output)h(bu\013er)100 1550 y Fg(IN)g(group)32 b Fk(handle)100 1674 y Fg(OUT)22 b(return)p 475 1674 V 26 w(status)32 b Fk(return)23 b(status)f(handle)221 1843 y Fg(MPI)p 365 1843 V 26 w(ALLCASTC\()e(in)n(buf,)i(outbuf,)i(len,)f (group\))221 1965 y Ff(MPI)p 318 1965 20 3 v 26 w(ALLCAST)q(C)28 b Fk(b)r(eha)n(v)n(es)c(lik)n(e)e Ff(MPI)p 1055 1965 V 25 w(ALLCAS)q(T)27 b Fk(restricted)f(to)d(blo)r(c)n(k)g(bu\013ers,)i(and)f(with)f(the)h(ad-)100 2041 y(ditional)g(restriction)h(that)e(all)f(blo)r(c)n(ks)h(sen)n(t)g(from)f (one)g(pro)r(cess)h(to)g(another)i(ha)n(v)n(e)e(the)g(same)d(length.)100 2116 y(The)f(routine)h(is)e(called)h(b)n(y)g(all)f(mem)n(b)r(ers)e(of)j(the)h (group)h(using)e(the)g(same)e(argumen)n(ts)j(for)g Ff(group)p Fk(,)i(and)100 2191 y Ff(len)p Fk(.)100 2313 y Fg(IN)h(in)n(buf)29 b Fk(\014rst)22 b(en)n(try)h(in)e(input)h(bu\013er)h(\(c)n(hoice\).)29 b(ro)r(ot)21 b(\(in)n(teger\))100 2437 y Fg(OUT)h(outbuf)30 b Fk(\014rst)22 b(en)n(try)h(in)e(output)i(bu\013er)g(\(c)n(hoice\).)100 2560 y Fg(IN)g(len)30 b Fk(n)n(um)n(b)r(er)21 b(of)g(en)n(tries)h(sen)n(t)g (from)e(eac)n(h)i(pro)r(cess)f(to)g(eac)n(h)g(other)i(\(including)g(itself)5 b(\).)100 2684 y Fg(IN)23 b(group)32 b Fk(group)23 b(handle)221 2806 y Ff(MPI)p 318 2806 V 26 w(ALLCAST)q(C)q(\()37 b(inbuf,)f(outbu)q(f,)h (len,)e(group\))25 b Fk(is)100 2927 y Ff(MPI_GS)q(I)q(Z)q(E\()37 b(group,)g(&size)f(\);)100 3003 y(MPI_RA)q(N)q(K)q(\()g(group,)h(&rank)f(\);) 100 3078 y(for)e(\(i=0;)i(i)d(<)f(rank;)k(i++\))195 3153 y({)227 3228 y(MPI_IR)q(E)q(C)q(V)q(C\()q(r)q(e)q(c)q(v)q(_h)q(a)q(n)q(d)q(le)q(s)q ([)q(i)q(])q(,)g(outbuf)q(,)h(len,)e(tag,)g(group\))q(;)227 3304 y(outbuf)i(+=)c(len;)195 3379 y(})100 3454 y(for)h(\(i=0;)i(i)d(<)f (size;)k(i++\))195 3530 y({)227 3605 y(MPI_IS)q(E)q(N)q(D)q(C\()q(s)q(e)q(n)q (d)q(_h)q(a)q(n)q(d)q(le)q([)q(i)q(])q(,)g(inbuf,)h(len,)e(i,)f(tag,)h (group\))q(;)p eop %%Page: 11 12 bop 100 -134 a Fh(1.6.)47 b(GLOBAL)19 b(COMPUTE)f(OPERA)-5 b(TIONS)1171 b Fk(11)195 60 y Ff(})100 135 y(MPI_WA)q(I)q(T)q(AL)q(L)q(\()q (s)q(en)q(d)q(_)q(h)q(a)q(nd)q(l)q(e)q(\))q(;)100 211 y(MPI_WA)q(I)q(T)q(AL)q (L)q(\()q(r)q(ec)q(v)q(_)q(h)q(a)q(nd)q(l)q(e)q(\))q(;)100 407 y Fj(1.6)94 b(Global)29 b(Comput)o(e)d(Op)s(erations)100 544 y Fg(Reduce)100 708 y(MPI)p 244 708 21 3 v 25 w(REDUCE\()21 b(in)n(buf,)h(outbuf,)i(group,)i(ro)r(ot,)g(op\))221 831 y Fk(Com)n(bines)d(the)h(v)m(alues)f(pro)n(vided)j(in)d(the)h(input)i(bu\013er) f(of)e(eac)n(h)h(pro)r(cess)g(in)g(the)g(group,)i(using)100 906 y(the)c(op)r(eration)j Ff(op)p Fk(,)d(and)h(returns)i(the)d(com)n(bined)g (v)m(alue)f(in)h(the)g(output)j(bu\013er)f(of)e(the)g(pro)r(cess)g(with)100 982 y(rank)h Ff(root)p Fk(.)33 b(Eac)n(h)21 b(pro)r(cess)h(can)g(pro)n(vide)i (one)e(v)m(alue,)g(or)g(a)f(sequence)h(of)g(v)m(alues,)g(in)f(whic)n(h)i (case)e(the)100 1057 y(com)n(bine)e(op)r(eration)j(is)c(executed)j(p)r(oin)n (t)n(wise)f(on)g(eac)n(h)g(en)n(try)h(of)f(the)g(sequence.)27 b(F)-5 b(or)20 b(example,)e(if)h(the)100 1132 y(op)r(eration)25 b(is)d Ff(max)j Fk(and)f(input)h(bu\013ers)g(con)n(tains)f(t)n(w)n(o)g (\015oating)h(p)r(oin)n(t)f(n)n(um)n(b)r(ers,)h(then)f(outbuf\(1\))j(=)100 1208 y(global)19 b(max\(in)n(buf\(1\)\))j(and)d(outbuf\(2\))j(=)c(global)h (max\(in)n(buf\(2\)\).)30 b(All)17 b(input)j(bu\013ers)f(should)h(de\014ne) 100 1283 y(sequences)e(of)g(equal)g(length)h(of)f(en)n(tries)h(of)f(t)n(yp)r (es)g(that)i(matc)n(h)d(the)i(t)n(yp)r(e)f(of)g(the)g(op)r(erands)i(of)e Ff(op)p Fk(.)28 b(The)100 1358 y(output)g(bu\013er)e(should)h(de\014ne)f(a)e (sequence)i(of)f(the)g(same)e(length)k(of)e(en)n(tries)h(of)f(t)n(yp)r(es)g (that)i(matc)n(h)100 1433 y(the)e(t)n(yp)r(e)g(of)g(the)h(result)f(of)g Ff(op)p Fk(.)40 b(\(Note)25 b(that,)i(here)f(as)e(for)h(all)g(other)h(comm)n (unication)e(op)r(erations,)100 1509 y(the)e(t)n(yp)r(e)g(of)f(en)n(tries)h (inserted)h(in)e(a)g(message)f(dep)r(end)i(on)g(the)g(information)h(pro)n (vided)g(b)n(y)f(the)g(input)100 1584 y(bu\013er)e(descriptor,)h(and)e(not)f (on)h(the)f(declarations)i(of)e(these)h(v)m(ariables)f(in)g(the)g(calling)g (program.)28 b(The)100 1659 y(t)n(yp)r(es)19 b(of)f(the)h(v)m(ariables)f(in)g (the)h(calling)f(program)h(need)g(not)g(matc)n(h)f(the)h(t)n(yp)r(es)f (de\014ned)i(b)n(y)f(the)g(bu\013er)100 1734 y(descriptor,)32 b(but)e(in)e(suc)n(h)h(case)f(the)g(outcome)g(of)g(a)g(reduce)i(op)r(eration) g(ma)n(y)d(b)r(e)h(implemen)n(tation)100 1810 y(dep)r(enden)n(t.\))221 1886 y(The)22 b(op)r(eration)i(de\014ned)g(b)n(y)e Ff(op)h Fk(is)e(asso)r(ciativ)n(e)h(and)h(comm)n(utativ)n(e,)e(and)i(the)g(implemen)n (tation)100 1961 y(can)h(tak)n(e)h(adv)m(an)n(tage)h(of)e(asso)r(ciativit)n (y)h(and)g(comm)n(utativit)n(y)f(in)g(order)i(to)e(c)n(hange)h(order)h(of)e (ev)m(alu-)100 2036 y(ation.)41 b(The)24 b(routine)j(is)d(called)h(b)n(y)g (all)g(group)i(mem)n(b)r(ers)22 b(using)k(the)f(same)e(argumen)n(ts)j(for)g Ff(group,)100 2112 y(root)d Fk(and)f Ff(op)p Fk(.)100 2241 y Fg(IN)h(in)n(buf)29 b Fk(handle)23 b(to)e(input)h(bu\013er)100 2370 y Fg(OUT)g(outbuf)30 b Fk(handle)23 b(to)e(output)i(bu\013er)g({)d (signi\014can)n(t)k(only)d(at)g(ro)r(ot)100 2498 y Fg(IN)i(group)32 b Fk(handle)23 b(to)e(group)100 2627 y Fg(IN)i(ro)r(ot)33 b Fk(rank)22 b(of)f(ro)r(ot)h(pro)r(cess)f(\(in)n(teger\))100 2756 y Fg(IN)i(op)31 b Fk(op)r(eration)23 b(\(status\))221 2885 y(W)-5 b(e)27 b(list)f(b)r(elo)n(w)h(the)h(op)r(erations)h(whic)n(h)f (are)f(supp)r(orted)j(for)e(F)-5 b(ortran,)31 b(eac)n(h)c(with)h(the)f (corre-)100 2960 y(sp)r(onding)c(v)m(alue)d(of)i(the)f Ff(op)h Fk(parameter.)100 3089 y Fg(MPI)p 244 3089 V 25 w(IMAX)31 b Fk(in)n(teger)22 b(maxim)n(um)100 3218 y Fg(MPI)p 244 3218 V 25 w(RMAX)29 b Fk(real)21 b(maxim)n(um)100 3347 y Fg(MPI)p 244 3347 V 25 w(DMAX)30 b Fk(double)22 b(precision)g(real)f(maxim)n(um)100 3476 y Fg(MPI)p 244 3476 V 25 w(IMIN)31 b Fk(in)n(teger)23 b(minim)n(um)100 3605 y Fg(MPI)p 244 3605 V 25 w(RMIN)30 b Fk(real)21 b(minim)n(um)p eop %%Page: 12 13 bop 100 -134 a Fk(12)938 b Fh(CHAPTER)17 b(1.)48 b(COLLECTIVE)17 b(COMMUNICA)-5 b(TION)100 60 y Fg(MPI)p 244 60 21 3 v 25 w(DMIN)30 b Fk(double)23 b(precision)f(real)f(minim)n(um)100 185 y Fg(MPI)p 244 185 V 25 w(ISUM)31 b Fk(in)n(teger)22 b(sum)100 310 y Fg(MPI)p 244 310 V 25 w(RSUM)29 b Fk(real)21 b(sum)100 435 y Fg(MPI)p 244 435 V 25 w(DSUM)30 b Fk(double)22 b(precision)g(real)f(sum)100 560 y Fg(MPI)p 244 560 V 25 w(CSUM)29 b Fk(complex)20 b(sum)100 685 y Fg(MPI)p 244 685 V 25 w(DCSUM)29 b Fk(double)22 b(precision)g(complex)e (sum)100 810 y Fg(MPI)p 244 810 V 25 w(IPR)n(OD)29 b Fk(in)n(teger)22 b(pro)r(duct)100 935 y Fg(MPI)p 244 935 V 25 w(RPR)n(OD)28 b Fk(real)21 b(pro)r(duct)100 1060 y Fg(MPI)p 244 1060 V 25 w(DPR)n(OD)28 b Fk(double)22 b(precision)g(real)f(pro)r(duct)100 1185 y Fg(MPI)p 244 1185 V 25 w(CPR)n(OD)28 b Fk(complex)19 b(pro)r(duct)100 1310 y Fg(MPI)p 244 1310 V 25 w(DCPR)n(OD)27 b Fk(double)22 b(precision)h(complex)c(pro)r(duct)100 1435 y Fg(MPI)p 244 1435 V 25 w(AND)29 b Fk(logical)21 b(and)100 1560 y Fg(MPI)p 244 1560 V 25 w(IAND)30 b Fk(in)n(teger)22 b(\(bit-wise\))g(and)100 1685 y Fg(MPI)p 244 1685 V 25 w(OR)29 b Fk(logical)21 b(or)100 1810 y Fg(MPI)p 244 1810 V 25 w(IOR)30 b Fk(in)n(teger)22 b(\(bit-wise\))g(or)100 1935 y Fg(MPI)p 244 1935 V 25 w(X)n(OR)29 b Fk(logical)21 b(xor)100 2060 y Fg(MPI)p 244 2060 V 25 w(IX)n(OR)29 b Fk(in)n(teger)23 b(\(bit-wise\))f(xor) 100 2185 y Fg(MPI)p 244 2185 V 25 w(MAXLOC)29 b Fk(rank)22 b(of)f(pro)r(cess)g(with)g(maxim)n(um)c(in)n(teger)22 b(v)m(alue)100 2310 y Fg(MPI)p 244 2310 V 25 w(MAXRLOC)28 b Fk(rank)22 b(of)f(pro)r(cess)g (with)g(maxim)n(um)c(real)k(v)m(alue)100 2435 y Fg(MPI)p 244 2435 V 25 w(MAXDLOC)28 b Fk(rank)22 b(of)f(pro)r(cess)g(with)g(maxim)n(um)c (double)23 b(precision)f(real)f(v)m(alue)100 2560 y Fg(MPI)p 244 2560 V 25 w(MINLOC)29 b Fk(rank)22 b(of)f(pro)r(cess)h(with)f(minim)n(um) c(in)n(teger)22 b(v)m(alue)100 2685 y Fg(MPI)p 244 2685 V 25 w(MINRLOC)29 b Fk(rank)22 b(of)f(pro)r(cess)g(with)g(minim)n(um)c(real)k(v)m (alue)100 2810 y Fg(MPI)p 244 2810 V 25 w(MINDLOC)29 b Fk(rank)22 b(of)f(pro)r(cess)g(with)g(minim)n(um)c(double)22 b(precision)g(real)f(v)m (alue)221 2982 y Fg(MPI)p 365 2982 V 26 w(REDUCEC\()e(in)n(buf,)k(outbuf,)h (len,)f(group,)i(ro)r(ot,)h(op\))221 3105 y Fk(Is)20 b(same)f(as)h Ff(MPI)p 610 3105 20 3 v 26 w(REDUCE)q Fk(,)k(restricted)f(to)e(a)f(blo)r(c)n (k)h(bu\013er.)100 3230 y Fg(IN)i(in)n(buf)29 b Fk(\014rst)22 b(lo)r(cation)g(in)f(input)h(bu\013er)100 3355 y Fg(OUT)g(outbuf)30 b Fk(\014rst)22 b(lo)r(cation)g(in)f(output)i(bu\013er)g({)e(signi\014can)n (t)i(only)e(at)g(ro)r(ot)100 3480 y Fg(IN)i(len)30 b Fk(n)n(um)n(b)r(er)21 b(of)g(en)n(tries)h(in)f(input)i(and)f(output)h(bu\013er)g(\(in)n(teger\))100 3605 y Fg(IN)g(group)32 b Fk(handle)23 b(to)e(group)p eop %%Page: 13 14 bop 100 -134 a Fh(1.6.)47 b(GLOBAL)19 b(COMPUTE)f(OPERA)-5 b(TIONS)1171 b Fk(13)100 60 y Fg(IN)23 b(ro)r(ot)33 b Fk(rank)22 b(of)f(ro)r(ot)h(pro)r(cess)f(\(in)n(teger\))100 183 y Fg(IN)i(op)31 b Fk(op)r(eration)23 b(\(status\))221 350 y Fg(MPI)p 365 350 21 3 v 26 w(USER)p 583 350 V 23 w(REDUCE\()e(in)n(buf,)h(outbuf,)i(group,)i (ro)r(ot,)g(function\))221 472 y Fk(Same)e(as)g(the)h(reduce)g(op)r(eration)i (function)g(ab)r(o)n(v)n(e)f(except)f(that)h(a)e(user)h(supplied)h(function)h (is)100 548 y(used.)g Ff(functi)q(o)q(n)21 b Fk(is)c(an)g(asso)r(ciativ)n(e)h (and)g(comm)n(utativ)n(e)f(function)j(with)d(t)n(w)n(o)h(argumen)n(ts.)28 b(The)17 b(t)n(yp)r(es)100 623 y(of)j(the)h(t)n(w)n(o)f(argumen)n(ts)i(and)f (of)f(the)g(returned)k(v)m(alue)19 b(of)i(the)f(function,)j(and)e(the)f(t)n (yp)r(es)h(of)f(all)f(en)n(tries)100 698 y(in)25 b(the)h(input)h(and)f (output)i(bu\013ers)f(all)d(agree.)41 b(The)25 b(output)j(bu\013er)f(has)f (the)f(same)f(length)i(as)f(the)100 773 y(input)e(bu\013er.)100 893 y Fg(IN)g(in)n(buf)29 b Fk(handle)23 b(to)e(input)h(bu\013er)100 1016 y Fg(OUT)g(outbuf)30 b Fk(handle)23 b(to)e(output)i(bu\013er)g({)d (signi\014can)n(t)k(only)d(at)g(ro)r(ot)100 1139 y Fg(IN)i(group)32 b Fk(handle)23 b(to)e(group)100 1262 y Fg(IN)i(ro)r(ot)33 b Fk(rank)22 b(of)f(ro)r(ot)h(pro)r(cess)f(\(in)n(teger\))100 1385 y Fg(IN)i(function)30 b Fk(user)22 b(pro)n(vided)h(function)221 1552 y Fg(MPI)p 365 1552 V 26 w(USER)p 583 1552 V 23 w(REDUCEC\()d(in)n(buf,) i(outbuf,)i(len,)f(group,)i(ro)r(ot,)i(function\))221 1674 y Fk(Is)20 b(same)f(as)h Ff(MPI)p 610 1674 20 3 v 636 1674 V 49 w(USER)p 783 1674 V 26 w(REDUCE)q Fk(,)k(restricted)f(to)e(a)f(blo)r(c)n (k)h(bu\013er.)100 1794 y Fg(IN)i(in)n(buf)29 b Fk(\014rst)22 b(lo)r(cation)g(in)f(input)h(bu\013er)100 1917 y Fg(OUT)g(outbuf)30 b Fk(\014rst)22 b(lo)r(cation)g(in)f(output)i(bu\013er)g({)e(signi\014can)n (t)i(only)e(at)g(ro)r(ot)100 2039 y Fg(IN)i(len)30 b Fk(n)n(um)n(b)r(er)21 b(of)g(en)n(tries)h(in)f(input)i(and)f(output)h(bu\013er)g(\(in)n(teger\))100 2162 y Fg(IN)g(group)32 b Fk(handle)23 b(to)e(group)100 2285 y Fg(IN)i(ro)r(ot)33 b Fk(rank)22 b(of)f(ro)r(ot)h(pro)r(cess)f(\(in)n (teger\))100 2408 y Fg(IN)i(op)31 b Fk(op)r(eration)23 b(\(status\))221 2629 y Fe(Discussion:)221 2695 y Fd(Do)e(w)n(e)h(also)d(w)n(an)n(t)i(a)g(v)n (ersion)f(of)g(reduce)i(that)f(broadcasts)f(the)h(result)f(to)h(all)e(pro)r (cesses)i(in)f(the)h(group?)100 2762 y(\(This)12 b(can)h(b)r(e)f(ac)n(hiev)n (ed)i(b)n(y)f(a)g(reduce)g(follo)n(w)n(ed)f(b)n(y)h(a)f(broadcast,)h(but)f(a) h(com)n(bined)e(function)h(ma)n(y)g(b)r(e)h(somewhat)100 2828 y(more)18 b(e\016cien)n(t.\))25 b(These)19 b(w)n(ould)f(b)r(e)g(resp)r(ectiv) n(ely:)221 2942 y Fe(MPI)p 352 2942 V 24 w(GOP\()23 b(in)n(buf,)c(outbuf,)g (group,)f(op\))221 3103 y(MPI)p 352 3103 V 24 w(GOPC\()23 b(in)n(buf,)c (outbuf,)g(len,)i(group,)d(op\))221 3264 y(MPI)p 352 3264 V 24 w(USER)p 551 3264 V 22 w(GOP\()k(in)n(buf,)e(outbuf,)f(group,)f (function\))221 3425 y(MPI)p 352 3425 V 24 w(USER)p 551 3425 V 22 w(GOPC\()k(in)n(buf,)e(outbuf,)f(len,)i(group,)d(function\))221 3538 y Fd(Do)g(w)n(e)h(w)n(an)n(t)g(a)f(user)g(pro)n(vided)f Fb(function)g Fd(\(t)n(w)n(o)j(IN)e(parameters,)f(one)h(OUT)g(v)m(alue\),)g (or)f(a)h(user)g(pro)n(vided)100 3605 y(pro)r(cedure)i(that)h(o)n(v)n (erwrites)h(the)f(second)g(input)e(\(ie.)31 b(one)21 b(IN)g(param,)d(one)i (INOUT)i(param,)c(the)k(equiv)m(alen)n(t)p eop %%Page: 14 15 bop 100 -134 a Fk(14)938 b Fh(CHAPTER)17 b(1.)48 b(COLLECTIVE)17 b(COMMUNICA)-5 b(TION)100 60 y Fd(of)24 b(C)h Fa(a)k(op=)g(b)c Fd(t)n(yp)r(e)g(assignmen)n(t\)?)42 b(The)25 b(second)f(c)n(hoice)i(ma)n(y)e (allo)n(w)f(a)i(more)e(e\016cien)n(t)j(implemen)n(tation,)100 127 y(without)18 b(c)n(hanging)f(the)i(seman)n(tics)g(of)f(the)h(MPI)f(call.) 100 396 y Fg(Scan)100 557 y(MPI)p 244 557 21 3 v 25 w(SCAN\()k(in)n(buf,)g (outbuf,)i(group,)i(op)e(\))221 680 y Fk(MPI)p 345 680 19 3 v 21 w(SCAN)16 b(is)g(used)i(to)g(p)r(erform)g(a)f(parallel)h(pre\014x)h (with)f(resp)r(ect)g(to)f(an)h(asso)r(ciativ)n(e)g(reduction)100 755 y(op)r(eration)k(on)e(data)h(distributed)i(across)d(the)g(group.)29 b(The)19 b(op)r(eration)j(returns)h(in)c(the)h(output)j(bu\013er)100 830 y(of)d(the)f(pro)r(cess)h(with)g(rank)g Ff(i)g Fk(the)f(reduction)k(of)c (the)h(v)m(alues)f(in)g(the)h(input)h(bu\013ers)g(of)f(pro)r(cesses)f(with) 100 906 y(ranks)k Ff(0,...,)q(i)q Fk(.)34 b(The)22 b(t)n(yp)r(e)h(of)f(op)r (erations)i(supp)r(orted)i(and)d(their)g(seman)n(tic,)f(and)h(the)f (constrain)n(ts)100 981 y(on)f(input)i(and)f(output)h(bu\013ers)g(are)e(as)f (for)i Ff(MPI)p 1225 981 20 3 v 25 w(REDUC)q(E)p Fk(.)100 1099 y Fg(IN)h(in)n(buf)29 b Fk(handle)23 b(to)e(input)h(bu\013er)100 1222 y Fg(OUT)g(outbuf)30 b Fk(handle)23 b(to)e(output)i(bu\013er)100 1344 y Fg(IN)g(group)32 b Fk(handle)23 b(to)e(group)100 1467 y Fg(IN)i(op)31 b Fk(op)r(eration)23 b(\(status\))221 1633 y Fg(MPI)p 365 1633 21 3 v 26 w(SCANC\()h(in)n(buf,)j(outbuf,)h(len,)f (group,)j(op)d(\))c Fk(Same)g(as)g Ff(MPI)p 1984 1633 20 3 v 26 w(SCAN)p Fk(,)j(restricted)g(to)100 1755 y(blo)r(c)n(k)21 b(bu\013ers.)100 1888 y Fg(IN)i(in)n(buf)29 b Fk(\014rst)22 b(input)h(bu\013er)g(elemen)n(t)d(\(c)n(hoice\))100 2010 y Fg(OUT)i(outbuf)30 b Fk(\014rst)22 b(output)i(bu\013er)f(elemen)n(t)d(\(c)n (hoice\))100 2133 y Fg(IN)j(len)30 b Fk(n)n(um)n(b)r(er)21 b(of)g(en)n(tries)h(in)f(input)i(and)f(output)h(bu\013er)g(\(in)n(teger\))100 2255 y Fg(IN)g(group)32 b Fk(handle)23 b(to)e(group)100 2378 y Fg(IN)i(op)31 b Fk(op)r(eration)23 b(\(status\))221 2557 y Fg(MPI)p 365 2557 21 3 v 26 w(USER)p 583 2557 V 23 w(SCAN\()f(in)n(buf,)g (outbuf,)i(group,)i(function)d(\))221 2680 y Fk(Same)16 b(as)h(the)h(scan)f (op)r(eration)j(function)g(ab)r(o)n(v)n(e)e(except)g(that)g(a)f(user)h (supplied)h(function)h(is)c(used.)100 2755 y Ff(functi)q(o)q(n)29 b Fk(is)24 b(an)h(asso)r(ciativ)n(e)g(and)h(comm)n(utativ)n(e)e(function)j (with)e(t)n(w)n(o)h(argumen)n(ts.)40 b(The)25 b(t)n(yp)r(es)g(of)100 2830 y(the)c(t)n(w)n(o)h(argumen)n(ts)g(and)g(of)f(the)g(returned)j(v)m (alues)d(all)f(agree.)100 2949 y Fg(IN)j(in)n(buf)29 b Fk(handle)23 b(to)e(input)h(bu\013er)100 3072 y Fg(OUT)g(outbuf)30 b Fk(handle)23 b(to)e(output)i(bu\013er)100 3194 y Fg(IN)g(group)32 b Fk(handle)23 b(to)e(group)100 3316 y Fg(IN)i(function)30 b Fk(user)22 b(pro)n(vided)h (function)221 3482 y Fg(MPI)p 365 3482 V 26 w(USER)p 583 3482 V 23 w(SCANC\()e(in)n(buf,)h(outbuf,)i(len,)f(group,)j(function\))221 3605 y Fk(Is)20 b(same)f(as)h Ff(MPI)p 610 3605 20 3 v 26 w(USER)p 760 3605 V 26 w(SCAN)p Fk(,)j(restricted)g(to)e(a)g(blo)r(c)n(k)g(bu\013er.)p eop %%Page: 15 16 bop 100 -134 a Fh(1.7.)47 b(CORRECTNESS)1733 b Fk(15)100 60 y Fg(IN)23 b(in)n(buf)29 b Fk(\014rst)22 b(lo)r(cation)g(in)f(input)h (bu\013er)100 180 y Fg(OUT)g(outbuf)30 b Fk(\014rst)22 b(lo)r(cation)g(in)f (output)i(bu\013er)100 300 y Fg(IN)g(len)30 b Fk(n)n(um)n(b)r(er)21 b(of)g(en)n(tries)h(in)f(input)i(and)f(output)h(bu\013er)g(\(in)n(teger\))100 420 y Fg(IN)g(group)32 b Fk(handle)23 b(to)e(group)100 540 y Fg(IN)i(function)30 b Fk(user)22 b(pro)n(vided)h(function)221 754 y Fe(Discussion:)221 830 y Fd(Do)e(w)n(e)h(w)n(an)n(t)f(scan)g(op)r (erations)e(executed)k(b)n(y)f(segmen)n(ts?)31 b(\(The)21 b(HPF)g (de\014nition)e(of)h(pre\014x)h(and)f(su\016x)100 905 y(op)r(eration)f(migh)n (t)i(b)r(e)g(handy)f({)h(in)f(addition)g(to)h(the)h(scanned)f(v)n(ector)i(of) e(v)m(alues)g(there)h(is)f(a)f(mask)h(that)g(tells)100 980 y(where)e(segmen)n(ts)g(start)g(and)e(end.\))100 1279 y Fj(1.7)94 b(Correctness)100 1525 y Fe(Discussion:)46 b Fd(This)18 b(is)f(still)g(v)n (ery)j(preliminary)221 1710 y Fk(The)25 b(seman)n(tics)g(of)h(the)g (collectiv)n(e)f(comm)n(unication)h(op)r(erations)h(can)f(b)r(e)f(deriv)n(ed) i(from)e(their)100 1785 y(op)r(erational)i(de\014nition)g(in)d(terms)f(of)h (p)r(oin)n(t-to-p)r(oin)o(t)k(comm)n(unication.)38 b(It)24 b(is)f(assumed)h(that)h(mes-)100 1861 y(sages)c(p)r(ertaining)j(to)d(one)h (op)r(eration)h(cannot)h(b)r(e)c(confused)k(with)d(messages)e(p)r(ertaining) 24 b(to)e(another)100 1936 y(op)r(eration.)29 b(Also)18 b(messages)f(p)r (ertaining)22 b(to)d(t)n(w)n(o)h(distinct)g(o)r(ccurrences)h(of)e(the)g(same) f(op)r(eration)j(can-)100 2011 y(not)h(b)r(e)e(confused,)j(if)e(the)g(t)n(w)n (o)h(o)r(ccurrences)h(ha)n(v)n(e)e(distinct)i(parameters.)28 b(The)21 b(relev)m(an)n(t)h(parameters)100 2086 y(for)f(this)g(purp)r(ose)h (are)e Ff(group)p Fk(,)k Ff(root)f Fk(and)e Ff(op)p Fk(.)28 b(messages)18 b(p)r(ertaining)23 b(to)e(another)h(o)r(ccurrence)g(of)f(the) 100 2162 y(same)h(op)r(eration,)28 b(with)d(di\013eren)n(t)h(parameters.)39 b(The)25 b(implemen)n(ter)e(can,)i(of)g(course,)h(use)e(another,)100 2237 y(more)c(e\016cien)n(t)h(implemen)n(tation,)f(as)g(long)i(as)e(it)h(has) g(the)g(same)e(e\013ect.)221 2414 y Fe(Discussion:)221 2480 y Fd(This)c(statemen)n(t)i(do)r(es)d(not)i(y)n(et)h(apply)d(to)i(the)g (curren)n(t,)g(incomplete)f(and)g(somewhat)f(careless)i(de\014nitions)100 2546 y(I)i(pro)n(vided)g(in)g(this)g(draft.)221 2613 y(The)j(de\014nition)e (ab)r(o)n(v)n(e)i(means)e(that)i(messages)f(p)r(ertaining)e(to)j(a)f (collectiv)n(e)i(comm)n(unication)c(carry)i(in-)100 2679 y(formation)c(iden)n (tifying)h(the)i(op)r(eration)d(itself,)h(and)h(the)h(v)m(alues)f(of)f(the)j Fa(group)e Fd(and,)f(where)i(relev)m(an)n(t,)f Fa(root)h Fd(or)100 2746 y Fa(op)g Fd(parameters.)k(Is)18 b(this)g(acceptable?)221 2931 y Fk(A)i(few)g(examples:)100 3055 y Ff(MPI_BC)q(A)q(S)q(T\()q(b)q(u)q(f) q(,)36 b(len,)f(group,)i(0\);)100 3131 y(MPI_BC)q(A)q(S)q(T\()q(b)q(u)q(f)q (,)f(len,)f(group,)i(1\);)221 3255 y Fk(Tw)n(o)18 b(consecutiv)n(e)i (broadcasts,)i(in)d(the)g(same)d(group,)21 b(with)e(the)h(same)c(tag,)k(but)g (di\013eren)n(t)h(ro)r(ots.)100 3330 y(Since)h(the)h(op)r(erations)h(are)e (distinguishable,)k(messages)20 b(from)h(one)h(broadcast)j(cannot)f(b)r(e)d (confused)100 3405 y(with)g(messages)e(from)h(the)i(other)g(broadcast;)h(the) f(program)g(is)e(safe)g(and)i(will)e(execute)h(as)f(exp)r(ected.)100 3530 y Ff(MPI_BC)q(A)q(S)q(T\()q(b)q(u)q(f)q(,)36 b(len,)f(group,)i(0\);)100 3605 y(MPI_BC)q(A)q(S)q(T\()q(b)q(u)q(f)q(,)f(len,)f(group,)i(0\);)p eop %%Page: 16 17 bop 100 -134 a Fk(16)938 b Fh(CHAPTER)17 b(1.)48 b(COLLECTIVE)17 b(COMMUNICA)-5 b(TION)221 60 y Fk(Tw)n(o)26 b(consecutiv)n(e)h(broadcasts,)j (in)c(the)h(same)d(group,)29 b(with)d(the)h(same)d(tag)j(and)g(ro)r(ot.)44 b(Since)100 135 y(p)r(oin)n(t-to-p)r(oin)o(t)22 b(comm)n(unication)c(preserv) n(es)i(the)g(order)g(of)e(messages)f(here,)j(to)r(o,)f(messages)d(from)i(one) 100 211 y(broadcast)j(will)c(not)j(b)r(e)e(confused)j(with)d(messages)f(from) h(the)h(other)h(broadcast;)i(the)d(program)h(is)d(safe)100 286 y(and)22 b(will)d(execute)i(as)g(in)n(tended.)100 428 y Ff(MPI_RA)q(N)q(K)q(\(&)q(r)q(a)q(n)q(k,)37 b(group\))100 503 y(if)c(\(rank)q(==)q(0)q(\))164 578 y({)195 653 y(MPI_B)q(CA)q(S)q(T)q(C)q (\(b)q(u)q(f)q(,)k(len,)e(group,)h(0\);)195 729 y(MPI_S)q(EN)q(D)q(C)q(\()q (bu)q(f)q(,)h(len,)e(2,)e(tag,)i(group\))q(;)164 804 y(})100 879 y(elseif)i(\(rank=)q(=1)q(\))164 955 y({)195 1030 y(MPI_R)q(EC)q(V)q(C)q (\()q(bu)q(f)q(,)g(len,)e(MPI_DO)q(N)q(TC)q(A)q(R)q(E)q(,)h(tag,)f(group\))q (;)195 1105 y(MPI_B)q(CA)q(S)q(T)q(C)q(\(b)q(u)q(f)q(,)i(len,)e(group,)h (0\);)195 1180 y(MPI_R)q(EC)q(V)q(C)q(\()q(bu)q(f)q(,)h(len,)e(MPI_DO)q(N)q (TC)q(A)q(R)q(E)q(,)h(tag,)f(group\))q(;)164 1256 y(})100 1331 y(else)164 1406 y({)195 1481 y(MPI_S)q(EN)q(D)q(C)q(\()q(bu)q(f)q(,)i(len,)e (2,)e(tag,)i(group\))q(;)195 1557 y(MPI_B)q(CA)q(S)q(T)q(C)q(\(b)q(u)q(f)q(,) i(len,)e(group,)h(0\);)164 1632 y(})221 1774 y Fk(Pro)r(cess)23 b(zero)g(executes)h(a)f(broadcast)k(follo)n(w)n(ed)d(b)n(y)g(a)f(send)h(to)g (pro)r(cess)g(one;)h(pro)r(cess)f(t)n(w)n(o)g(ex-)100 1849 y(ecutes)f(a)f(send)h(to)g(pro)r(cess)g(one,)g(follo)n(w)n(ed)g(b)n(y)g(a)f (broadcast;)k(and)e(pro)r(cess)f(one)g(executes)f(a)h(receiv)n(e,)100 1924 y(a)29 b(broadcast)i(and)f(a)f(receiv)n(e.)52 b(A)27 b(p)r(ossible)j (outcome)e(is)g(for)i(the)g(op)r(erations)h(to)e(b)r(e)g(matc)n(hed)g(as)100 2000 y(illustrated)23 b(b)n(y)e(the)h(diagram)e(b)r(elo)n(w.)227 2292 y Ff(0)733 b(1)701 b(2)609 2442 y(/)33 b(-)f(>)65 b(receiv)q(e)386 b(/)33 b(-)g(send)545 2518 y(/)797 b(/)100 2593 y(broadc)q(a)q(s)q(t)100 b(/)287 b(broadc)q(a)q(s)q(t)227 b(/)96 b(broadc)q(a)q(s)q(t)450 2668 y(/)764 b(/)164 2743 y(send)98 b(-)415 b(receiv)q(e)68 b(<)33 b(-)221 3036 y Fk(The)20 b(reason)g(is)f(that)i(broadcast)i(is)18 b(not)j(a)e(sync)n(hronous)k(op)r(eration;)g(the)d(call)f(at)h(a)f(pro)r (cess)h(ma)n(y)100 3111 y(return)27 b(b)r(efore)e(the)g(other)h(pro)r(cesses) e(ha)n(v)n(e)h(en)n(tered)i(the)d(broadcast.)40 b(Th)n(us,)26 b(the)f(message)d(sen)n(t)j(b)n(y)100 3186 y(pro)r(cess)e(zero)g(can)g(arriv) n(e)h(to)f(pro)r(cess)g(one)g(b)r(efore)h(the)f(message)e(sen)n(t)j(b)n(y)f (pro)r(cess)g(t)n(w)n(o,)g(and)h(b)r(efore)100 3261 y(the)d(call)f(to)i (broadcast)h(on)e(pro)r(cess)h(one.)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-comm@CS.UTK.EDU Wed Apr 21 11:56:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01628; Wed, 21 Apr 93 11:56:15 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA29787; Wed, 21 Apr 93 11:53:34 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 21 Apr 1993 11:53:27 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from almaden.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA29755; Wed, 21 Apr 93 11:52:26 -0400 Message-Id: <9304211552.AA29755@CS.UTK.EDU> Received: from almaden.ibm.com by almaden.ibm.com (IBM VM SMTP V2R2) with BSMTP id 2750; Wed, 21 Apr 93 08:52:59 PDT Date: Wed, 21 Apr 93 08:44:31 PDT From: "Ching-Tien (Howard) Ho" To: mpi-comm@cs.utk.edu Subject: the CCL paper by Bala et al. Hi, This is the postscript file of the CCL paper I brought to the last MPI meeting (same version), in case you missed a copy then. It has appeared as an IBM Research Report RJ 9284, April 1993. Let me know if you prefer to have a Latex file of it. All comments are welcome. Regards, -- Howard %!PS-Adobe-2.0 %%Creator: dvips 5.47 (RS/6000 1.0) Copyright 1986-91 Radical Eye Software %%Title: CCL.dvi %%Pages: 24 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: tex.pro /TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{ ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image} imagemask restore}B /D{/cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval(Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{ moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{ S p tail}B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w }B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{clear SS restore}B end %%EndProcSet %%BeginProcSet: special.pro TeXDict begin /SDict 200 dict N SDict begin /@SpecialDefaults{/hs 612 N /vs 792 N /ho 0 N /vo 0 N /hsc 1 N /vsc 1 N /ang 0 N /CLIP false N /BBcalc false N /p 3 def}B /@scaleunit 100 N /@hscale{@scaleunit div /hsc X}B /@vscale{ @scaleunit div /vsc X}B /@hsize{/hs X /CLIP true N}B /@vsize{/vs X /CLIP true N}B /@hoffset{/ho X}B /@voffset{/vo X}B /@angle{/ang X}B /@rwi{10 div /rwi X} B /@llx{/llx X}B /@lly{/lly X}B /@urx{/urx X}B /@ury{/ury X /BBcalc true N}B /magscale true def end /@MacSetUp{userdict /md known{userdict /md get type /dicttype eq{md begin /letter{}N /note{}N /legal{}N /od{txpose 1 0 mtx defaultmatrix dtransform S atan/pa X newpath clippath mark{transform{ itransform moveto}}{transform{itransform lineto}}{6 -2 roll transform 6 -2 roll transform 6 -2 roll transform{itransform 6 2 roll itransform 6 2 roll itransform 6 2 roll curveto}}{{closepath}}pathforall newpath counttomark array astore /gc xdf pop ct 39 0 put 10 fz 0 fs 2 F/|______Courier fnt invertflag{ PaintBlack}if}N /txpose{pxs pys scale ppr aload pop por{noflips{pop S neg S TR pop 1 -1 scale}if xflip yflip and{pop S neg S TR 180 rotate 1 -1 scale ppr 3 get ppr 1 get neg sub neg ppr 2 get ppr 0 get neg sub neg TR}if xflip yflip not and{pop S neg S TR pop 180 rotate ppr 3 get ppr 1 get neg sub neg 0 TR}if yflip xflip not and{ppr 1 get neg ppr 0 get neg TR}if}{noflips{TR pop pop 270 rotate 1 -1 scale}if xflip yflip and{TR pop pop 90 rotate 1 -1 scale ppr 3 get ppr 1 get neg sub neg ppr 2 get ppr 0 get neg sub neg TR}if xflip yflip not and{TR pop pop 90 rotate ppr 3 get ppr 1 get neg sub neg 0 TR}if yflip xflip not and{TR pop pop 270 rotate ppr 2 get ppr 0 get neg sub neg 0 S TR}if} ifelse scaleby96{ppr aload pop 4 -1 roll add 2 div 3 1 roll add 2 div 2 copy TR .96 dup scale neg S neg S TR}if}N /cp{pop pop showpage pm restore}N end}if} if}N /normalscale{Resolution 72 div VResolution 72 div neg scale magscale{ DVImag dup scale}if}N /psfts{S 65536 div N}N /startTexFig{/psf$SavedState save N userdict maxlength dict begin /magscale false def normalscale currentpoint TR /psf$ury psfts /psf$urx psfts /psf$lly psfts /psf$llx psfts /psf$y psfts /psf$x psfts currentpoint /psf$cy X /psf$cx X /psf$sx psf$x psf$urx psf$llx sub div N /psf$sy psf$y psf$ury psf$lly sub div N psf$sx psf$sy scale psf$cx psf$sx div psf$llx sub psf$cy psf$sy div psf$ury sub TR /showpage{}N /erasepage{}N /copypage{}N /p 3 def @MacSetUp}N /doclip{psf$llx psf$lly psf$urx psf$ury currentpoint 6 2 roll newpath 4 copy 4 2 roll moveto 6 -1 roll S lineto S lineto S lineto closepath clip newpath moveto}N /endTexFig{end psf$SavedState restore}N /@beginspecial{SDict begin /SpecialSave save N gsave normalscale currentpoint TR @SpecialDefaults}N /@setspecial{CLIP{newpath 0 0 moveto hs 0 rlineto 0 vs rlineto hs neg 0 rlineto closepath clip}if ho vo TR hsc vsc scale ang rotate BBcalc{rwi urx llx sub div dup scale llx neg lly neg TR}if /showpage{}N /erasepage{}N /copypage{}N newpath}N /@endspecial{grestore clear SpecialSave restore end}N /@defspecial{SDict begin}N /@fedspecial{end}B /li{lineto}B /rl{rlineto}B /rc{rcurveto}B /np{/SaveX currentpoint /SaveY X N 1 setlinecap newpath}N /st{stroke SaveX SaveY moveto}N /fil{fill SaveX SaveY moveto}N /ellipse{/endangle X /startangle X /yrad X /xrad X /savematrix matrix currentmatrix N TR xrad yrad scale 0 0 1 startangle endangle arc savematrix setmatrix}N end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 1 108 df<3C000C000C00180018001800187031 903230340038007F00618061906190C1A0C0C00C117E9010>107 D E /Fb 6 116 df<07C00C20107020706000C000C000C00080008000C010C02060C03F000C0E7E8D0F> 99 D<00180038001000000000000000000000000001C0022004300430086000600060006000C0 00C000C000C001800180018001806300E300C60078000D1D80960E>106 D<1F0006000600060006000C000C000C000C00181C1866188E190C32003C003F00318060C060C4 60C460C8C0C8C0700F177E9612>I<383C0044C6004702004602008E06000C06000C06000C0C00 180C00180C40181840181880300880300F00120E7F8D15>110 D<38F04518463846308C000C00 0C000C001800180018001800300030000D0E7F8D10>114 D<07C00C201870187038001E000FC0 03E000606060E060C0C0C1803F000C0E7E8D10>I E /Fc 18 122 df<03CC0E2E181C381C301C 701CE038E038E038E038C072C072C07260F261341E180F107C8F14>97 D<7E000E000E000E001C 001C001C001C00380038003BC03C307830701870187018E038E038E038E038C070C060C0E060C0 63801E000D1A7C9912>I<01F006080C181838301070006000E000E000E000E000E008E0106020 30C01F000D107C8F12>I<001F80000380000380000380000700000700000700000700000E0000 0E0003CE000E2E00181C00381C00301C00701C00E03800E03800E03800E03800C07200C07200C0 720060F2006134001E1800111A7C9914>I<01E006181C08380870087010FFE0E000E000E000E0 00E0086010602030C01F000D107C8F12>I<00F300038B800607000E07000C07001C0700380E00 380E00380E00380E00301C00301C00301C00183C0018780007B800003800003800007000607000 E0E000C1C0007F000011177E8F12>103 D<1F8000038000038000038000070000070000070000 0700000E00000E00000E7C000F86001E07001E07001C07001C0700380E00380E00380E00381C00 701C80701C80703880703900E01900600E00111A7E9914>I<030706000000000000384C4E8E9C 9C1C3838707272E2E4643808197C980C>I<1F8003800380038007000700070007000E000E000E 0E0E131C271C431C801F003C003F8039C038E070E270E270E270E4E0646038101A7E9912>107 D<3F0707070E0E0E0E1C1C1C1C3838383870707070E4E4E4E46830081A7D990A>I<307C005986 009E07009E07009C07009C0700380E00380E00380E00381C00701C80701C80703880703900E019 00600E0011107C8F16>110 D<01F006180C0C180E300E700E600EE00EE00EE00CE01CE018E030 606030C01F000F107C8F14>I<030F000590C009E0C009C06009C06009C0600380E00380E00380 E00380E00701C00701800703800703000E8E000E78000E00000E00001C00001C00001C00001C00 00FF00001317808F14>I<30F059189E389C189C009C0038003800380038007000700070007000 E00060000D107C8F10>114 D<03E004300830187018601C001F801FC00FE000E00060E060E060 80C041803E000C107D8F10>I<06000E000E000E000E001C001C00FFC01C003800380038003800 7000700070007000E100E100E100E200640038000A177C960D>I<38064C074E0E8E0E9C0E9C0E 1C1C381C381C381C7039703970393079389A0F0C10107C8F15>I<38064C074E0E8E0E9C0E9C0E 1C1C381C381C381C703870387038307838F00F700070006060E0E1C0C18047003C0010177C8F13 >121 D E /Fd 19 120 df<00FC000182000703000607000E02000E00000E00000E00000E0000 0E0000FFFF000E07000E07000E07000E07000E07000E07000E07000E07000E07000E07000E0700 0E07000E07000E07007F0FE0131A809915>12 D<60F0F06004047D830B>46 D<0FC21836200E6006C006C002C002C002E00070007E003FE01FF807FC003E000E000700038003 80038003C002C006E004D81887E0101A7E9915>83 D<3F8070C070E020700070007007F01C7030 707070E070E071E071E0F171FB1E3C10107E8F13>97 D<07F80C1C381C30087000E000E000E000 E000E000E0007000300438080C1807E00E107F8F11>99 D<007E00000E00000E00000E00000E00 000E00000E00000E00000E00000E0003CE000C3E00380E00300E00700E00E00E00E00E00E00E00 E00E00E00E00E00E00600E00700E00381E001C2E0007CFC0121A7F9915>I<07C01C3030187018 600CE00CFFFCE000E000E000E0006000300438080C1807E00E107F8F11>I<01F0031807380E10 0E000E000E000E000E000E00FFC00E000E000E000E000E000E000E000E000E000E000E000E000E 000E007FE00D1A80990C>I104 D<18003C003C00180000000000000000000000 0000FC001C001C001C001C001C001C001C001C001C001C001C001C001C001C00FF80091A80990A >I109 DI<07E01C38300C700E6006E007E007E007E007E007E007 6006700E381C1C3807E010107F8F13>I114 D<1F2060E04020C020C020F0007F003FC01FE000F0 80708030C030C020F0408F800C107F8F0F>I<0400040004000C000C001C003C00FFC01C001C00 1C001C001C001C001C001C001C201C201C201C201C200E4003800B177F960F>IIII E /Fe 2 51 df<0C003C00CC000C000C000C000C 000C000C000C000C000C000C000C000C00FF8009107E8F0F>49 D<1F00618040C08060C0600060 006000C00180030006000C00102020207FC0FFC00B107F8F0F>I E /Ff 4 52 df<003000003000003000003000003000003000003000003000003000003000003000FFFF FCFFFFFC0030000030000030000030000030000030000030000030000030000030000030001618 7E931B>43 D<03000700FF00070007000700070007000700070007000700070007000700070007 000700070007007FF00C157E9412>49 D<0F8030E040708030C038E03840380038007000700060 00C00180030006000C08080810183FF07FF0FFF00D157E9412>I<0FE030306018701C701C001C 00180038006007E000300018000C000E000EE00EE00EC00C401830300FE00F157F9412>I E /Fg 8 83 df<07F0001FFE00383F007C1F80FE0FC0FE0FC0FE0FE0FE07E07C07E03807E0000F E0000FC0000FC0001F80001F00003E0000780000F00000E00001C0000380600700600E00601C00 E01FFFC03FFFC07FFFC0FFFFC0FFFFC0131D7D9C1A>50 D68 D70 D<0007FC0200003FFF0E0000FE03DE0003F000FE0007E000 3E000FC0001E001F80001E003F00000E003F00000E007F000006007E000006007E00000600FE00 000000FE00000000FE00000000FE00000000FE00000000FE003FFFE0FE003FFFE07E00007E007E 00007E007F00007E003F00007E003F00007E001F80007E000FC0007E0007E0007E0003F000FE00 00FE01FE00003FFF8E000007FC0600231F7D9E29>I73 D77 D<001FF80000FFFF0001F81F8007E007E00FC003F01F8001F81F00 00F83F0000FC7F0000FE7E00007E7E00007EFE00007FFE00007FFE00007FFE00007FFE00007FFE 00007FFE00007FFE00007FFE00007F7E00007E7F0000FE7F0000FE3F0000FC3F8001FC1F8001F8 0FC003F007E007E001F81F8000FFFF00001FF800201F7D9E27>79 D82 D E /Fh 61 122 df<001000000800000400000600000200000300000300000300000100000180 000180000180000180000180000180000180000380000380000380000300000300000300000700 000700000600000600000E00000C00000C00001C0000180000380000300000700000600000E000 00C0000180000100000300000600000C0000180000300000600000800000112E80A113>41 D<1C3C3C3C3C040408081020204080060E7D840E>44 D<7FF0FFE07FE00C037D8A10>I<70F8F8 F0E005057B840E>I<00000040000000C000000180000001800000030000000300000006000000 060000000C0000001800000018000000300000003000000060000000C0000000C0000001800000 018000000300000003000000060000000C0000000C000000180000001800000030000000300000 0060000000C0000000C0000001800000018000000300000003000000060000000C0000000C0000 001800000018000000300000003000000060000000C0000000C0000000800000001A2D7FA117> I<000F800030E000E07001C0700380300380380700380F00780F00780E00781E00781E00703C00 F03C00F03C00F03C00F07801E07801E07801E07801C07003C0F003C0F00380F00780F007007007 00700E00701C003038001870000FC000151F7C9D17>I<000200020006000E003C00DC031C001C 0038003800380038007000700070007000E000E000E000E001C001C001C001C003800380038003 800780FFF80F1E7B9D17>I<001F000061800080E00100E00200700220700420700410700820F0 0820F00820F00840E00881E00703C0000380000700000C00001800006000008000030000040000 0800401000401000802001807E030047FF0041FE0080FC00807800141F7C9D17>I<001F800060 E00080700100300200380420380420380410380420700460700380600000E00001C000030000FE 00001C00000600000700000780000780000780300780780780780780F00F00800F00401E00401C 0040380020E0001F8000151F7C9D17>I<0000600000E00000E00000E00001C00001C00001C000 0380000380000300000700000700000600000E00000C0000180000180000300000300000630000 C700008700010700030700060E00040E00080E003F8E00607C00801FC0001C00001C0000380000 380000380000380000700000700000600013277E9D17>I<00C06000FFC001FF8001FE00010000 010000020000020000020000020000040000047800058C000606000C0700080700000780000780 000780000780000F00700F00F00F00F00E00E01E00801C0080380080300040600061C0001F0000 131F7B9D17>I<08E0100BF01017F8201FF8603E19C0380E802000806001004003008003000006 00000E00000C00001C00001C0000380000380000700000700000F00000F00001E00001E00001E0 0003C00003C00003C00007C000078000078000030000141F799D17>55 D<001F000061800080C0 0100600300600600600600600600600E00C00F00800F818007C30007E40003F80001F80003FC00 047E00183F00300F00200700600700C00300C00300C00300800600800600C00C00C00800403000 3060001F8000131F7B9D17>I<001F0000718000C0C00180C00380E00700E00F00E00F01E01E01 E01E01E01E01E01E01C01C03C01C03C01C03C01C07C01C0F800C0F8006378003C700000F00000E 00000E00001C00601C00F03800F07000E0600080C0004380003E0000131F7B9D17>I<070F1F1F 0E0000000000000000000070F8F8F0E008147B930E>I<00000200000006000000060000000E00 00001E0000001E0000003F0000002F0000004F0000004F0000008F0000010F0000010F0000020F 0000020F0000040F00000C0F0000080F0000100F0000100F0000200F80003FFF800040078000C0 07800080078001000780010007800200078002000780060007801E000F80FF807FF81D207E9F22 >65 D<01FFFFC0001E00F0001E0078001E0038001E003C003C003C003C003C003C003C003C003C 0078007800780078007800F0007801E000F0078000FFFE0000F00F8000F003C001E001C001E001 E001E001E001E001E003C001E003C001E003C001E003C001C0078003C00780078007800F000780 1E000F007800FFFFE0001E1F7D9E20>I<0000FE0200078186001C004C0038003C0060003C00C0 001C01C0001803800018070000180F0000181E0000101E0000103C0000003C0000007800000078 0000007800000078000000F0000000F0000000F0000000F0000000F00000807000008070000080 700001003800010038000200180004000C001800060020000381C00000FE00001F217A9F21>I< 01FFFF80001E00E0001E0070001E0038001E001C003C001C003C000E003C000E003C000E007800 0E0078000E0078000E0078000E00F0001E00F0001E00F0001E00F0001E01E0003C01E0003C01E0 003C01E0007803C0007003C0007003C000E003C001C0078001C00780038007800E0007801C000F 007000FFFFC0001F1F7D9E22>I<01FFFFFE001E001C001E000C001E0004001E0004003C000400 3C0004003C0004003C00040078080800780800007808000078180000F0300000FFF00000F03000 00F0300001E0200001E0200001E0200001E0001003C0002003C0002003C0004003C00040078000 800780018007800100078007000F001F00FFFFFE001F1F7D9E1F>I<01FFFFFC001E0038001E00 18001E0008001E0008003C0008003C0008003C0008003C00080078001000780800007808000078 080000F0100000F0300000FFF00000F0300001E0200001E0200001E0200001E0200003C0000003 C0000003C0000003C00000078000000780000007800000078000000F800000FFF800001E1F7D9E 1E>I<0000FC040007030C001C00980030007800E0007801C00038038000300380003007000030 0E0000301E0000201E0000203C0000003C00000078000000780000007800000078000000F00000 00F000FFF0F0000780F0000780F0000F0070000F0070000F0070000F0070001E0038001E001800 3E001C002E000E00CC000383040000FC00001E217A9F23>I<01FFF3FFE0001F003E00001E003C 00001E003C00001E003C00003C007800003C007800003C007800003C007800007800F000007800 F000007800F000007800F00000F001E00000FFFFE00000F001E00000F001E00001E003C00001E0 03C00001E003C00001E003C00003C007800003C007800003C007800003C007800007800F000007 800F000007800F000007800F00000F801F0000FFF1FFE000231F7D9E22>I<01FFF0001F00001E 00001E00001E00003C00003C00003C00003C0000780000780000780000780000F00000F00000F0 0000F00001E00001E00001E00001E00003C00003C00003C00003C0000780000780000780000780 000F8000FFF800141F7D9E12>I<001FFF0000F80000F00000F00000F00001E00001E00001E000 01E00003C00003C00003C00003C0000780000780000780000780000F00000F00000F00000F0000 1E00001E00301E00781E00F83C00F83C00F0780080700040E00021C0001F000018207D9E18>I< 01FFF800001F0000001E0000001E0000001E0000003C0000003C0000003C0000003C0000007800 0000780000007800000078000000F0000000F0000000F0000000F0000001E0000001E0000001E0 000001E0008003C0010003C0010003C0030003C00200078006000780060007800C0007801C000F 007800FFFFF800191F7D9E1D>76 D<01FE00007FC0001E0000FC00001E0000F800001700017800 00170001780000270002F00000270004F00000270004F00000270008F00000470009E000004700 11E00000470021E00000470021E00000870043C00000838043C00000838083C00000838083C000 0103810780000103820780000103820780000103840780000203840F00000203880F0000020390 0F00000203900F00000401E01E00000401E01E00000401C01E00000C01801E00001C01803E0000 FF8103FFC0002A1F7D9E29>I<01FF007FE0001F000F00001F0004000017800400001780040000 278008000023C008000023C008000023C008000041E010000041E010000041F010000040F01000 0080F0200000807820000080782000008078200001003C400001003C400001003C400001001E40 0002001E800002001E800002000F800002000F800004000F0000040007000004000700000C0007 00001C00020000FF80020000231F7D9E22>I<0001FC0000070700001C01C0003000E000E00060 01C000700380007007800038070000380E0000381E0000381C0000383C0000383C000038780000 78780000787800007878000078F00000F0F00000F0F00000E0F00001E0F00001C0F00003C07000 03807000070078000F0038001E0038003C001C0070000E00E0000783800001FC00001D217A9F23 >I<01FFFF80001E00E0001E0070001E0038001E003C003C003C003C003C003C003C003C003C00 78007800780078007800F0007800E000F003C000F00F0000FFFC0000F0000001E0000001E00000 01E0000001E0000003C0000003C0000003C0000003C00000078000000780000007800000078000 000F800000FFF000001E1F7D9E1F>I<01FFFF00001E03C0001E00E0001E0070001E0078003C00 78003C0078003C0078003C0078007800F0007800F0007801E0007801C000F0070000F01E0000FF F00000F0380001E01C0001E01E0001E00E0001E00F0003C01E0003C01E0003C01E0003C01E0007 803C0007803C0807803C0807803C100F801C10FFF00C20000007C01D207D9E21>82 D<0007E040001C18C0003005800060038000C0038001C001800180010003800100038001000380 01000380000003C0000003C0000003F8000001FF800001FFE000007FF000001FF0000001F80000 00780000007800000038000000380020003800200038002000300060007000600060006000E000 7000C000E8038000C606000081F800001A217D9F1A>I<0FFFFFF01E0780E01807802010078020 20078020200F0020600F0020400F0020400F0020801E0040001E0000001E0000001E0000003C00 00003C0000003C0000003C00000078000000780000007800000078000000F0000000F0000000F0 000000F0000001E0000001E0000001E0000001E0000003E00000FFFF00001C1F789E21>I<7FFC 1FF807C003C00780010007800100078001000F0002000F0002000F0002000F0002001E0004001E 0004001E0004001E0004003C0008003C0008003C0008003C000800780010007800100078001000 78001000F0002000F0002000F0002000F0004000F0004000700080007001000030020000380400 000C18000007E000001D20779E22>III<00 FFF07FE0000F801F00000F001C00000F801000000780300000078020000007C040000003C08000 0003C100000003E200000001E400000001EC00000001F800000000F000000000F800000000F800 000000F8000000017C000000023C000000063C000000043E000000081E000000101E000000201F 000000400F000000800F000001800F8000010007800007000780001F000FC000FFC07FF800231F 7E9E22>I<00F1800389C00707800E03801C03803C0380380700780700780700780700F00E00F0 0E00F00E00F00E20F01C40F01C40703C40705C40308C800F070013147C9317>97 D<07803F8007000700070007000E000E000E000E001C001C001CF01D0C3A0E3C0E380F380F700F 700F700F700FE01EE01EE01EE01CE03CE038607060E031C01F0010207B9F15>I<007E0001C100 0300800E07801E07801C07003C0200780000780000780000F00000F00000F00000F00000F00000 70010070020030040018380007C00011147C9315>I<0000780003F80000700000700000700000 700000E00000E00000E00000E00001C00001C000F1C00389C00707800E03801C03803C03803807 00780700780700780700F00E00F00E00F00E00F00E20F01C40F01C40703C40705C40308C800F07 0015207C9F17>I<007C01C207010E011C013C013802780C7BF07C00F000F000F000F000700070 0170023804183807C010147C9315>I<00007800019C00033C00033C000718000700000700000E 00000E00000E00000E00000E0001FFE0001C00001C00001C00001C000038000038000038000038 0000380000700000700000700000700000700000700000E00000E00000E00000E00000C00001C0 0001C0000180003180007B0000F300006600003C00001629829F0E>I<003C6000E27001C1E003 80E00700E00F00E00E01C01E01C01E01C01E01C03C03803C03803C03803C03803C07003C07001C 0F001C17000C2E0003CE00000E00000E00001C00001C00301C00783800F0700060E0003F800014 1D7E9315>I<01E0000FE00001C00001C00001C00001C000038000038000038000038000070000 070000071E000763000E81800F01C00E01C00E01C01C03801C03801C03801C0380380700380700 380700380E10700E20700C20701C20700C40E00CC060070014207D9F17>I<00C001E001E001C0 00000000000000000000000000000E003300230043804300470087000E000E000E001C001C001C 003840388030807080310033001C000B1F7C9E0E>I<01E0000FE00001C00001C00001C00001C0 000380000380000380000380000700000700000703C00704200E08E00E11E00E21E00E40C01C80 001D00001E00001FC00038E000387000387000383840707080707080707080703100E03100601E 0013207D9F15>107 D<03C01FC0038003800380038007000700070007000E000E000E000E001C 001C001C001C0038003800380038007000700070007100E200E200E200E200640038000A207C9F 0C>I<1C0F80F0002630C318004740640C004780680E004700700E004700700E008E00E01C000E 00E01C000E00E01C000E00E01C001C01C038001C01C038001C01C038001C01C070803803807100 3803806100380380E10038038062007007006600300300380021147C9325>I<1C0F802630C047 40604780604700704700708E00E00E00E00E00E00E00E01C01C01C01C01C01C01C038438038838 03083807083803107003303001C016147C931A>I<007C0001C3000301800E01C01E01C01C01E0 3C01E07801E07801E07801E0F003C0F003C0F003C0F00780F00700700F00700E00301800187000 07C00013147C9317>I<01C1E002621804741C04781C04701E04701E08E01E00E01E00E01E00E0 1E01C03C01C03C01C03C01C0380380780380700380E003C1C0072380071E000700000700000E00 000E00000E00000E00001C00001C0000FFC000171D809317>I<00F0400388C00705800E03801C 03803C0380380700780700780700780700F00E00F00E00F00E00F00E00F01C00F01C00703C0070 5C0030B8000F380000380000380000700000700000700000700000E00000E0000FFE00121D7C93 15>I<1C1E002661004783804787804707804703008E00000E00000E00000E00001C00001C0000 1C00001C000038000038000038000038000070000030000011147C9313>I<00FC030206010C03 0C070C060C000F800FF007F803FC003E000E700EF00CF00CE008401020601F8010147D9313>I< 018001C0038003800380038007000700FFF007000E000E000E000E001C001C001C001C00380038 0038003820704070407080708031001E000C1C7C9B0F>I<0E00C03300E02301C04381C04301C0 4701C08703800E03800E03800E03801C07001C07001C07001C07101C0E20180E20180E201C1E20 0C264007C38014147C9318>I<0E03803307802307C04383C04301C04700C08700800E00800E00 800E00801C01001C01001C01001C02001C02001C04001C04001C08000E300003C00012147C9315 >I<0E00C1C03300E3C02301C3E04381C1E04301C0E04701C060870380400E0380400E0380400E 0380401C0700801C0700801C0700801C0701001C0701001C0602001C0F02000C0F04000E130800 03E1F0001B147C931E>I<0383800CC4401068E01071E02071E02070C040E00000E00000E00000 E00001C00001C00001C00001C040638080F38080F38100E5810084C60078780013147D9315>I< 0E00C03300E02301C04381C04301C04701C08703800E03800E03800E03801C07001C07001C0700 1C07001C0E00180E00180E001C1E000C3C0007DC00001C00001C00003800F03800F07000E06000 C0C0004380003E0000131D7C9316>I E /Fi 14 113 df0 D<70F8F8F87005057C8D0D>I<400004C0000C6000183000301800600C00C00601800303000186 0000CC0000780000300000300000780000CC000186000303000601800C00C01800603000306000 18C0000C40000416187A9623>I<00020000000600000006000000060000000600000006000000 0600000006000000060000000600000006000000060000000600000006000000060000FFFFFFF0 FFFFFFF00006000000060000000600000006000000060000000600000006000000060000000600 0000060000000600000006000000060000FFFFFFF0FFFFFFF01C207D9E23>6 D<03C00FF01FF83FFC7FFE7FFEFFFFFFFFFFFFFFFFFFFFFFFF7FFE7FFE3FFC1FF80FF003C01012 7D9317>15 D<000000C0000003C000000F0000003C000000F0000003C00000070000001C000000 78000001E00000078000001E00000078000000E0000000780000001E0000000780000001E00000 00780000001C0000000700000003C0000000F00000003C0000000F00000003C0000000C0000000 000000000000000000000000000000000000000000000000007FFFFF80FFFFFFC01A247C9C23> 20 D<0FC000101FF000103FF80010703E0030E00F0070C007C0E08001FFC08000FF8080003F00 00000000000000000FC000101FF000103FF80010703E0030E00F0070C007C0E08001FFC08000FF 8080003F001C147D9523>25 D<003FF800FFF803C0000700000C00001800003000003000006000 00600000C00000C00000C00000FFFFF8FFFFF8C00000C00000C000006000006000003000003000 001800000C000007000003C00000FFF8003FF8151C7C981E>50 D<00FE0003FF800F01E01C0070 30001860000C60000CC00006C00006C00006C00006C00006C00006C00006C00006C00006C00006 C00006C00006C00006C00006C00006C00006C00006C00006C00006C00006400002171C7D9A1E> 92 D<7FE0FFE0C000C000C000C000C000C000C000C000C000C000C000C000C000C000C000C000 C000C000C000C000C000C000C000C000C000C000C000C000C000C000C000C000C000C000C000C0 00C000C000C000C000C000C000C00040000B2E78A114>100 DI<000F0038006000E001C001C001C001C001C001C001C001C001C001C001C001 C001C001C001C0038007001E00F8001E000700038001C001C001C001C001C001C001C001C001C0 01C001C001C001C001C001C000E000600038000F102D7DA117>II<0000000008000000001800000000300000000030000000006000000000600000 0000C000000000C000000001800000000180000000030000000003000000000600000000060000 00000C000000000C00000000180000000018000000003000000000300000000060000000006000 000000C000060000C0001E000180002F000180004F000300008780030000078006000003C00600 0003C00C000003C00C000001E018000001E018000000F030000000F03000000078600000007860 0000003CC00000003CC00000001F800000001F800000000F000000000F00000000060000000006 000000252E7E8126>112 D E /Fj 22 117 df<70F8FCFC74040404080810102040060E7C840D> 59 D<000100030003000600060006000C000C000C001800180018003000300030006000600060 00C000C000C00180018001800300030003000600060006000C000C000C00180018001800300030 003000600060006000C000C000C000102D7DA117>61 DI<0000FE0200078186001C004C0038003C0060003C00C0 001C01C0001803800018070000180F0000181E0000101E0000103C0000003C0000007800000078 0000007800000078000000F0000000F0000000F0000000F0000000F00000807000008070000080 700001003800010038000200180004000C001800060020000381C00000FE00001F217E9F20>67 D<00007E0100038183000E00460038002E0070001E00E0000E01C0000C0380000C0700000C0F00 000C1E0000081E0000083C0000003C00000078000000780000007800000078000000F0000000F0 007FFCF00001E0F00001E0F00003C0700003C0700003C0700003C038000780380007801C000F80 0C000B80060033000380C100007F000020217E9F24>71 D<0001FC0000070700001C01C0003000 E000E0006001C000700380007007800038070000380E0000381E0000381C0000383C0000383C00 003878000078780000787800007878000078F00000F0F00000F0F00000E0F00001E0F00001C0F0 0003C0700003807000070078000F0038001E0038003C001C0070000E00E0000783800001FC0000 1D217E9F23>79 D<00FFFFC0000F0070000F0038000F001C000F001E001E001E001E001E001E00 1E001E001E003C003C003C003C003C0078003C0070007800E000780380007FFE000078000000F0 000000F0000000F0000000F0000001E0000001E0000001E0000001E0000003C0000003C0000003 C0000003C0000007C00000FFFC00001F1F7E9E1D>I<0007E0800018118000300B000060070000 C0070001C0030001800200038002000380020003800200038000000380000003C0000003F80000 03FF800001FFC00000FFE000003FF0000003F0000000F000000070000000700000007000200070 0020007000200060006000E0006000C0006001C00070018000E8030000C60E000081F800001921 7D9F1C>83 D<0FFFFFFC1E03C0381803C0181003C0082003C00820078008600780084007800840 078008800F0010000F0000000F0000000F0000001E0000001E0000001E0000001E0000003C0000 003C0000003C0000003C00000078000000780000007800000078000000F0000000F0000000F000 0000F0000001F000007FFFC0001E1F7F9E1B>I<00FFF83FF8000FC00F80000F80060000078004 000007C008000003C010000003C020000003E040000001E080000001F100000000F300000000F6 00000000FC0000000078000000007C000000007C000000007C00000000BE000000011E00000002 1E000000061F0000000C0F000000080F800000100780000020078000004007C000008003C00001 0003E000030003E0000F0007E000FFE01FFE00251F7F9E26>88 DI<00007C0000CE00019E00039E00030C000700000700000700000700000E00000E00000E0000 FFF0000E00000E00001C00001C00001C00001C00001C0000380000380000380000380000380000 700000700000700000700000700000E00000E00000E00000E00000C00001C000318000798000F3 00006200003C000017297E9F16>102 D<00E001E001E000C00000000000000000000000000000 0E00130023804380438043808700070007000E000E001C001C001C20384038403840388019000E 000B1F7E9E10>105 D<0000C00001E00001E00001C00000000000000000000000000000000000 00000000001E00006300004380008380010380010380020700000700000700000700000E00000E 00000E00000E00001C00001C00001C00001C000038000038000038000038000070000070003070 0078E000F1C0006380003E00001328819E13>I<01E0000FE00001C00001C00001C00001C00003 80000380000380000380000700000700000701E00706100E08700E10F00E20F00E40601C80001D 00001E00001FC000387000383800383800381C20703840703840703840701880E01880600F0014 207E9F18>I<1E07C07C00231861860023A032030043C034030043803803804380380380870070 07000700700700070070070007007007000E00E00E000E00E00E000E00E00E000E00E01C101C01 C01C201C01C038201C01C038401C01C0184038038018801801800F0024147E9328>109 D<1E07802318C023A06043C0704380704380708700E00700E00700E00700E00E01C00E01C00E01 C00E03821C03841C07041C07081C03083803101801E017147E931B>I<03C1E004621804741C08 781C08701E08701E10E01E00E01E00E01E00E01E01C03C01C03C01C03C01C03803807803807003 80E003C1C0072380071E000700000700000E00000E00000E00000E00001C00001C0000FFC00017 1D819317>112 D<00F0400388C00705800E03801C03803C0380380700780700780700780700F0 0E00F00E00F00E00F00E00F01C00F01C00703C00705C0030B8000F380000380000380000700000 700000700000700000E00000E0000FFE00121D7E9314>I<1E1E0023210023C38043C780438780 4383008700000700000700000700000E00000E00000E00000E00001C00001C00001C00001C0000 38000018000011147E9315>I<007C018203010603060706060E00078007F803FC01FE001F0007 7007F006F006E004400820301FC010147E9315>I<00C000E001C001C001C001C003800380FFF8 038007000700070007000E000E000E000E001C001C001C001C10382038203820384018800F000D 1C7F9B10>I E /Fk 45 123 df<000FF000007FFC0001F80E0003E01F0007C03F000F803F000F 803F000F801E000F800C000F8000000F8000000F8000000F800000FFFFFF00FFFFFF000F801F00 0F801F000F801F000F801F000F801F000F801F000F801F000F801F000F801F000F801F000F801F 000F801F000F801F000F801F000F801F000F801F000F801F000F801F007FF0FFE07FF0FFE01B23 7FA21F>12 D<387CFEFEFE7C3807077C8610>46 D<00180000780001F800FFF800FFF80001F800 01F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F800 01F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F8007FFFE07FFFE0 13207C9F1C>49 D<03FC000FFF003C1FC07007E07C07F0FE03F0FE03F8FE03F8FE01F87C01F838 03F80003F80003F00003F00007E00007C0000F80001F00003E0000380000700000E01801C01803 80180700180E00380FFFF01FFFF03FFFF07FFFF0FFFFF0FFFFF015207D9F1C>I<00FE0007FFC0 0F07E01E03F03F03F03F81F83F81F83F81F81F03F81F03F00003F00003E00007C0001F8001FE00 01FF000007C00001F00001F80000FC0000FC3C00FE7E00FEFF00FEFF00FEFF00FEFF00FC7E01FC 7801F81E07F00FFFC001FE0017207E9F1C>I<0000E00001E00003E00003E00007E0000FE0001F E0001FE00037E00077E000E7E001C7E00187E00307E00707E00E07E00C07E01807E03807E07007 E0E007E0FFFFFEFFFFFE0007E00007E00007E00007E00007E00007E00007E000FFFE00FFFE1720 7E9F1C>I<1000201E01E01FFFC01FFF801FFF001FFE001FF8001BC00018000018000018000018 000019FC001FFF001E0FC01807E01803E00003F00003F00003F80003F83803F87C03F8FE03F8FE 03F8FC03F0FC03F07007E03007C01C1F800FFF0003F80015207D9F1C>I<000070000000007000 000000F800000000F800000000F800000001FC00000001FC00000003FE00000003FE00000003FE 00000006FF000000067F0000000E7F8000000C3F8000000C3F800000183FC00000181FC0000038 1FE00000300FE00000300FE00000600FF000006007F00000E007F80000FFFFF80000FFFFF80001 8001FC00018001FC00038001FE00030000FE00030000FE000600007F000600007F00FFE00FFFF8 FFE00FFFF825227EA12A>65 DI<0003FE0080 001FFF818000FF01E38001F8003F8003E0001F8007C0000F800F800007801F800007803F000003 803F000003807F000001807E000001807E00000180FE00000000FE00000000FE00000000FE0000 0000FE00000000FE00000000FE00000000FE000000007E000000007E000001807F000001803F00 0001803F000003801F800003000F8000030007C000060003F0000C0001F800380000FF00F00000 1FFFC0000003FE000021227DA128>II<0003 FE0040001FFFC0C0007F00F1C001F8003FC003F0000FC007C00007C00FC00003C01F800003C03F 000001C03F000001C07F000000C07E000000C07E000000C0FE00000000FE00000000FE00000000 FE00000000FE00000000FE00000000FE00000000FE000FFFFC7E000FFFFC7F00001FC07F00001F C03F00001FC03F00001FC01F80001FC00FC0001FC007E0001FC003F0001FC001FC003FC0007F80 E7C0001FFFC3C00003FF00C026227DA12C>71 D73 D76 DII<00 07FC0000003FFF800000FC07E00003F001F80007E000FC000FC0007E001F80003F001F80003F00 3F00001F803F00001F807F00001FC07E00000FC07E00000FC0FE00000FE0FE00000FE0FE00000F E0FE00000FE0FE00000FE0FE00000FE0FE00000FE0FE00000FE0FE00000FE07E00000FC07F0000 1FC07F00001FC03F00001F803F80003F801F80003F000FC0007E0007E000FC0003F001F80000FC 07E000003FFF80000007FC000023227DA12A>II82 D<01FC0407FF8C1F03FC3C 007C7C003C78001C78001CF8000CF8000CFC000CFC0000FF0000FFE0007FFF007FFFC03FFFF01F FFF80FFFFC03FFFE003FFE0003FF00007F00003F00003FC0001FC0001FC0001FE0001EE0001EF0 003CFC003CFF00F8C7FFE080FF8018227DA11F>I<7FFFFFFF807FFFFFFF807E03F80F807803F8 07807003F803806003F80180E003F801C0E003F801C0C003F800C0C003F800C0C003F800C0C003 F800C00003F800000003F800000003F800000003F800000003F800000003F800000003F8000000 03F800000003F800000003F800000003F800000003F800000003F800000003F800000003F80000 0003F800000003F800000003F800000003F800000003F8000003FFFFF80003FFFFF80022227EA1 27>II<07FC001FFF803F07C03F03E03F01E0 3F01F01E01F00001F00001F0003FF003FDF01FC1F03F01F07E01F0FC01F0FC01F0FC01F0FC01F0 7E02F07E0CF81FF87F07E03F18167E951B>97 DI<00FF8007FFE00F83F01F03F03E03F07E03F07C01E07C0000FC0000FC 0000FC0000FC0000FC0000FC00007C00007E00007E00003E00301F00600FC0E007FF8000FE0014 167E9519>I<0001FE000001FE0000003E0000003E0000003E0000003E0000003E0000003E0000 003E0000003E0000003E0000003E0000003E0001FC3E0007FFBE000F81FE001F007E003E003E00 7E003E007C003E00FC003E00FC003E00FC003E00FC003E00FC003E00FC003E00FC003E00FC003E 007C003E007C003E003E007E001E00FE000F83BE0007FF3FC001FC3FC01A237EA21F>I<00FE00 07FF800F87C01E01E03E01F07C00F07C00F8FC00F8FC00F8FFFFF8FFFFF8FC0000FC0000FC0000 7C00007C00007E00003E00181F00300FC07003FFC000FF0015167E951A>I<003F8000FFC001E3 E003C7E007C7E00F87E00F83C00F80000F80000F80000F80000F80000F8000FFFC00FFFC000F80 000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80 000F80000F80000F80000F80007FF8007FF80013237FA211>I<03FC1E0FFF7F1F0F8F3E07CF3C 03C07C03E07C03E07C03E07C03E07C03E03C03C03E07C01F0F801FFF0013FC0030000030000038 00003FFF801FFFF00FFFF81FFFFC3800FC70003EF0001EF0001EF0001EF0001E78003C7C007C3F 01F80FFFE001FF0018217E951C>II<1C003F007F007F007F003F001C000000000000000000000000000000FF00FF001F 001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F00FFE0FFE0 0B247EA310>I107 DI< FF07F007F000FF1FFC1FFC001F303E303E001F403E403E001F801F801F001F801F801F001F001F 001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F00 1F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F 001F00FFE0FFE0FFE0FFE0FFE0FFE02B167E9530>II< 00FE0007FFC00F83E01E00F03E00F87C007C7C007C7C007CFC007EFC007EFC007EFC007EFC007E FC007EFC007E7C007C7C007C3E00F81F01F00F83E007FFC000FE0017167E951C>II<00FE030007FF87000FC1C7001F006F003F003F007E003F007E00 1F007C001F00FC001F00FC001F00FC001F00FC001F00FC001F00FC001F00FC001F007E001F007E 001F003E003F001F007F000FC1DF0007FF9F0001FC1F0000001F0000001F0000001F0000001F00 00001F0000001F0000001F0000001F000000FFE00000FFE01B207E951E>II<0FF3003FFF00781F00600700E0 0300E00300F00300FC00007FE0007FF8003FFE000FFF0001FF00000F80C00780C00380E00380E0 0380F00700FC0E00EFFC00C7F00011167E9516>I<018000018000018000018000038000038000 0780000780000F80003F8000FFFF00FFFF000F80000F80000F80000F80000F80000F80000F8000 0F80000F80000F80000F80000F81800F81800F81800F81800F81800F830007C30003FE0000F800 11207F9F16>IIII<7FFFF07FFFF07C03E07007C0600FC0E01F80C01F00C03E00C07E0000FC0000F80001F00003 F03007E03007C0300F80701F80703F00603E00E07C03E0FFFFE0FFFFE014167E9519>122 D E /Fl 61 126 df<007000F001E003C007800F001E001C00380038007000700070007000E000 E000E000E000E000E000E000E0007000700070007000380038001C001E000F00078003C001F000 F000700C24799F18>40 D<6000F00078003C001E000F000780038001C001C000E000E000E000E0 0070007000700070007000700070007000E000E000E000E001C001C0038007800F001E003C0078 00F00060000C247C9F18>I<01C00001C00001C00001C000C1C180F1C780F9CF807FFF001FFC00 07F00007F0001FFC007FFF00F9CF80F1C780C1C18001C00001C00001C00001C00011147D9718> I<1C3E7E7F3F1F070E1E7CF860080C788518>44 D<7FFF00FFFF80FFFF807FFF0011047D8F18> I<01F00007FC000FFE001F1F001C07003803807803C07001C07001C0E000E0E000E0E000E0E000 E0E000E0E000E0E000E0E000E0E000E0F001E07001C07001C07803C03803801C07001F1F000FFE 0007FC0001F000131C7E9B18>48 D<01800380038007800F803F80FF80FB804380038003800380 03800380038003800380038003800380038003800380038003807FFCFFFE7FFC0F1C7B9B18>I< 03F0000FFE003FFF007C0F807003C0E001C0F000E0F000E06000E00000E00000E00001C00001C0 0003C0000780000F00001E00003C0000780000F00001E00007C0000F80001E00E03C00E07FFFE0 FFFFE07FFFE0131C7E9B18>I<07F8001FFE003FFF007807807803C07801C03001C00001C00003 C0000380000F0003FF0003FE0003FF000007800003C00001C00000E00000E00000E0F000E0F000 E0F001C0F003C07C07803FFF001FFE0003F800131C7E9B18>I<001F00003F0000770000770000 E70001E70001C7000387000787000707000E07001E07003C0700380700780700F00700FFFFF8FF FFF8FFFFF8000700000700000700000700000700000700007FF000FFF8007FF0151C7F9B18>I< 1FFF803FFF803FFF803800003800003800003800003800003800003800003800003BF8003FFE00 3FFF003C07801803C00001C00000E00000E06000E0F000E0F000E0E001C07003C07C0F803FFF00 1FFC0003F000131C7E9B18>I<007E0001FF0007FF800F83C01E03C01C03C03801803800007000 00700000E1F800E7FE00FFFF00FE0780F803C0F001C0F000E0E000E0F000E07000E07000E07000 E03801C03C03C01E07800FFF0007FE0001F800131C7E9B18>I<3078FCFC783000000000000000 003078FCFC78300614779318>58 D<183C7E7E3C180000000000000000183C7E7E3E1E0E1C3C78 F060071A789318>I<000300000780001F80003F00007E0001FC0003F00007E0001FC0003F0000 7E0000FC0000FC00007E00003F00001FC00007E00003F00001FC00007E00003F00001F80000780 00030011187D9918>I<7FFFC0FFFFE0FFFFE0FFFFE0000000000000000000000000FFFFE0FFFF E0FFFFE07FFFC0130C7E9318>I<00700000F80000F80000D80000D80001DC0001DC0001DC0001 8C00038E00038E00038E00038E000306000707000707000707000707000FFF800FFF800FFF800E 03800E03801C01C01C01C07F07F0FF8FF87F07F0151C7F9B18>65 DI<00F8E003FEE007FFE00F07E01E03E03C01E03800E07000E07000E0700000E00000E00000E0 0000E00000E00000E00000E00000E000007000007000E07000E03800E03C00E01E01C00F07C007 FF8003FE0000F800131C7E9B18>I<7FF800FFFE007FFF001C0F801C03C01C03C01C01E01C00E0 1C00E01C00F01C00701C00701C00701C00701C00701C00701C00701C00701C00F01C00E01C00E0 1C01E01C01C01C03C01C0F807FFF00FFFE007FF800141C7F9B18>II<01F1C003FDC00FFFC01F0FC01C03C03803C03801C07001C07001C0700000E00000E00000E0 0000E00000E00000E00FF0E01FF0E00FF07001C07001C07003C03803C03803C01C07C01F0FC00F FFC003FDC001F1C0141C7E9B18>71 D<7F07F0FF8FF87F07F01C01C01C01C01C01C01C01C01C01 C01C01C01C01C01C01C01C01C01FFFC01FFFC01FFFC01C01C01C01C01C01C01C01C01C01C01C01 C01C01C01C01C01C01C01C01C07F07F0FF8FF87F07F0151C7F9B18>I<7FFF00FFFF807FFF0001 C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001 C00001C00001C00001C00001C00001C00001C00001C00001C0007FFF00FFFF807FFF00111C7D9B 18>I<7FE000FFE0007FE0000E00000E00000E00000E00000E00000E00000E00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E00700E00700E00700E00700E0070 7FFFF0FFFFF07FFFF0141C7F9B18>76 DI<7E07F0FF0FF87F07F0 1D81C01D81C01D81C01DC1C01CC1C01CC1C01CE1C01CE1C01CE1C01C61C01C71C01C71C01C31C0 1C39C01C39C01C39C01C19C01C19C01C1DC01C0DC01C0DC01C0DC07F07C0FF87C07F03C0151C7F 9B18>I<0FF8003FFE007FFF00780F00700700F00780E00380E00380E00380E00380E00380E003 80E00380E00380E00380E00380E00380E00380E00380E00380E00380E00380F00780700700780F 007FFF003FFE000FF800111C7D9B18>II<7FFFF8FFFFF8FFFFF8 E07038E07038E07038E07038007000007000007000007000007000007000007000007000007000 00700000700000700000700000700000700000700000700000700007FF0007FF0007FF00151C7F 9B18>84 D86 D<7F8FE07F9FE07F8FE00E07000F0700070E00078E 00039C0003DC0001F80001F80000F00000F00000700000F00000F80001F80001DC00039E00038E 00070F000707000E07800E03801E03C07F07F0FF8FF87F07F0151C7F9B18>88 D91 D93 D<7FFF00FFFF80FFFF807FFF0011047D7F18>95 D<1FE0003FF8007FFC00781E00300E 0000070000070000FF0007FF001FFF007F0700780700E00700E00700E00700F00F00781F003FFF F01FFBF007E1F014147D9318>97 D<7E0000FE00007E00000E00000E00000E00000E00000E0000 0E3E000EFF800FFFC00FC1E00F80E00F00700E00700E00380E00380E00380E00380E00380E0038 0F00700F00700F80E00FC1E00FFFC00EFF80063E00151C809B18>I<01FE0007FF001FFF803E07 80380300700000700000E00000E00000E00000E00000E00000E000007000007001C03801C03E03 C01FFF8007FF0001FC0012147D9318>I<001F80003F80001F8000038000038000038000038000 038003E3800FFB801FFF803C1F80380F80700780700380E00380E00380E00380E00380E00380E0 0380700780700780380F803C1F801FFFF00FFBF803E3F0151C7E9B18>I<01F00007FC001FFE00 3E0F00380780700380700380E001C0E001C0FFFFC0FFFFC0FFFFC0E000007000007001C03801C0 3E03C01FFF8007FF0001FC0012147D9318>I<001F80007FC000FFE000E1E001C0C001C00001C0 0001C0007FFFC0FFFFC0FFFFC001C00001C00001C00001C00001C00001C00001C00001C00001C0 0001C00001C00001C00001C00001C0007FFF007FFF007FFF00131C7F9B18>I<01E1F007FFF80F FFF81E1E301C0E003807003807003807003807003807001C0E001E1E001FFC001FF80039E00038 00001C00001FFE001FFFC03FFFE07801F0700070E00038E00038E00038E000387800F07E03F01F FFC00FFF8001FC00151F7F9318>I<7E0000FE00007E00000E00000E00000E00000E00000E0000 0E3E000EFF800FFFC00FC1C00F80E00F00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E0 0E00E00E00E00E00E00E00E07FC3FCFFE7FE7FC3FC171C809B18>I<03800007C00007C00007C0 000380000000000000000000000000007FC000FFC0007FC00001C00001C00001C00001C00001C0 0001C00001C00001C00001C00001C00001C00001C00001C00001C000FFFF00FFFF80FFFF00111D 7C9C18>I107 D<7FE000FFE0007FE00000E00000E00000E00000 E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000 E00000E00000E00000E00000E00000E0007FFFC0FFFFE07FFFC0131C7E9B18>I<7CE0E000FFFB F8007FFFF8001F1F1C001E1E1C001E1E1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C 1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C007F1F1F00FFBFBF807F1F1F0019148193 18>I<7E3E00FEFF807FFFC00FC1C00F80E00F00E00E00E00E00E00E00E00E00E00E00E00E00E0 0E00E00E00E00E00E00E00E00E00E07FC3FCFFE7FE7FC3FC1714809318>I<01F0000FFE001FFF 003E0F803803807001C07001C0E000E0E000E0E000E0E000E0E000E0F001E07001C07803C03C07 803E0F801FFF000FFE0001F00013147E9318>I<7E3E00FEFF807FFFC00FC1E00F80E00F00700E 00700E00380E00380E00380E00380E00380E00380F00700F00700F80E00FC1E00FFFC00EFF800E 3E000E00000E00000E00000E00000E00000E00000E00007FC000FFE0007FC000151E809318>I< 7F87E0FF9FF07FBFF803F87803F03003E00003C00003C000038000038000038000038000038000 0380000380000380000380007FFE00FFFF007FFE0015147F9318>114 D<07F7003FFF007FFF00 780F00E00700E00700E007007C00007FE0001FFC0003FE00001F00600780E00380E00380F00380 F80F00FFFF00FFFC00E7F00011147D9318>I<0180000380000380000380000380007FFFC0FFFF C0FFFFC00380000380000380000380000380000380000380000380000380000380400380E00380 E00380E001C1C001FFC000FF80003E0013197F9818>I<7E07E0FE0FE07E07E00E00E00E00E00E 00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E01E00F03E007FFFC03 FFFE01FCFC1714809318>I<7F8FF0FF8FF87F8FF01E03C00E03800E03800E0380070700070700 070700038E00038E00038E00038E0001DC0001DC0001DC0000F80000F80000700015147F9318> II<7F8FF07F9FF07F8FF007 0700078E00039E0001DC0001F80000F80000700000F00000F80001DC00039E00038E000707000F 07807F8FF0FF8FF87F8FF015147F9318>I<7F8FF0FF8FF87F8FF00E01C00E03800E0380070380 070700070700038700038600038E0001CE0001CE0000CC0000CC0000DC00007800007800007800 00700000700000700000F00000E00079E0007BC0007F80003F00001E0000151E7F9318>I<3FFF F07FFFF07FFFF07001E07003C0700780000F00001E00003C0000F80001F00003C0000780000F00 701E00703C0070780070FFFFF0FFFFF0FFFFF014147F9318>I<0007E0001FE0007FE000780000 E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00001E0007FC000FF 8000FF80007FC00001E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000 E00000E000007800007FE0001FE00007E013247E9F18>I<7C0000FF0000FFC00003C00000E000 00E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000F000007FC0003FE0 003FE0007FC000F00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E000 00E00003C000FFC000FF00007C000013247E9F18>125 D E /Fm 30 121 df<000E00001E00007E0007FE00FFFE00FFFE00F8FE0000FE0000FE0000FE0000FE0000FE0000 FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000 FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE007FFFFE7FFFFE7F FFFE17277BA622>49 D<00FF800003FFF0000FFFFC001F03FE003800FF007C007F80FE003FC0FF 003FC0FF003FE0FF001FE0FF001FE07E001FE03C003FE000003FE000003FC000003FC000007F80 00007F000000FE000000FC000001F8000003F0000003E00000078000000F0000001E0000003C00 E0007000E000E000E001C001C0038001C0070001C00FFFFFC01FFFFFC03FFFFFC07FFFFFC0FFFF FF80FFFFFF80FFFFFF801B277DA622>I<007F800003FFF00007FFFC000F81FE001F00FF003F80 FF003F807F803F807F803F807F801F807F800F007F800000FF000000FF000000FE000001FC0000 01F8000007F00000FFC00000FFF0000001FC0000007E0000007F0000007F8000003FC000003FC0 00003FE000003FE03C003FE07E003FE0FF003FE0FF003FE0FF003FC0FF007FC07E007F807C007F 003F01FE001FFFFC0007FFF00000FF80001B277DA622>I<00000E0000001E0000003E0000007E 000000FE000000FE000001FE000003FE0000077E00000E7E00000E7E00001C7E0000387E000070 7E0000E07E0000E07E0001C07E0003807E0007007E000E007E000E007E001C007E0038007E0070 007E00E0007E00FFFFFFF8FFFFFFF8FFFFFFF80000FE000000FE000000FE000000FE000000FE00 0000FE000000FE000000FE00007FFFF8007FFFF8007FFFF81D277EA622>I<0C0003000F803F00 0FFFFE000FFFFC000FFFF8000FFFF0000FFFE0000FFFC0000FFE00000E0000000E0000000E0000 000E0000000E0000000E0000000E7FC0000FFFF8000F80FC000E003E000C003F0000001F800000 1FC000001FC000001FE000001FE018001FE07C001FE0FE001FE0FE001FE0FE001FE0FE001FC0FC 001FC078003F8078003F803C007F001F01FE000FFFF80003FFF00000FF80001B277DA622>I<1C 003E007F00FF80FF80FF807F003E001C000000000000000000000000000000000000001C003E00 7F00FF80FF80FF807F003E001C00091B7B9A13>58 D<000003800000000007C00000000007C000 0000000FE0000000000FE0000000000FE0000000001FF0000000001FF0000000003FF800000000 3FF8000000003FF80000000073FC0000000073FC00000000F3FE00000000E1FE00000000E1FE00 000001C0FF00000001C0FF00000003C0FF80000003807F80000007807FC0000007003FC0000007 003FC000000E003FE000000E001FE000001E001FF000001C000FF000001FFFFFF000003FFFFFF8 00003FFFFFF80000780007FC0000700003FC0000700003FC0000E00001FE0000E00001FE0001E0 0001FF0001C00000FF0001C00000FF00FFFE001FFFFEFFFE001FFFFEFFFE001FFFFE2F297EA834 >65 D<00003FF001800003FFFE0380000FFFFF8780003FF007DF8000FF8001FF8001FE00007F80 03FC00003F8007F000001F800FF000000F801FE0000007801FE0000007803FC0000007803FC000 0003807FC0000003807F80000003807F8000000000FF8000000000FF8000000000FF8000000000 FF8000000000FF8000000000FF8000000000FF8000000000FF8000000000FF80000000007F8000 0000007F80000000007FC0000003803FC0000003803FC0000003801FE0000003801FE000000700 0FF00000070007F000000E0003FC00001E0001FE00003C0000FF8000F800003FF007E000000FFF FFC0000003FFFF000000003FF8000029297CA832>67 D<00007FE003000003FFFC0700001FFFFF 0F00003FF00FFF0000FF8001FF0001FE0000FF0003F800003F0007F000003F000FF000001F001F E000000F001FE000000F003FC000000F003FC0000007007FC0000007007F80000007007F800000 0000FF8000000000FF8000000000FF8000000000FF8000000000FF8000000000FF8000000000FF 8000000000FF8000000000FF8001FFFFF87F8001FFFFF87F8001FFFFF87FC00000FF003FC00000 FF003FC00000FF001FE00000FF001FE00000FF000FF00000FF0007F00000FF0003F80000FF0001 FE0000FF0000FF8001FF00003FF007BF00001FFFFF1F000003FFFE0F0000007FF003002D297CA8 36>71 D73 D76 D80 D82 D<007F806003FFF0E007FFF9E00F807FE01F001FE03E0007E07C0003E07C0001E0FC0001E0FC00 01E0FC0000E0FE0000E0FE0000E0FF000000FFC000007FFE00007FFFE0003FFFFC001FFFFE000F FFFF8007FFFFC003FFFFE000FFFFE00007FFF000007FF000000FF8000007F8000003F8600001F8 E00001F8E00001F8E00001F8F00001F0F00001F0F80003F0FC0003E0FF0007C0FFE01F80F3FFFF 00E0FFFE00C01FF0001D297CA826>I<01FF800007FFF0000F81F8001FC07E001FC07E001FC03F 000F803F8007003F8000003F8000003F8000003F80000FFF8000FFFF8007FC3F800FE03F803F80 3F803F003F807F003F80FE003F80FE003F80FE003F80FE003F807E007F807F00DF803F839FFC0F FF0FFC01FC03FC1E1B7E9A21>97 D<001FF80000FFFE0003F01F0007E03F800FC03F801F803F80 3F801F007F800E007F0000007F000000FF000000FF000000FF000000FF000000FF000000FF0000 00FF0000007F0000007F0000007F8000003F8001C01F8001C00FC0038007E0070003F01E0000FF FC00001FE0001A1B7E9A1F>99 D<00003FF80000003FF80000003FF800000003F800000003F800 000003F800000003F800000003F800000003F800000003F800000003F800000003F800000003F8 00000003F800000003F800001FE3F80000FFFBF80003F03FF80007E00FF8000FC007F8001F8003 F8003F8003F8007F0003F8007F0003F8007F0003F800FF0003F800FF0003F800FF0003F800FF00 03F800FF0003F800FF0003F800FF0003F8007F0003F8007F0003F8007F0003F8003F8003F8001F 8003F8000F8007F80007C00FF80003F03BFF8000FFF3FF80003FC3FF80212A7EA926>I<003FE0 0001FFF80003F07E0007C01F000F801F801F800F803F800FC07F000FC07F0007C07F0007E0FF00 07E0FF0007E0FFFFFFE0FFFFFFE0FF000000FF000000FF0000007F0000007F0000007F0000003F 8000E01F8000E00FC001C007E0038003F81F0000FFFE00001FF0001B1B7E9A20>I<0007F0003F FC00FE3E01F87F03F87F03F07F07F07F07F03E07F00007F00007F00007F00007F00007F00007F0 00FFFFC0FFFFC0FFFFC007F00007F00007F00007F00007F00007F00007F00007F00007F00007F0 0007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F0007FFF807FFF 807FFF80182A7EA915>I<07000FC01FE03FE03FE03FE01FE00FC0070000000000000000000000 00000000FFE0FFE0FFE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00F E00FE00FE00FE00FE00FE00FE0FFFEFFFEFFFE0F2B7DAA14>105 D108 DII<003FE00001FFFC0003 F07E000FC01F801F800FC03F800FE03F0007E07F0007F07F0007F07F0007F0FF0007F8FF0007F8 FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F87F0007F07F0007F03F800FE03F800F E01F800FC00FC01F8007F07F0001FFFC00003FE0001D1B7E9A22>II114 D<03FE300FFFF01E03F03800F0700070F00070F00070F80070FC0000FFE0007F FE007FFF803FFFE01FFFF007FFF800FFF80003FC0000FC60007CE0003CF0003CF00038F80038FC 0070FF01E0F7FFC0C1FF00161B7E9A1B>I<00700000700000700000700000F00000F00000F000 01F00003F00003F00007F0001FFFF0FFFFF0FFFFF007F00007F00007F00007F00007F00007F000 07F00007F00007F00007F00007F00007F00007F00007F03807F03807F03807F03807F03807F038 03F03803F87001F86000FFC0001F8015267FA51B>II120 D E /Fn 82 128 df<001F83E000F06E3001C078780380F8780300F03007007000070070000700 700007007000070070000700700007007000FFFFFF800700700007007000070070000700700007 007000070070000700700007007000070070000700700007007000070070000700700007007000 070070000700700007007000070070007FE3FF001D20809F1B>11 D<003F0000E0C001C0C00381 E00701E00701E0070000070000070000070000070000070000FFFFE00700E00700E00700E00700 E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700 E00700E07FC3FE1720809F19>I<003FE000E0E001C1E00381E00700E00700E00700E00700E007 00E00700E00700E00700E0FFFFE00700E00700E00700E00700E00700E00700E00700E00700E007 00E00700E00700E00700E00700E00700E00700E00700E00700E00700E07FE7FE1720809F19>I< 001F81F80000F04F040001C07C06000380F80F000300F00F000700F00F00070070000007007000 000700700000070070000007007000000700700000FFFFFFFF0007007007000700700700070070 070007007007000700700700070070070007007007000700700700070070070007007007000700 70070007007007000700700700070070070007007007000700700700070070070007007007007F E3FE3FF02420809F26>I<70F8F8F8F8F8F8F87070707070707070707020202020200000000000 70F8F8F87005217CA00D>33 D<7038F87CFC7EFC7E743A04020402040208040804100810082010 40200F0E7E9F17>I<70F8FCFC74040404080810102040060E7C9F0D>39 D<0020004000800100020006000C000C00180018003000300030007000600060006000E000E000 E000E000E000E000E000E000E000E000E000E0006000600060007000300030003000180018000C 000C000600020001000080004000200B2E7DA112>I<800040002000100008000C000600060003 00030001800180018001C000C000C000C000E000E000E000E000E000E000E000E000E000E000E0 00E000C000C000C001C001800180018003000300060006000C00080010002000400080000B2E7D A112>I<0006000000060000000600000006000000060000000600000006000000060000000600 00000600000006000000060000000600000006000000060000FFFFFFF0FFFFFFF0000600000006 000000060000000600000006000000060000000600000006000000060000000600000006000000 0600000006000000060000000600001C207D9A23>43 D<70F8FCFC74040404080810102040060E 7C840D>II<70F8F8F87005057C840D>I<000100030003000600060006 000C000C000C00180018001800300030003000600060006000C000C000C0018001800180030003 0003000600060006000C000C000C00180018001800300030003000600060006000C000C000C000 102D7DA117>I<03F0000E1C001C0E00180600380700700380700380700380700380F003C0F003 C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C07003807003 807003807807803807001806001C0E000E1C0003F000121F7E9D17>I<018003800F80F3800380 038003800380038003800380038003800380038003800380038003800380038003800380038003 8003800380038007C0FFFE0F1E7C9D17>I<03F0000C1C00100E00200700400780800780F007C0 F803C0F803C0F803C02007C00007C0000780000780000F00000E00001C00003800007000006000 00C0000180000300000600400C00401800401000803FFF807FFF80FFFF80121E7E9D17>I<03F0 000C1C00100E00200F00780F80780780780780380F80000F80000F00000F00000E00001C000038 0003F000003C00000E00000F000007800007800007C02007C0F807C0F807C0F807C0F007804007 80400F00200E001C3C0003F000121F7E9D17>I<000600000600000E00000E00001E00002E0000 2E00004E00008E00008E00010E00020E00020E00040E00080E00080E00100E00200E00200E0040 0E00C00E00FFFFF0000E00000E00000E00000E00000E00000E00000E0000FFE0141E7F9D17>I< 1803001FFE001FFC001FF8001FE00010000010000010000010000010000010000011F000161C00 180E001007001007800003800003800003C00003C00003C07003C0F003C0F003C0E00380400380 400700200600100E000C380003E000121F7E9D17>I<007C000182000701000E03800C07801C07 80380300380000780000700000700000F1F000F21C00F40600F80700F80380F80380F003C0F003 C0F003C0F003C0F003C07003C07003C07003803803803807001807000C0E00061C0001F000121F 7E9D17>I<4000007FFFC07FFF807FFF8040010080020080020080040000080000080000100000 200000200000400000400000C00000C00001C00001800003800003800003800003800007800007 8000078000078000078000078000078000030000121F7D9D17>I<03F0000C0C00100600300300 2001806001806001806001807001807803003E03003F06001FC8000FF00003F80007FC000C7E00 103F00300F806003804001C0C001C0C000C0C000C0C000C0C000806001802001001002000C0C00 03F000121F7E9D17>I<03F0000E18001C0C00380600380700700700700380F00380F00380F003 C0F003C0F003C0F003C0F003C07007C07007C03807C0180BC00E13C003E3C00003800003800003 80000700300700780600780E00700C002018001070000FC000121F7E9D17>I<70F8F8F8700000 000000000000000070F8F8F87005147C930D>I<70F8F8F8700000000000000000000070F0F8F8 78080808101010202040051D7C930D>I<7FFFFFE0FFFFFFF00000000000000000000000000000 000000000000000000000000000000000000FFFFFFF07FFFFFE01C0C7D9023>61 D<000100000003800000038000000380000007C0000007C0000007C0000009E0000009E0000009 E0000010F0000010F0000010F00000207800002078000020780000403C0000403C0000403C0000 801E0000801E0000FFFE0001000F0001000F0001000F00020007800200078002000780040003C0 0E0003C01F0007E0FFC03FFE1F207F9F22>65 DI<000FC04000 7030C001C009C0038005C0070003C00E0001C01E0000C01C0000C03C0000C07C0000407C000040 78000040F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F80000 00780000007C0000407C0000403C0000401C0000401E0000800E000080070001000380020001C0 040000703800000FC0001A217D9F21>IIII<000FE0200078186000E004E0038002E0070001 E00F0000E01E0000601E0000603C0000603C0000207C00002078000020F8000000F8000000F800 0000F8000000F8000000F8000000F8000000F8007FFCF80003E0780001E07C0001E03C0001E03C 0001E01E0001E01E0001E00F0001E0070001E0038002E000E0046000781820000FE0001E217D9F 24>III<0FFFC0007C00003C00003C00003C00003C00003C00003C00003C00003C0000 3C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C0020 3C00F83C00F83C00F83C00F0380040780040700030E0000F800012207E9E17>IIIII<001F800000F0F00001C0380007801E000F000F000E0007001E000780 3C0003C03C0003C07C0003E0780001E0780001E0F80001F0F80001F0F80001F0F80001F0F80001 F0F80001F0F80001F0F80001F0F80001F0780001E07C0003E07C0003E03C0003C03C0003C01E00 07800E0007000F000F0007801E0001C0380000F0F000001F80001C217D9F23>II82 D<07E0800C1980100780300380 600180600180E00180E00080E00080E00080F00000F000007800007F00003FF0001FFC000FFE00 03FF00001F800007800003C00003C00001C08001C08001C08001C08001C0C00180C00380E00300 F00600CE0C0081F80012217D9F19>I<7FFFFFE0780F01E0600F0060400F0020400F0020C00F00 30800F0010800F0010800F0010800F0010000F0000000F0000000F0000000F0000000F0000000F 0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F000000 0F0000000F0000000F0000000F0000001F800007FFFE001C1F7E9E21>IIII<7FFFF87C00F87000 F06001E04001E0C003C0C003C0800780800F80800F00001E00001E00003C00003C0000780000F8 0000F00001E00001E00003C00403C0040780040F80040F000C1E000C1E00083C00183C00187800 38F801F8FFFFF8161F7D9E1C>90 DI<0804100820102010402040 20804080408040B85CFC7EFC7E7C3E381C0F0E7B9F17>II<1FE0 00303000781800781C00300E00000E00000E00000E0000FE00078E001E0E00380E00780E00F00E 10F00E10F00E10F01E10781E103867200F83C014147E9317>97 D<0E0000FE00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E3E000EC3800F01C00F00E00E00E0 0E00700E00700E00780E00780E00780E00780E00780E00780E00700E00700E00E00F00E00D01C0 0CC300083E0015207F9F19>I<03F80E0C1C1E381E380C70007000F000F000F000F000F000F000 70007000380138011C020E0C03F010147E9314>I<000380003F80000380000380000380000380 00038000038000038000038000038000038003E380061B801C0780380380380380700380700380 F00380F00380F00380F00380F00380F003807003807003803803803807801C07800E1B8003E3F8 15207E9F19>I<03F0000E1C001C0E00380700380700700700700380F00380F00380FFFF80F000 00F00000F000007000007000003800801800800C010007060001F80011147F9314>I<007C00C6 018F038F07060700070007000700070007000700FFF00700070007000700070007000700070007 000700070007000700070007000700070007007FF01020809F0E>I<0000E003E3300E3C301C1C 30380E00780F00780F00780F00780F00780F00380E001C1C001E380033E0002000002000003000 003000003FFE001FFF800FFFC03001E0600070C00030C00030C00030C000306000603000C01C03 8003FC00141F7F9417>I<0E0000FE00000E00000E00000E00000E00000E00000E00000E00000E 00000E00000E00000E3E000E43000E81800F01C00F01C00E01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0FFE7FC16207F9F19>I<1C001E 003E001E001C000000000000000000000000000E007E000E000E000E000E000E000E000E000E00 0E000E000E000E000E000E000E000E000E00FFC00A1F809E0C>I<00E001F001F001F000E00000 00000000000000000000007007F000F00070007000700070007000700070007000700070007000 700070007000700070007000700070007000706070F060F0C061803F000C28829E0E>I<0E0000 FE00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E0FF00E03C0 0E03000E02000E04000E08000E10000E30000E70000EF8000F38000E1C000E1E000E0E000E0700 0E07800E03800E03C00E03E0FFCFF815207F9F18>I<0E00FE000E000E000E000E000E000E000E 000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00 0E000E000E00FFE00B20809F0C>I<0E1F01F000FE618618000E81C81C000F00F00E000F00F00E 000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E0 0E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E00FFE7FE7FE02314 7F9326>I<0E3E00FE43000E81800F01C00F01C00E01C00E01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0FFE7FC16147F9319>I<01F800070E00 1C03803801C03801C07000E07000E0F000F0F000F0F000F0F000F0F000F0F000F07000E07000E0 3801C03801C01C0380070E0001F80014147F9317>I<0E3E00FEC3800F01C00F00E00E00E00E00 F00E00700E00780E00780E00780E00780E00780E00780E00700E00F00E00E00F01E00F01C00EC3 000E3E000E00000E00000E00000E00000E00000E00000E00000E0000FFE000151D7F9319>I<03 E0800619801C05803C0780380380780380700380F00380F00380F00380F00380F00380F0038070 03807803803803803807801C0B800E138003E38000038000038000038000038000038000038000 0380000380003FF8151D7E9318>I<0E78FE8C0F1E0F1E0F0C0E000E000E000E000E000E000E00 0E000E000E000E000E000E000E00FFE00F147F9312>I<1F9030704030C010C010C010E0007800 7F803FE00FF00070803880188018C018C018E030D0608F800D147E9312>I<0200020002000600 06000E000E003E00FFF80E000E000E000E000E000E000E000E000E000E000E000E080E080E080E 080E080610031001E00D1C7F9B12>I<0E01C0FE1FC00E01C00E01C00E01C00E01C00E01C00E01 C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E03C00603C0030DC001F1FC1614 7F9319>III<7FC3FC0F01E00701C007018003810001C20000E40000EC00007800003800003C00007C00 004E000087000107000303800201C00601E01E01E0FF07FE1714809318>II<3FFF380E200E201C40384078407000E001E001C00380078007010E011E011C 0338027006700EFFFE10147F9314>III<30 307878F87C787830300E057C9E17>127 D E /Fo 49 123 df<007E0001C1800301800703C00E 03C00E01800E00000E00000E00000E00000E0000FFFFC00E01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C07F87F8151D809C 17>12 D<003F07E00001C09C18000380F018000701F03C000E01E03C000E00E018000E00E00000 0E00E000000E00E000000E00E000000E00E00000FFFFFFFC000E00E01C000E00E01C000E00E01C 000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E0 1C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C007FC7FCFF80211D809C23>14 D<004000800100020006000C000C0018001800300030007000600060006000E000E000E000E000 E000E000E000E000E000E000E000E000600060006000700030003000180018000C000C00060002 000100008000400A2A7D9E10>40 D<800040002000100018000C000C0006000600030003000380 01800180018001C001C001C001C001C001C001C001C001C001C001C001C0018001800180038003 000300060006000C000C00180010002000400080000A2A7E9E10>I<60F0F07010101010202040 80040C7C830C>44 DI<60F0F06004047C830C>I<03C00C301818300C30 0C700E60066006E007E007E007E007E007E007E007E007E007E007E007E007E00760066006700E 300C300C18180C3007E0101D7E9B15>48 D<030007003F00C70007000700070007000700070007 000700070007000700070007000700070007000700070007000700070007000F80FFF80D1C7C9B 15>I<00F0030C06040C0E181E301E300C700070006000E3E0E430E818F00CF00EE006E007E007 E007E007E007600760077006300E300C18180C3003E0101D7E9B15>54 D<4000007FFF807FFF00 7FFF0040020080040080040080080000100000100000200000600000400000C00000C00001C000 018000018000038000038000038000038000078000078000078000078000078000078000030000 111D7E9B15>I<03C00C301818300C700C600EE006E006E007E007E007E007E0076007700F300F 18170C2707C700060006000E300C780C78187010203030C00F80101D7E9B15>57 D<60F0F0600000000000000000000060F0F06004127C910C>I<00060000000600000006000000 0F0000000F0000000F00000017800000178000001780000023C0000023C0000023C0000041E000 0041E0000041E0000080F0000080F0000180F8000100780001FFF80003007C0002003C0002003C 0006003E0004001E0004001E000C001F001E001F00FF80FFF01C1D7F9C1F>65 DI<001F808000E0618001801980070007800E0003801C0003801C 00018038000180780000807800008070000080F0000000F0000000F0000000F0000000F0000000 F0000000F0000000F0000000700000807800008078000080380000801C0001001C0001000E0002 00070004000180080000E03000001FC000191E7E9C1E>I69 D73 D76 DI<003F800000E0E000 0380380007001C000E000E001C0007003C00078038000380780003C0780003C0700001C0F00001 E0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0700001C0780003C07800 03C0380003803C0007801C0007000E000E0007001C000380380000E0E000003F80001B1E7E9C20 >79 DI82 D<07E0801C1980300580700380600180E0 0180E00080E00080E00080F00000F800007C00007FC0003FF8001FFE0007FF0000FF80000F8000 07C00003C00001C08001C08001C08001C0C00180C00180E00300D00200CC0C0083F800121E7E9C 17>I87 D<1FC000307000783800781C00301C00001C00001C0001FC000F1C00381C00701C00601C00E01C 40E01C40E01C40603C40304E801F870012127E9115>97 D I<07E00C301878307870306000E000E000E000E000E000E00060007004300418080C3007C00E12 7E9112>I<003F0000070000070000070000070000070000070000070000070000070000070003 E7000C1700180F00300700700700600700E00700E00700E00700E00700E00700E0070060070070 0700300700180F000C370007C7E0131D7E9C17>I<03E00C301818300C700E6006E006FFFEE000 E000E000E00060007002300218040C1803E00F127F9112>I<00F8018C071E061E0E0C0E000E00 0E000E000E000E00FFE00E000E000E000E000E000E000E000E000E000E000E000E000E000E000E 000E007FE00F1D809C0D>I<00038003C4C00C38C01C3880181800381C00381C00381C00381C00 1818001C38000C300013C0001000003000001800001FF8001FFF001FFF803003806001C0C000C0 C000C0C000C06001803003001C0E0007F800121C7F9215>II<18003C003C0018000000000000000000000000000000FC001C001C001C001C001C001C001C 001C001C001C001C001C001C001C001C001C00FF80091D7F9C0C>I108 DII<03F0000E1C00180600300300700380600180E001 C0E001C0E001C0E001C0E001C0E001C06001807003803003001806000E1C0003F00012127F9115 >II<03C1000C3300180B00300F00700700700700E00700E00700E00700E00700 E00700E00700600700700700300F00180F000C370007C700000700000700000700000700000700 000700000700003FE0131A7E9116>II<1F9030704030C010C010E010F8007F803FE00F F000F880388018C018C018E010D0608FC00D127F9110>I<04000400040004000C000C001C003C 00FFE01C001C001C001C001C001C001C001C001C001C101C101C101C101C100C100E2003C00C1A 7F9910>IIII<7F8FF00F03800F030007020003840001C80001D80000F00000700000 780000F800009C00010E00020E000607000403801E07C0FF0FF81512809116>I I<7FFC70386038407040F040E041C003C0038007000F040E041C043C0C380870087038FFF80E12 7F9112>I E /Fp 7 117 df<00038000000380000007C0000007C0000007C000000FE000000FE0 00001FF000001BF000001BF0000031F8000031F8000061FC000060FC0000E0FE0000C07E0000C0 7E0001803F0001FFFF0003FFFF8003001F8003001F8006000FC006000FC00E000FE00C0007E0FF C07FFEFFC07FFE1F1C7E9B24>65 D<0FF8001C1E003E0F803E07803E07C01C07C00007C0007FC0 07E7C01F07C03C07C07C07C0F807C0F807C0F807C0780BC03E13F80FE1F815127F9117>97 DI<03FC000E0E001C1F003C1F00781F00780E00F80000F8 0000F80000F80000F80000F800007800007801803C01801C03000E0E0003F80011127E9115>I< FE3E00FE47001E8F801E8F801E8F801F07001F00001F00001F00001F00001F00001F00001F0000 1F00001F00001F0000FFF000FFF00011127F9114>114 D<1FD830786018E018E018F000FF807F E07FF01FF807FC007CC01CC01CE01CE018F830CFC00E127E9113>I<0300030003000300070007 000F000F003FFCFFFC1F001F001F001F001F001F001F001F001F001F0C1F0C1F0C1F0C0F080798 03F00E1A7F9913>I E /Fq 3 122 df0 D<020002000200C218F2 783AE00F800F803AE0F278C2180200020002000D0E7E8E12>3 D<060006000600060006000600 06000600FFF0FFF006000600060006000600060006000600060006000600060006000600060006 000600060006000C1D7E9611>121 D E /Fr 51 122 df<70F8FCFC7404040404080810102040 060F7C840E>44 DI<70F8F8F87005057C840E>I<01F000071C000C0600 1803003803803803807001C07001C07001C07001C0F001E0F001E0F001E0F001E0F001E0F001E0 F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001E07001C07001C07001C07803C0380380 3803801C07000C0600071C0001F00013227EA018>48 D<008003800F80F3800380038003800380 038003800380038003800380038003800380038003800380038003800380038003800380038003 8003800380038007C0FFFE0F217CA018>I<03F0000C1C001007002007804003C04003C08003E0 F003E0F801E0F801E0F801E02003E00003E00003C00003C0000780000700000E00001C00001800 00300000600000C0000180000100000200200400200800201800603000403FFFC07FFFC0FFFFC0 13217EA018>I<1000801E07001FFF001FFE001FF80013E0001000001000001000001000001000 0010000010F800130E001407001803801003800001C00001C00001E00001E00001E00001E07001 E0F001E0F001E0E001C08001C04003C04003802007001006000C1C0003F00013227EA018>53 D<007E0001C1000300800601C00E03C01C03C0180180380000380000780000700000700000F0F8 00F30C00F40600F40300F80380F801C0F001C0F001E0F001E0F001E0F001E0F001E07001E07001 E07001E03801C03801C01803801C03000C0600070C0001F00013227EA018>I<01F800060E0008 03001001802001802000C06000C06000C06000C07000C07801803E01003F02001FC4000FF80003 F80003FC00067F00083F80100F803007C06001C06000E0C000E0C00060C00060C00060C0006060 00406000C03000801803000E0E0003F00013227EA018>56 D<01F000060C000C06001807003803 80700380700380F001C0F001C0F001C0F001E0F001E0F001E0F001E0F001E07001E07003E03803 E01805E00C05E00619E003E1E00001C00001C00001C0000380000380300300780700780600700C 002018001030000FC00013227EA018>I<0001800000018000000180000003C0000003C0000003 C0000005E0000005E000000DF0000008F0000008F0000010F800001078000010780000203C0000 203C0000203C0000401E0000401E0000401E0000800F0000800F0000FFFF000100078001000780 030007C0020003C0020003C0040003E0040001E0040001E00C0000F00C0000F03E0001F8FF800F FF20237EA225>65 DI<0007E010 0038183000E0063001C00170038000F0070000F00E0000701E0000701C0000303C0000303C0000 307C0000107800001078000010F8000000F8000000F8000000F8000000F8000000F8000000F800 0000F800000078000000780000107C0000103C0000103C0000101C0000201E0000200E00004007 0000400380008001C0010000E0020000381C000007E0001C247DA223>III72 DI<03FF F0001F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F 00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00700F 00F80F00F80F00F80E00F01E00401C0020380018700007C00014237EA119>II77 DI<000FE00000783C0000E00E00 03C00780078003C00F0001E00E0000E01E0000F03C0000783C0000787C00007C7C00007C780000 3C7800003CF800003EF800003EF800003EF800003EF800003EF800003EF800003EF800003EF800 003E7800003C7C00007C7C00007C3C0000783E0000F81E0000F00F0001E00F0001E0078003C003 C0078000E00E0000783C00000FE0001F247DA226>II82 D<03F0200C0C601802603001E07000E0600060E00060E00060E00020E00020E00020F00000F000 007800007F00003FF0001FFE000FFF0003FF80003FC00007E00001E00000F00000F00000708000 70800070800070800070C00060C00060E000C0F000C0C80180C6070081FC0014247DA21B>I<7F FFFFF87807807860078018400780084007800840078008C007800C800780048007800480078004 800780040007800000078000000780000007800000078000000780000007800000078000000780 000007800000078000000780000007800000078000000780000007800000078000000780000007 80000007800000078000000FC00003FFFF001E227EA123>I86 DI89 D<0FE0001838003C0C003C0E00180700000700000700 00070000FF0007C7001E07003C0700780700700700F00708F00708F00708F00F087817083C2390 0FC1E015157E9418>97 D<0E0000FE00001E00000E00000E00000E00000E00000E00000E00000E 00000E00000E00000E00000E00000E1F000E61C00E80600F00300E00380E003C0E001C0E001E0E 001E0E001E0E001E0E001E0E001E0E001E0E001C0E003C0E00380F00700C80600C41C0083F0017 237FA21B>I<01FE000703000C07801C0780380300780000700000F00000F00000F00000F00000 F00000F00000F000007000007800403800401C00800C010007060001F80012157E9416>I<0000 E0000FE00001E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000 E001F8E00704E00C02E01C01E03800E07800E07000E0F000E0F000E0F000E0F000E0F000E0F000 E0F000E07000E07800E03800E01801E00C02E0070CF001F0FE17237EA21B>I<01FC000707000C 03801C01C03801C07801E07000E0F000E0FFFFE0F00000F00000F00000F00000F0000070000078 00203800201C00400E008007030000FC0013157F9416>I<00007001F198071E180E0E181C0700 1C07003C07803C07803C07803C07801C07001C07000E0E000F1C0019F000100000100000180000 1800001FFE000FFFC00FFFE03800F0600030400018C00018C00018C000186000306000303800E0 0E038003FE0015217F9518>103 D<0E0000FE00001E00000E00000E00000E00000E00000E0000 0E00000E00000E00000E00000E00000E00000E1F800E60C00E80E00F00700F00700E00700E0070 0E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E0070 FFE7FF18237FA21B>I<1C001E003E001E001C00000000000000000000000000000000000E00FE 001E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00FFC0 0A227FA10E>I<0E0000FE00001E00000E00000E00000E00000E00000E00000E00000E00000E00 000E00000E00000E00000E03FC0E01F00E01C00E01800E02000E04000E08000E10000E38000EF8 000F1C000E1E000E0E000E07000E07800E03C00E01C00E01E00E00F00E00F8FFE3FE17237FA21A >107 D<0E00FE001E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E 000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00FFE00B237FA2 0E>I<0E1FC07F00FE60E183801E807201C00F003C00E00F003C00E00E003800E00E003800E00E 003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E0 0E003800E00E003800E00E003800E00E003800E00E003800E0FFE3FF8FFE27157F942A>I<0E1F 80FE60C01E80E00F00700F00700E00700E00700E00700E00700E00700E00700E00700E00700E00 700E00700E00700E00700E00700E00700E0070FFE7FF18157F941B>I<01FC000707000C018018 00C03800E0700070700070F00078F00078F00078F00078F00078F00078F000787000707800F038 00E01C01C00E038007070001FC0015157F9418>I<0E1F00FE61C00E80600F00700E00380E003C 0E001C0E001E0E001E0E001E0E001E0E001E0E001E0E001E0E003C0E003C0E00380F00700E80E0 0E41C00E3F000E00000E00000E00000E00000E00000E00000E00000E00000E0000FFE000171F7F 941B>I<0E3CFE461E8F0F0F0F060F000E000E000E000E000E000E000E000E000E000E000E000E 000E000F00FFF010157F9413>114 D<0F8830786018C018C008C008E008F0007F803FE00FF001 F8003C801C800C800CC00CC008E018D0308FC00E157E9413>I<02000200020002000600060006 000E001E003E00FFF80E000E000E000E000E000E000E000E000E000E000E000E040E040E040E04 0E040E040708030801F00E1F7F9E13>I<0E0070FE07F01E00F00E00700E00700E00700E00700E 00700E00700E00700E00700E00700E00700E00700E00700E00700E00F00E00F006017003827800 FC7F18157F941B>I IIII E /Fs 25 122 df<78FCFCFCFC780000000000000000000000000000000000000078FCFCFCFC78061F7A 9E12>58 D<00000300000000000300000000000300000000000780000000000780000000000FC0 000000000FC0000000000FC00000000017E00000000013E00000000013E00000000023F0000000 0021F00000000021F00000000040F80000000040F80000000040F800000000807C00000000807C 00000001807E00000001003E00000001003E00000002003F00000002001F00000002001F000000 04000F80000004000F80000004000F800000080007C00000080007C00000180007E000001FFFFF E000001FFFFFE00000200003F00000200001F00000200001F00000400001F80000400000F80000 400000F800008000007C00008000007C00008000007C00010000003E00010000003E0003000000 3F00030000001F00070000001F001F8000003F80FFE00003FFFCFFE00003FFFC2E327EB132>65 D<00003FE0010001FFF8030007F01E03001F800307003E000087007800004F00F000002F01E000 001F03C000000F078000000F0F800000070F000000071F000000031E000000033E000000033C00 0000017C000000017C000000017C000000017800000000F800000000F800000000F800000000F8 00000000F800000000F800000000F800000000F800000000F800000000F800000000F800000000 78000000007C000000007C000000017C000000013C000000013E000000011E000000011F000000 020F000000020F80000006078000000403C000000801E000000800F00000100078000020003E00 00C0001F8003800007F00F000001FFFC0000003FE00028337CB130>67 D76 D80 D<007F802001FFE02007C0 78600F001C601E0006E03C0003E0380001E0780000E0700000E070000060F0000060F0000060F0 000020F0000020F0000020F8000020F80000007C0000007E0000003F0000003FC000001FF80000 0FFF800007FFF80003FFFC0000FFFF00000FFF800000FFC000001FE0000007E0000003F0000001 F0000000F0000000F8000000F88000007880000078800000788000007880000078C0000078C000 0070E00000F0E00000E0F00000E0F80001C0EC000380C7000700C1F01E00807FFC00800FF0001D 337CB125>83 D<7FFFFFFFFFE07FFFFFFFFFE07E000F8007E078000F8001E070000F8000E06000 0F80006040000F80002040000F800020C0000F800030C0000F80003080000F80001080000F8000 1080000F80001080000F80001080000F80001080000F80001000000F80000000000F8000000000 0F80000000000F80000000000F80000000000F80000000000F80000000000F80000000000F8000 0000000F80000000000F80000000000F80000000000F80000000000F80000000000F8000000000 0F80000000000F80000000000F80000000000F80000000000F80000000000F80000000000F8000 0000000F80000000000F80000000000F80000000000F80000000000F80000000000F8000000000 0F80000000000F80000000001FC00000000FFFFF8000000FFFFF80002C317EB030>I<00FE0000 0303C0000C00E00010007000100038003C003C003E001C003E001E003E001E0008001E0000001E 0000001E0000001E00000FFE0000FC1E0003E01E000F801E001F001E003E001E003C001E007C00 1E00F8001E04F8001E04F8001E04F8003E04F8003E0478003E047C005E043E008F080F0307F003 FC03E01E1F7D9E21>97 D<07800000FF800000FF8000000F800000078000000780000007800000 078000000780000007800000078000000780000007800000078000000780000007800000078000 0007800000078000000781FC0007860700078801C0079000E007A0007007C00078078000380780 003C0780003C0780001E0780001E0780001F0780001F0780001F0780001F0780001F0780001F07 80001F0780001F0780001F0780001E0780003E0780003C0780003C0780007807C00070072000F0 072001E00618038006060F000401F80020327EB125>I<003F8000E0600380180700040F00041E 001E1C003E3C003E7C003E7C0008780000F80000F80000F80000F80000F80000F80000F80000F8 0000F800007800007C00007C00003C00011E00011E00020F000207000403801800E060003F8018 1F7D9E1D>I<000001E000003FE000003FE0000003E0000001E0000001E0000001E0000001E000 0001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0 000001E0001F81E000F061E001C019E0078005E00F0003E00E0003E01E0001E03C0001E03C0001 E07C0001E0780001E0F80001E0F80001E0F80001E0F80001E0F80001E0F80001E0F80001E0F800 01E0F80001E0780001E0780001E03C0001E03C0001E01C0001E01E0003E00E0005E0070009E003 8011F000E061FF003F81FF20327DB125>I<003F800000E0E0000380380007003C000E001E001E 001E001C000F003C000F007C000F0078000F8078000780F8000780F8000780FFFFFF80F8000000 F8000000F8000000F8000000F8000000F8000000780000007C0000003C0000003C0000801E0000 800E0001000F0002000780020001C00C0000F03000001FC000191F7E9E1D>I<0007E0001C1000 383800707C00E07C01E07C01C03803C00003C00003C00003C00003C00003C00003C00003C00003 C00003C00003C00003C000FFFFC0FFFFC003C00003C00003C00003C00003C00003C00003C00003 C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003 C00003C00003C00003C00003C00003C00007E0007FFF007FFF0016327FB114>I<07000F801F80 1F800F800700000000000000000000000000000000000000000000000780FF80FF800F80078007 800780078007800780078007800780078007800780078007800780078007800780078007800780 0780078007800FC0FFF8FFF80D307EAF12>105 D<0780FF80FF800F8007800780078007800780 078007800780078007800780078007800780078007800780078007800780078007800780078007 800780078007800780078007800780078007800780078007800780078007800780078007800FC0 FFFCFFFC0E327EB112>108 D<0780FE001FC000FF83078060F000FF8C03C18078000F9001E200 3C0007A001E4003C0007A000F4001E0007C000F8001E0007C000F8001E00078000F0001E000780 00F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E 00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000 F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00 078000F0001E00078000F0001E000FC001F8003F00FFFC1FFF83FFF0FFFC1FFF83FFF0341F7E9E 38>I<0780FE0000FF83078000FF8C03C0000F9001E00007A001E00007A000F00007C000F00007 C000F000078000F000078000F000078000F000078000F000078000F000078000F000078000F000 078000F000078000F000078000F000078000F000078000F000078000F000078000F000078000F0 00078000F000078000F000078000F000078000F000078000F0000FC001F800FFFC1FFF80FFFC1F FF80211F7E9E25>I<001FC00000F0780001C01C00070007000F0007801E0003C01C0001C03C00 01E03C0001E0780000F0780000F0780000F0F80000F8F80000F8F80000F8F80000F8F80000F8F8 0000F8F80000F8F80000F8780000F07C0001F03C0001E03C0001E01E0003C01E0003C00F000780 07800F0001C01C0000F07800001FC0001D1F7E9E21>I<0781FC00FF860700FF8803C00F9001E0 07A000F007C00078078000780780003C0780003C0780003E0780001E0780001F0780001F078000 1F0780001F0780001F0780001F0780001F0780001F0780001F0780003E0780003E0780003C0780 007C0780007807C000F007A000F007A001E00798038007860F000781F800078000000780000007 80000007800000078000000780000007800000078000000780000007800000078000000FC00000 FFFC0000FFFC0000202D7E9E25>I<0783E0FF8C18FF907C0F907C07A07C07C03807C00007C000 07C000078000078000078000078000078000078000078000078000078000078000078000078000 0780000780000780000780000780000780000780000FC000FFFE00FFFE00161F7E9E19>114 D<01FC100E03301800F0300070600030E00030E00010E00010E00010F00010F800007E00003FF0 001FFF000FFFC003FFE0003FF00001F80000F880003C80003C80001CC0001CC0001CE0001CE000 18F00038F00030CC0060C301C080FE00161F7E9E1A>I<00400000400000400000400000400000 C00000C00000C00001C00001C00003C00007C0000FC0001FFFE0FFFFE003C00003C00003C00003 C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003 C01003C01003C01003C01003C01003C01003C01003C01001C02001E02000E0400078C0001F0014 2C7FAB19>I<078000F000FF801FF000FF801FF0000F8001F000078000F000078000F000078000 F000078000F000078000F000078000F000078000F000078000F000078000F000078000F0000780 00F000078000F000078000F000078000F000078000F000078000F000078000F000078000F00007 8000F000078001F000078001F000078001F000038002F00003C004F00001C008F800007030FF80 001FC0FF80211F7E9E25>II121 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin %%EndSetup %%Page: 1 1 bop 384 339 a Fs(CCL:)21 b(A)h(P)n(ortable)g(and)f(T)-6 b(unable)21 b(Collectiv)n(e)160 431 y(Comm)n(unication)f(Library)i(for)f(Scalable)g(P)n (arallel)h(Computers)209 588 y Fr(V)l(asan)o(th)17 b(Bala)495 570 y Fq(y)571 588 y Fr(Jehosh)o(ua)g(Bruc)o(k)910 570 y Fq(\003)985 588 y Fr(Rob)q(ert)f(Cypher)1305 570 y Fq(\003)1381 588 y Fr(P)o(ablo)h (Elustondo)1733 570 y Fq(\003)348 647 y Fr(Alex)e(Ho)523 629 y Fq(\003)599 647 y Fr(Ching-Tien)i(Ho)918 629 y Fq(\003)994 647 y Fr(Shlomo)e(Kipnis)1306 629 y Fq(y)1382 647 y Fr(Marc)h(Snir)1595 629 y Fq(y)725 763 y Fr(IBM)f(Researc)o(h)g(Division)1218 745 y Fq(\003)694 821 y Fr(Almaden)f(Researc)o(h)i(Cen)o(ter)800 879 y(650)i(Harry)e(Road)759 937 y(San)h(Jose,)f(CA)g(95120)932 995 y(and)725 1054 y(IBM)f(Researc)o(h)g(Division)1218 1036 y Fq(y)657 1112 y Fr(T.J.)h(W)l(atson)h(Researc)o(h)f(Cen)o(ter)830 1170 y(P)l(.O.)f(Bo)o(x)h(218)660 1228 y(Y)l(orkto)o(wn)g(Heigh)o(ts,)f(NY)g (10598)880 1410 y Fp(Abstract)250 1504 y Fo(A)g(collectiv)o(e)f(comm)o (unication)e(library)h(for)i(parallel)e(computers)i(includes)g(frequen)o(tly) g(used)188 1554 y(op)q(erations)j(suc)o(h)h(as)f(broadcast,)i(reduce,)g (scatter,)h(gather,)e(concatenate,)h(sync)o(hronize,)g(and)188 1604 y(shift.)g(Suc)o(h)15 b(a)f(library)g(pro)o(vides)g(users)i(with)f(a)f (con)o(v)o(enien)o(t)h(programmi)o(ng)d(in)o(terface,)i(e\016cien)o(t)188 1653 y(comm)o(unicatio)o(n)e(op)q(erations,)j(and)f(the)i(adv)n(an)o(tage)e (of)g(p)q(ortabilit)o(y)m(.)19 b(A)c(library)f(of)g(this)h(nature,)188 1703 y(the)e(Collectiv)o(e)g(Comm)o(unicati)o(on)d(Library)j(\(CCL\),)g(in)o (tended)h(for)e(the)i(line)f(of)f(scalable)h(parallel)188 1753 y(computer)g(pro)q(ducts)h(b)o(y)f(IBM,)g(has)g(b)q(een)h(designed.)k(CCL)13 b(is)g(part)g(of)g(the)g(parallel)f(application)188 1803 y(programmi)o(ng)g (in)o(terface)k(of)e(the)i(recen)o(tly)g(announced)f(IBM)h(9076)e(Scalable)g (PO)o(WERparallel)188 1853 y(System)j(1)h(\(SP1\).)30 b(In)18 b(this)g(pap)q(er,)h(w)o(e)f(examine)f(sev)o(eral)h(issues)h(related)g(to)f (the)g(functional-)188 1902 y(it)o(y)m(,)13 b(correctness,)k(and)e(p)q (erformance)f(of)g(a)h(p)q(ortable)g(collectiv)o(e)f(comm)o(unication)d (library)j(while)188 1952 y(fo)q(cusing)i(on)f(three)j(no)o(v)o(el)d(asp)q (ects)j(in)d(the)i(design)f(and)g(implemen)o(tatio)o(n)d(of)j(CCL:)f(\(i\))h (the)g(in-)188 2002 y(tro)q(duction)g(of)e(pro)q(cess)k(groups,)d(\(ii\))g (the)h(de\014nition)f(of)g(seman)o(tics)g(that)h(ensures)h(correctness,)188 2052 y(and)e(\(iii\))g(the)h(design)g(of)f(new)h(and)f(tunable)h(algorithms)d (based)j(on)f(a)h(realistic)f(p)q(oin)o(t-to-p)q(oin)o(t)188 2102 y(comm)o(unicatio)o(n)c(mo)q(del.)960 2727 y Fn(1)p eop %%Page: 2 2 bop 74 157 a Fm(1)69 b(In)n(tro)r(duction)74 290 y Fn(The)13 b(need)h(for)f(collectiv)o(e)i(comm)o(unication)f(arises)f(frequen)o(tly)g (in)h(parallel)h(computation.)k(Collectiv)o(e)74 346 y(comm)o(unication)g(op) q(erations)f(simplify)i(the)e(programming)f(of)h(applications)h(for)f (parallel)h(comput-)74 403 y(ers,)c(facilitate)h(the)f(implemen)o(tation)h (of)e(e\016cien)o(t)i(comm)o(unication)g(sc)o(hemes)f(on)g(v)m(arious)g(mac)o (hines,)74 459 y(promote)d(the)g(p)q(ortabilit)o(y)i(of)e(applications)i (across)e(di\013eren)o(t)g(arc)o(hitectures,)h(and)g(re\015ect)f(conceptual) 74 516 y(grouping)k(of)f(pro)q(cesses.)20 b(In)c(particular,)g(collectiv)o(e) h(comm)o(unication)f(is)g(extensiv)o(ely)h(used)f(in)g(man)o(y)74 572 y(scien)o(ti\014c)f(applications)h(for)c(whic)o(h)j(the)f(in)o(terlea)o (ving)g(of)f(stages)g(of)g(lo)q(cal)i(computations)e(with)h(stages)74 629 y(of)h(global)h(comm)o(unication)g(is)f(p)q(ossible)i(\(see)f([20)o(]\).) 74 716 y(This)21 b(pap)q(er)g(discusses)g(issues)g(related)f(to)g(the)g (design)h(and)g(implemen)o(tation)g(of)f(a)g(p)q(ortable)g(and)74 773 y(tunable)15 b(Collectiv)o(e)g(Comm)o(unication)f(Library)g(\(CCL\).)f (This)h(library)h(is)f(in)o(tended)h(for)e(distributed-)74 829 y(memory)h(parallel)j(computers)e(where)g(explicit)i(comm)o(unication)f (among)e(pro)q(cesses)h(is)g(ac)o(hiev)o(ed)h(via)74 885 y(message-passing.) 32 b(In)20 b(addition)g(to)f(p)q(oin)o(t)h(to)e(p)q(oin)o(t)i(comm)o (unication)g(op)q(erations,)g(suc)o(h)f(as)g Fl(send)74 942 y Fn(and)i Fl(receive)p Fn(,)f(users)g(can)h(program)e(using)i(CCL)g (routines)g(for)e(collectiv)o(e)k(op)q(erations.)35 b(The)21 b(set)74 998 y(of)e(collectiv)o(e)h(comm)o(unication)g(op)q(erations)f(in)h (CCL)e(consists)i(of)e(the)h(follo)o(wing)h(routines:)27 b Fl(bcast)74 1055 y Fn(|)21 b(one-to-all)g(broadcast,)f Fl(reduce)f Fn(|)i(all-to-one)g(reduction,)h Fl(combine)d Fn(|)h(all-to-all)i(reduction,) 74 1111 y Fl(scatter)13 b Fn(|)g(one-to-all)i(p)q(ersonalized)g(comm)o (unication,)f Fl(gather)f Fn(|)h(all-to-one)g(p)q(ersonalized)i(com-)74 1168 y(m)o(unication,)c Fl(concat)d Fn(|)i(all-to-all)g(broadcast,)g Fl(index)e Fn(|)i(all-to-all)g(p)q(ersonalized)h(comm)o(unication\),)74 1224 y Fl(prefix)j Fn(|)i(scanned)g(reduction,)g Fl(shift)e Fn(|)i(one-to-one)f(cyclic)i(p)q(erm)o(utation,)e(and)g Fl(sync)p Fn(|barrier)74 1281 y(sync)o(hronization.)29 b(A)18 b(complete)h(listing)h (with)e(brief)h(descriptions)g(of)f(the)g(functionalit)o(y)h(of)e(all)i(the) 74 1337 y(CCL)c(routines)h(is)g(pro)o(vided)g(in)g(the)f(App)q(endix.)74 1425 y(The)j(library)g(w)o(as)f(designed)i(for)e(the)h(new)f(IBM)h(line)h(of) f(scalable)g(parallel)i(computers.)26 b(\(The)18 b(\014rst)74 1481 y(computer)c(of)g(this)g(line)i(of)e(pro)q(ducts,)g(the)g(IBM)h(9076)d (Scalable)k(PO)o(WERparallel)g(System)e(1)g(\(SP1\),)74 1538 y(has)j(b)q(een)h(announced)f(recen)o(tly)l(.\))25 b(Most)16 b(of)g(CCL)h(is)h(implemen)o(ted)g(as)e(part)g(of)h(the)g(parallel)h(appli-) 74 1594 y(cation)e(programming)e(in)o(terface)i(of)e(SP1.)74 1682 y(A)f(ma)s(jor)f(goal)h(in)h(the)f(design)i(of)d(CCL)i(w)o(as)e(the)h (creation)h(of)e(a)h(truly)h(p)q(ortable)f(library)h(that)f(can)g(run)74 1738 y(correctly)h(and)f(e\016cien)o(tly)i(on)e(a)h(wide)g(range)f(of)g (parallel)i(and)e(distributed)i(computer)f(arc)o(hitectures,)74 1795 y(regardless)i(of)f(the)g(top)q(ology)g(of)g(the)h(in)o(terconnection)h (net)o(w)o(ork)d(and)h(the)h(sp)q(eeds)g(of)f(the)h(pro)q(cessors)74 1851 y(and)e(the)g(comm)o(unication)h(net)o(w)o(ork.)k(Also,)14 b(CCL)g(can)g(b)q(e)h(easily)g(implemen)o(ted)h(on)e(top)f(of)h(message-)74 1907 y(passing)k(libraries)g(suc)o(h)f(as)g(IBM)g(EUI)g([21],)f(IBM)h(VIPER)h (Op)q(erating)g(System)f([18)o(,)f(23],)g(P)o(arasoft)74 1964 y(Express)f([30)o(],)g(and)g(ONRL)i(PVM)e([6)o(].)74 2052 y(F)l(or)d (simplicit)o(y)j(and)e(clarit)o(y)g(of)f(the)g(discussion)j(in)e(this)g(pap)q (er,)g(w)o(e)g(only)g(address)f(scenarios)h(in)h(whic)o(h)74 2108 y(dynamic)23 b(pro)q(cess)f(creation)g(or)g(deletion)h(in)g(run-time)f (is)h(not)e(p)q(ermitted.)41 b(The)22 b(CCL)g(routines)74 2164 y(can)f(either)h(op)q(erate)e(o)o(v)o(er)g(the)h(en)o(tire)g(set)g(of)f(pro)q (cesses)i(that)e(are)g(created)h(at)f(the)h(b)q(eginning)i(of)74 2221 y(an)d(application)i(or)d(o)o(v)o(er)g(user-sp)q(eci\014ed)j(groups)e (of)f(pro)q(cesses.)35 b(F)l(or)19 b(example,)i(a)f(user)g(can)g(view)74 2277 y(sev)o(eral)12 b(pro)q(cesses)f(as)g(forming)g(a)g(t)o(w)o (o-dimensional)h(arra)o(y)e(and)i(p)q(erforming)g(indep)q(enden)o(t)h (broadcast)74 2334 y(op)q(erations)i(in)g(di\013eren)o(t)g(columns)h(of)e (this)h(arra)o(y)l(.)k(Moreo)o(v)o(er,)13 b(the)i(CCL)f(routines)h(can)g (also)g(op)q(erate)74 2390 y(on)g(groups)f(of)h(pro)q(cesses)g(that)f(are)h (determined)h(dynamically)l(,)g(namely)l(,)g(in)f(run-time.)21 b(T)l(o)15 b(facilitate)74 2447 y(the)d(use)g(of)f(suc)o(h)h(groups)f(of)g (pro)q(cesses,)h(CCL)g(includes)h(routines)f(that)f(enable)i(the)e (de\014nition)j(and)d(the)74 2503 y(handling)18 b(of)e(dynamic)h(Pro)q(cess)f (Groups.)22 b(Section)17 b(2)f(examines)h(issues)g(related)g(to)e(pro)q(cess) i(groups)74 2560 y(in)f(CCL.)960 2727 y(2)p eop %%Page: 3 3 bop 74 157 a Fn(CCL)19 b(is)h(a)f(soft)o(w)o(are)e(la)o(y)o(er)i(that)g(can)g (sit)h(on)f(top)g(of)f(p)q(oin)o(t-to-p)q(oin)o(t)i(comm)o(unication.)33 b(A)19 b(crucial)74 214 y(step)g(in)h(the)f(design)h(of)f(CCL)g(w)o(as)f(to)g (de\014ne)i(a)f(simple)i(and)e(realistic)h(mo)q(del)g(for)e(the)i(underlying) 74 270 y(p)q(oin)o(t-to-p)q(oin)o(t)f(comm)o(unication.)28 b(This)19 b(allo)o(ws)f(CCL)f(to)h(b)q(e)g(b)q(oth)g(p)q(ortable)g(and)g (e\016cien)o(t)h(o)o(v)o(er)e(a)74 327 y(wide)c(range)f(of)g(parallel)i(mac)o (hines.)20 b(This)13 b(mo)q(del)g(is)g(discussed)h(in)f(Section)g(3.)19 b(In)12 b(particular,)i(Section)74 383 y(3)f(examines)h(sev)o(eral)g (fundamen)o(tal)f(issues)i(related)e(to)g(the)g(seman)o(tics)h(of)f (collectiv)o(e)i(comm)o(unication)74 439 y(op)q(erations)g(o)o(v)o(er)g(Pro)q (cess)g(Groups.)74 527 y(F)l(or)g(CCL)g(to)f(b)q(e)i(scalable)g(and)f (e\016cien)o(t)h(on)f(a)g(v)m(ariet)o(y)g(of)g(parallel)h(mac)o(hines,)g(it)f (is)h(imp)q(ortan)o(t)e(that)74 583 y(the)19 b(algorithms)g(designed)i(for)d (collectiv)o(e)j(comm)o(unication)e(op)q(erations)h(are)e(tunable)i(and)f (exploit)74 640 y(the)h(c)o(haracteristics)h(of)e(the)h(particular)h (parallel)h(computer)e(in)h(a)e(w)o(a)o(y)g(that)h(is)g(hidden)i(from)d(the) 74 696 y(user.)h(In)c(Section)g(4,)e(w)o(e)h(address)g(issues)h(related)g(to) e(the)h(design)i(of)d(tunable)i(algorithms)f(for)g(collec-)74 753 y(tiv)o(e)j(comm)o(unication)g(op)q(erations)g(in)g(CCL.)f(Finally)l(,)j (in)e(Section)g(5)g(w)o(e)f(presen)o(t)g(some)g(concluding)74 809 y(remarks.)74 982 y Fm(2)69 b(Pro)r(cess)23 b(Groups)h(in)e(CCL)74 1115 y Fn(Pro)q(cess)12 b(Groups)f(is)h(a)g(mec)o(hanism)g(for)f(grouping)h (pro)q(cesses)g(in)o(to)g(logical)h(sets)e(and)h(for)f(manipulating)74 1171 y(and)19 b(comm)o(unicating)h(among)e(pro)q(cesses)i(in)f(suc)o(h)h (sets.)31 b(This)19 b(section)h(describ)q(es)g(the)f(concept)h(of)74 1228 y(Pro)q(cess)15 b(Groups)f(in)i(CCL,)e(whic)o(h)i(is)f(based)g(on)g(a)f (similar)i(concept)g(that)e(w)o(as)g(presen)o(ted)h(in)h([2)o(].)j(\(In)74 1284 y(the)c(IBM)h(EUI)f(do)q(cumen)o(tation)h([21)o(],)f(Pro)q(cess)g (Groups)f(are)h(called)i(T)l(ask)e(Groups.\))74 1435 y Fk(2.1)56 b(De\014ning)18 b(Pro)r(cess)g(Groups)74 1552 y Fn(A)13 b(Pro)q(cess)g(Group) g(is)h(an)f(ordered)g(set)g(of)g(pro)q(cesses)g(that)g(has)g(a)g(system-wide) g(unique)i(name.)k(It)13 b(can)74 1609 y(b)q(e)j(op)q(erated)e(up)q(on)i(as)e (a)g(single)j(ob)s(ject.)i(Eac)o(h)14 b(Pro)q(cess)h(Group)f(is)h(iden)o (ti\014ed)i(b)o(y)e(a)f(unique)j(Pro)q(cess)74 1665 y(Group)f(Iden)o (ti\014er)i(\(PGID\))d(in)i(an)f(application)i(program.)k(All)17 b(the)g(pro)q(cesses)f(that)g(are)g(created)g(at)74 1721 y(the)d (initialization)i(of)d(an)g(application)i(b)q(elong)g(to)d(the)i (prede\014ned)h(Pro)q(cess)e(Group)g Fl(ALL)p Fn(.)g(In)h(addition,)74 1778 y(eac)o(h)i(pro)q(cess)h(has)f(a)f(unique)j(Pro)q(cess)e(Iden)o (ti\014er)h(\(PID\))f(named)g Fl(mypid)g Fn(whic)o(h)h(could)g(b)q(e)f(iden)o (ti\014ed)74 1834 y(with)h(the)f(singleton)h(set)f(consisting)h(only)g(of)f (that)f(pro)q(cess.)74 1922 y(There)e(are)f(t)o(w)o(o)f(mec)o(hanisms)i(in)h (CCL)e(for)g(de\014ning)i(new)f(Pro)q(cess)f(Groups.)19 b(One)12 b(mec)o(hanism)g(can)g(b)q(e)74 1978 y(used)g(to)f(de\014ne)i(a)f(new)g(Pro)q (cess)f(Group)h(b)o(y)f(pro)o(viding)i(an)e(explicit)j(list)f(of)e(pro)q (cess)h(iden)o(ti\014ers)h(\(PIDs\))74 2035 y(that)h(comprise)h(the)g(new)g (Pro)q(cess)f(Group.)20 b(This)15 b(mec)o(hanism)g(requires)g(eac)o(h)g(mem)o (b)q(er)g(pro)q(cess)g(of)f(a)74 2091 y(to-b)q(e-formed)j(new)f(Pro)q(cess)g (Group)g(to)g(kno)o(w)g(the)g(PIDs)h(of)e(all)j(the)e(pro)q(cesses)h(that)e (are)h(mem)o(b)q(ers)74 2148 y(of)f(this)g(group.)k(Using)d(this)f(mec)o (hanism,)h(all)f(the)g(pro)q(cesses)h(that)e(comprise)h(a)g(new)g(Pro)q(cess) g(Group)74 2204 y(m)o(ust)g(call)h(the)f Fl(group\(\))g Fn(routine:)146 2290 y Fl(pgid)23 b(=)h(group\(size,)e(list,)h(label\);)74 2377 y Fn(The)18 b(usage)g(of)g Fl(label)f Fn(will)i(b)q(ecome)g(clear)f (later.)28 b(The)18 b(call)i(to)d Fl(group\(\))g Fn(m)o(ust)g(b)q(e)i(made)f (b)o(y)f(eac)o(h)74 2433 y(mem)o(b)q(er)d(pro)q(cess)f(of)g(the)g(to-b)q (e-formed)g(Pro)q(cess)g(Group,)g(and)h(eac)o(h)f(mem)o(b)q(er)g(pro)q(cess)h (m)o(ust)e(supply)74 2490 y(the)19 b(same)g(list)g(of)g(PIDs)g(and)g(in)h (the)f(same)f(order.)31 b(Pro)q(cesses)19 b(that)f(b)q(elong)i(to)e(the)h (new)h(Pro)q(cess)74 2546 y(Group)e(obtain)g(a)f(unique)i(PGID)f(for)f(this)h (group.)27 b(Pro)q(cesses)18 b(that)f(do)h(not)f(wish)i(to)e(b)q(e)h(part)f (of)h(a)74 2602 y(new)e(Pro)q(cess)f(Group)g(do)g(not)g(call)h(the)f Fl(group\(\))f Fn(routine.)960 2727 y(3)p eop %%Page: 4 4 bop 74 157 a Fn(The)17 b(other)f(mec)o(hanism)h(for)f(de\014ning)h(new)g(Pro) q(cess)f(Groups)g(in)i(CCL)e(is)h(b)o(y)f(partitioning)i(existing)74 214 y(Pro)q(cess)d(Groups)f(according)h(to)f(some)g(criteria.)20 b(This)15 b(mec)o(hanism)g(do)q(es)g(not)f(require)i(the)e(mem)o(b)q(ers)74 270 y(of)i(a)f(to-b)q(e-formed)i(Pro)q(cess)e(Group)h(to)f(kno)o(w)h(the)g (PIDs)g(of)g(all)h(the)f(pro)q(cesses)g(that)f(will)j(comprise)74 327 y(this)g(group.)28 b(When)18 b(partitioning)g(an)g(existing)h(Pro)q(cess) f(Group)f Fl(G)p Fn(,)g(sev)o(eral)h(new)g(Pro)q(cess)g(Groups)74 383 y(are)13 b(created)h(as)f(follo)o(ws.)20 b(Ev)o(ery)13 b(pro)q(cess)h(in)g(the)g(group)f Fl(G)g Fn(supplies)j(a)d(lo)q(cal)i(in)o (teger)f(v)m(alue)h(to)e Fl(myval)p Fn(.)74 439 y(All)h(the)e(pro)q(cesses)h (that)e(supply)j(the)e(same)g(v)m(alue)i(to)d Fl(myval)h Fn(are)g(made)g(mem) o(b)q(ers)g(of)g(the)h(same)f(newly)74 496 y(partitioned)j(Pro)q(cess)g (Group.)k(The)c(partitioning)g(of)f(an)h(existing)g(Pro)q(cess)g(Group)f(is)h (p)q(erformed)g(b)o(y)74 552 y(the)g Fl(partition\(\))f Fn(routine:)146 640 y Fl(pgid)23 b(=)h(partition\(G,)e(myval,)h(key\);)74 728 y Fn(The)c(call)g(to)e Fl(partition\(\))g Fn(m)o(ust)h(b)q(e)h(made)f(b)o(y)g (ev)o(ery)g(mem)o(b)q(er)g(of)g(the)h(existing)g(Pro)q(cess)f(Group)74 784 y Fl(G)p Fn(.)e(No)g(mem)o(b)q(er)h(of)f(the)g(existing)i(group)e Fl(G)g Fn(returns)g(from)g(this)h(call)h(un)o(til)f(all)g(pro)q(cesses)g(in)h (group)e Fl(G)74 841 y Fn(ha)o(v)o(e)i(made)h(the)g(call.)32 b(\(In)19 b(that)f(sense,)h(the)g(call)h(to)e Fl(partition\(\))f Fn(imp)q(oses)j(an)e(implicit)j(barrier)74 897 y(sync)o(hronization)h(on)e (the)h(mem)o(b)q(ers)g(of)f Fl(G)p Fn(\).)g(This)h(call)h(results)f(in)h (partitioning)f(the)g(mem)o(b)q(ers)g(of)74 954 y(group)16 b Fl(G)f Fn(in)o(to)h(as)f(man)o(y)g(new)h(groups)f(as)g(the)h(n)o(um)o(b)q (er)g(of)f(distinct)i(in)o(teger)f(v)m(alues)h(supplied)h(b)o(y)d(the)74 1010 y(mem)o(b)q(er)h(pro)q(cesses.)74 1098 y(A)e(pro)q(cess)g(can)g (participate)h(in)f(sev)o(eral)g Fl(group\(\))f Fn(or)h Fl(partition\(\))e Fn(calls)j(and,)f(th)o(us,)f(b)q(e)i(a)e(mem)o(b)q(er)74 1154 y(of)h(sev)o(eral)h(di\013eren)o(t)f(Pro)q(cess)g(Groups.)19 b(Eac)o(h)c(call)g(to)f Fl(group\(\))f Fn(or)h Fl(partition\(\))f Fn(ma)o(y)g(corresp)q(ond)74 1211 y(to)19 b(creating)h(Pro)q(cess)f(Groups)h (based)f(on)h(a)f(new)h(criterion.)34 b(Eac)o(h)19 b(suc)o(h)h(call,)i (therefore,)e(creates)74 1267 y(unique)d(PGIDs)e(for)f(the)i(resultan)o(t)f (groups.)74 1355 y(All)20 b(the)e(pro)q(cesses)g(that)g(b)q(elong)h(to)e(a)h (particular)h(Pro)q(cess)f(Group)g Fl(G)p Fn(,)f(regardless)h(of)g(whether)g (the)74 1411 y(group)e(w)o(as)e(created)i(b)o(y)g(calling)h Fl(group\(\))e Fn(or)g Fl(partition\(\))p Fn(,)f(are)i(rank)o(ed)f(from)g(0)h (to)f Fj(n)10 b Fi(\000)h Fn(1,)k(where)74 1467 y Fj(n)21 b Fn(is)h(the)f(size)h(of)f(the)g(group.)37 b(The)22 b(ranking)f(of)g(the)g (pro)q(cesses)g(in)h(a)f(Pro)q(cess)g(Group)g(that)f(w)o(as)74 1524 y(created)14 b(b)o(y)g(a)f(call)i(to)e Fl(group\(\))g Fn(is)h(the)g(same)f(as)h(the)g(order)f(of)h(the)f(PIDs)h(in)h(the)f(list)g (supplied)i(to)d(the)74 1580 y Fl(group\(\))i Fn(call.)24 b(The)16 b(ranking)g(of)g(the)g(pro)q(cesses)g(in)h(a)f(Pro)q(cess)g(Group)f(that)h(w) o(as)f(created)h(b)o(y)g(a)f(call)74 1637 y(to)j Fl(partition\(\))g Fn(is)h(determined)h(b)o(y)f(the)g(additional)i(parameter,)d Fl(key)p Fn(,)h(that)f(is)i(supplied)h(to)d(the)74 1693 y Fl(partition\(\))f Fn(call)i(as)f(an)g(input)h(argumen)o(t.)27 b(Ties)19 b(in)g(the)f(ranking)g (as)g(a)g(consequence)h(of)f(calling)74 1750 y Fl(partition\(\))c Fn(are)h(to)f(b)q(e)i(brok)o(en)f(b)o(y)g(pro)q(cesses')h(ranking)f(in)h(the) f(paren)o(t)g(group.)74 1837 y(F)l(or)i(example,)i(supp)q(ose)f(that)f(there) h(are)f(7)g(pro)q(cesses,)h(with)g(PID's)f(from)g(0)g(to)g(6)h(\(this)f(is)h (the)g Fl(ALL)74 1894 y Fn(group\).)27 b(F)l(or)17 b(con)o(v)o(enience,)i(w)o (e)f(refer)f(to)g(a)g(pro)q(cess)h(with)g(an)g(ev)o(en)g(\(resp.)27 b(o)q(dd\))18 b(PID)f(as)h(an)f Fh(even)74 1950 y Fn(\(resp.)34 b Fh(o)n(dd)p Fn(\))19 b(pro)q(cess.)34 b(Supp)q(ose)21 b(that)e(w)o(e)h(w)o (an)o(t)e(to)h(create)h(the)g(follo)o(wing)g(t)o(w)o(o)f(t)o(yp)q(es)h(of)f (groups)74 2007 y(while)i(main)o(taining)g(the)f(original)g(ordering:)29 b Fj(G)20 b Fn(in)g(eac)o(h)g(ev)o(en)g(pro)q(cess)g(is)g(de\014ned)h(as)e (the)h(group)74 2063 y(that)c(consists)h(of)f(all)h(ev)o(en)g(pro)q(cesses)g (and)g Fj(G)f Fn(in)h(eac)o(h)g(o)q(dd)g(pro)q(cess)f(is)h(de\014ned)h(as)e (the)h(group)f(that)74 2120 y(consists)g(of)f(all)h(o)q(dd)g(pro)q(cesses.)22 b(There)15 b(are)h(t)o(w)o(o)e(w)o(a)o(ys)g(to)h(do)h(it.)21 b(The)16 b(\014rst)f(\(the)g(static\))g(is)h(to)f(ha)o(v)o(e)74 2176 y(the)g(follo)o(wing)h(co)q(de:)146 2264 y Fl(EXAMPLE)23 b(1:)146 2320 y(if)g(\(pid_is_even\(\)\))f({)217 2377 y(label)i(=)f(EVEN;)217 2433 y(G)h(=)g(group\(4,)f([0,2,4,6],)f(label\);)146 2490 y(})146 2546 y(else)h({)217 2602 y(label)h(=)f(ODD;)960 2727 y Fn(4)p eop %%Page: 5 5 bop 217 157 a Fl(G)24 b(=)g(group\(3,)f([1,3,5],)f(label\);)146 214 y(})74 305 y Fn(The)16 b(second)f(approac)o(h)g(\(the)g(dynamic\))h(is)g (to)e(use)i Fl(partition)p Fn(.)i(The)e(co)q(de)f(in)i(this)e(case)g(is:)146 397 y Fl(G)23 b(=)h(partition\(ALL,)e(is_mypid_odd,)g(key\);)74 489 y Fn(Consider)d(another)e(example)i(where)f(all)h(pro)q(cesses)f(are)g (partitioned)g(in)o(to)g(t)o(w)o(o)f(groups)g(dep)q(ending)74 545 y(on)e(some)g(lo)q(cally)i(computed)f(v)m(alue)g(\(temp)q(erature\).)j (The)d(co)q(de)f(lo)q(oks)h(lik)o(e:)146 637 y Fl(EXAMPLE)23 b(2:)146 693 y(if)g(\(temperature)g(<)g(BOILING\))217 750 y(myval)h(=)f (COLD;)146 806 y(else)217 863 y(myval)h(=)f(HOT;)146 919 y(key)g(=)h (temperature;)146 976 y(G)f(=)h(partition\(ALL,)e(myval,)h(key\);)74 1067 y Fn(In)16 b(this)f(example,)h(eac)o(h)f(of)f(the)h(t)o(w)o(o)f(newly)i (formed)f(groups)f(is)i(sorted)e(according)i(to)e(the)h(ascending)74 1124 y(order)g(of)g Fl(key)p Fn(,)f(whic)o(h)i(w)o(as)f(assigned)g(the)h(in)o (teger)f(v)m(alue)i(of)d(temp)q(erature.)74 1211 y(CCL)j(also)g(pro)o(vides)g (utilit)o(y)i(routines)e(for)f(pro)q(cesses)i(to)e(determine)i(the)f(size)h (of)e(a)h(Pro)q(cess)g(Group)74 1268 y(\()p Fl(getsize\(G\))p Fn(\),)f(to)i(obtain)h(the)g(list)h(of)e(PIDs)g(in)i(a)e(giv)o(en)h(Pro)q (cess)g(Group)f(\()p Fl(getmembers\(G\))p Fn(\),)e(to)74 1324 y(query)f(the)f(rank)g(of)g(a)g(pro)q(cess)h(in)g(a)f(Pro)q(cess)h(Group)f (\()p Fl(getrank\(G,)22 b(pid\))p Fn(\),)13 b(and)i(to)e(query)i(the)f(PID)74 1381 y(of)j(a)g(pro)q(cess)h(with)f(a)h(giv)o(en)f(rank)g(in)i(a)e(Pro)q (cess)g(Group)g(\()p Fl(getpid\(G,)22 b(rank\))p Fn(\).)k(Finally)l(,)19 b(the)e(user)74 1437 y(can)d(call)i Fl(getlabel\(G\))c Fn(to)i(obtain)g(the)g Fl(label)g Fn(supplied)i(to)e Fl(group\(\))f Fn(and)h(the)g Fl(my)p 1550 1437 15 2 v 17 w(val)g Fn(supplied)i(to)74 1494 y Fl(partition\(\))p Fn(.)74 1646 y Fk(2.2)56 b(Using)18 b(Pro)r(cess)g (Groups)74 1763 y Fn(Once)f(established,)g(Pro)q(cess)f(Groups)f(allo)o(w)h (en)o(tire)g(collections)i(of)d(pro)q(cesses)h(to)f(b)q(e)i(iden)o(ti\014ed)g (and)74 1819 y(manipulated)k(in)f(a)g(single)g(call.)34 b(Note)20 b(that)e(when)i(di\013eren)o(t)g(actions)g(are)f(required)h(for)f(di\013eren) o(t)74 1876 y(disjoin)o(t)f(groups,)e(the)h(user)g(can)g(use)g Fl(label)f Fn(to)g(distinguish)j(groups.)24 b(F)l(or)16 b(example,)i(the)f (follo)o(wing)74 1932 y(co)q(de)f(ma)o(y)e(follo)o(w)i(the)f(co)q(de)h(of)f (Example)g(1)g(ab)q(o)o(v)o(e.)146 2024 y Fl(if)23 b(\(getlabel\(G\))g(==)g (EVEN\))g({)217 2080 y(code-segment-1;)146 2137 y(})146 2193 y(else)47 b({)217 2250 y(code-segment-2;)146 2306 y(})74 2398 y Fn(As)15 b(another)g(example,)h(the)f(follo)o(wing)h(co)q(de)g(ma)o(y)e (follo)o(w)i(the)f(co)q(de)h(of)e(Example)i(2)f(ab)q(o)o(v)o(e.)146 2490 y Fl(if)23 b(\(getlabel\(G\))g(==)g(COLD\))g({)217 2546 y(code-segment-1;)146 2602 y(})960 2727 y Fn(5)p eop %%Page: 6 6 bop 146 157 a Fl(else)23 b({)217 214 y(code-segment-2;)146 270 y(})74 365 y Fn(A)e(group-wide)h(transaction)f(of)f(particular)i(in)o (terest)f(is)h(collectiv)o(e)g(comm)o(unication)g(o)o(v)o(er)e(all)i(the)74 421 y(mem)o(b)q(ers)15 b(of)f(a)h(Pro)q(cess)f(Group.)19 b(F)l(or)c(example,) g(a)f(group-wide)i(reduction)f(op)q(eration,)g(whic)o(h)g(tak)o(es)74 478 y(the)g(PGID)g(of)g(a)g(Pro)q(cess)g(Group)g(as)g(a)g(parameter,)f(can)h (b)q(e)h(written)f(as:)146 573 y Fl(combine\(G,)22 b(rfunc\(\),)h(data,)g (result\);)74 668 y Fn(Suc)o(h)15 b(a)e(routine)i(could)g(implemen)o(t)g(a)e (reduction,)i(using)g(a)e(supplied)k(asso)q(ciativ)o(e)d(function)h Fl(rfunc\(\))74 724 y Fn(to)21 b(com)o(bine)i(data)e(from)g(the)h(mem)o(b)q (ers)g(of)g(Pro)q(cess)f(Group)h Fl(G)p Fn(,)f(and)h(return)g(the)g(\014nal)h (result)f(of)74 781 y(the)17 b(reduction)h(to)f(eac)o(h)g(mem)o(b)q(er)g(in)h (the)f(bu\013er)g Fl(result)p Fn(.)24 b(All)19 b(pro)q(cesses)e(in)h(the)f (group)g(mak)o(e)f(the)74 837 y(call)k(with)f(the)g(same)f(actual)h (parameters)f Fl(G)g Fn(and)h Fl(rfunc\(\))p Fn(.)29 b(Similarly)l(,)22 b(a)c(group-wide)h(broadcast)74 894 y(op)q(eration)d(can)f(b)q(e)h(written)f (as:)146 989 y Fl(bcast\(G,)23 b(orig,)g(data\);)74 1084 y Fn(This)14 b(routine)f(w)o(ould)g(implemen)o(t)h(a)f(broadcast)f(comm)o (unication)h(from)f(a)h(particular)g(mem)o(b)q(er)g(of)f(the)74 1140 y(Pro)q(cess)j(Group)f Fl(G)p Fn(,)f(whose)i(PID)f(is)h Fl(orig)p Fn(,)f(to)f(all)j(of)e(the)g(other)g(mem)o(b)q(ers)h(of)f Fl(G)p Fn(.)g(All)h(pro)q(cesses)g(in)g(the)74 1196 y(group)g(mak)o(e)g(the)g (call)h(with)g(the)f(same)g(actual)g(parameters)g Fl(G)g Fn(and)g Fl(orig)p Fn(.)74 1284 y(A)j(participating)h(pro)q(cess)f(can)h(pro)q(ceed)f (past)g(the)g(call)h(to)e(the)h(collectiv)o(e)i(comm)o(unication)f(imme-)74 1341 y(diately)i(after)e(it)h(has)f(\014nished)i(its)f(participation)h(in)g (it)e(\(ev)o(en)h(though)g(the)f(comm)o(unication)i(as)e(a)74 1397 y(whole)e(ma)o(y)e(still)i(b)q(e)g(in)g(progress\).)k(F)l(or)16 b(example)g(in)h(a)f(grid-t)o(yp)q(e)h(problem)f(domain,)g(consider)h(the)74 1453 y(sequence)g(of)d(calls:)146 1548 y Fl(row_pgid)23 b(=)g (partition\(grid_pgid,)e(row_id,)i(key\))146 1605 y(col_pgid)g(=)g (partition\(grid_pgid,)e(col_id,)i(key\))146 1661 y(bcast\(row_pgid,)f (orig1,)h(data1\);)146 1718 y(bcast\(col_pgid,)f(orig2,)h(data2\);)74 1813 y Fn(where)12 b Fl(row)p 277 1813 15 2 v 17 w(pgid)e Fn(and)i Fl(col)p 557 1813 V 17 w(pgid)e Fn(represen)o(t)i(t)o(w)o(o)e(Pro)q(cess)h (Groups)g(for)g(eac)o(h)g(of)g(the)h(pro)q(cesses)f(in)o(v)o(ok-)74 1869 y(ing)17 b(the)f(t)o(w)o(o)e Fl(partition\(\))h Fn(calls.)23 b(Since)17 b Fl(row)p 910 1869 V 17 w(pgid)e Fn(and)h Fl(col)p 1199 1869 V 17 w(pgid)f Fn(are)h(not)g(disjoin)o(t,)g(the)g(second)74 1926 y(broadcast)c(op)q(eration)h(ma)o(y)f(b)q(e)i(partially)g(completed)g (in)f(parallel)i(with)e(the)g(\014rst)f(one.)20 b(The)13 b(program)74 1982 y(execution)g(order)f(guaran)o(tees)f(that)h(the)g(pro)q(cesses)g(in)h Fl(row)p 1102 1982 V 17 w(pgid)f Fi(\\)g Fl(col)p 1341 1982 V 17 w(pgid)f Fn(will)j(reac)o(h)e(the)g(second)74 2038 y(broadcast)18 b(only)i(after)e(they)h(\014nish)h(their)f(participation)h(in)g(the)f (\014rst)g(broadcast.)30 b(CCL)19 b(prev)o(en)o(ts)74 2095 y(p)q(ossible)e(mixing)f(of)f(messages)f(b)q(et)o(w)o(een)i(the)f(t)o(w)o(o)f (broadcast)g(op)q(erations)h(b)o(y)g(using)h(seman)o(tics)f(and)74 2151 y(prop)q(erties)22 b(describ)q(ed)h(in)f(Section)g(3)f(so)g(that)g(the)g (underlying)i(comm)o(unication)f(subsystem)f(can)74 2208 y(distinguish)c(b)q (et)o(w)o(een)f(the)f(t)o(w)o(o)f(messages.)74 2361 y Fk(2.3)56 b(Unique)18 b(Pro)r(cess)g(Groups)h(Iden)n(ti\014ers)74 2478 y Fn(The)h(actual)f(v)m(alue)i(of)e(a)g(PGID)g(is)h(determined)g(b)o(y)g (CCL,)f(and)g(user)h(co)q(de)g(that)e(dep)q(ends)j(on)e(the)74 2534 y(actual)e(v)m(alue)g(of)f(a)g(PGID)g(\(suc)o(h)h(as)f(branc)o(hing)h (based)g(on)f(the)g(v)m(alue)i(of)e(a)g(PGID\))g(is)g(not)g(allo)o(w)o(ed.)74 2591 y(That)h(is,)i(suc)o(h)f(co)q(de)g(ma)o(y)f(op)q(erate)g(correctly)h (with)g(one)g(implemen)o(tation)h(of)e(CCL)h(and)g(not)f(with)960 2727 y(6)p eop %%Page: 7 7 bop 74 157 a Fn(another.)19 b(On)c(the)g(other)f(hand,)h(the)g(user)f(is)h (allo)o(w)o(ed)g(to)f(c)o(hec)o(k)h(equalit)o(y)g(of)f(PGIDs)h(and)f(to)g (include)74 214 y(PGIDs)e(in)h(messages)e(sen)o(t)h(from)f(one)i(pro)q(cess)f (to)f(another.)19 b(As)12 b(a)g(result,)g(CCL)g(m)o(ust)g(guaran)o(tee)f (that)74 270 y(all)19 b(the)f(pro)q(cesses)h(in)f(a)g(single)h(Pro)q(cess)f (Group)g(use)g(the)g(same)g(system-wide)h(unique)g(PGID,)e(and)74 327 y(that)c(distinct)j(Pro)q(cess)e(Groups)f(ha)o(v)o(e)h(distinct)h(PGIDs)f (\(whether)g(or)f(not)h(they)g(ha)o(v)o(e)f(an)o(y)h(pro)q(cesses)74 383 y(in)i(common\).)74 471 y(W)l(e)h(no)o(w)e(describ)q(e)j(a)e(simple)i (mec)o(hanism)f(for)f(generating)g(PGIDs)g(that)f(are)h(system-wide)h (unique.)74 527 y(Eac)o(h)c(pro)q(cess)g(main)o(tains)h(a)e(lo)q(cal)i(coun)o (ter,)f(whic)o(h)h(is)g(initialized)i(to)c(0.)19 b(Logically)l(,)c(a)e(PGID)f (consists)74 583 y(of)g(t)o(w)o(o)g(\014elds:)19 b Fl(\(counter,)k(pid\))p Fn(.)18 b(When)13 b(a)g(new)g(Pro)q(cess)f(Group)g(is)h(created,)g(the)g(pro) q(cess)g(that)f(has)74 640 y(rank)17 b(0)h(in)g(this)g(group)g(determines)g (the)g(PGID)f(b)o(y)h(pairing)g(its)g(lo)q(cal)h(coun)o(ter)e(and)h(its)g (PID.)f(This)74 696 y(pro)q(cess)h(broadcasts)f(the)g(v)m(alue)i(of)e(the)h (PGID)f(to)g(all)h(the)g(pro)q(cesses)g(in)g(the)g(Pro)q(cess)f(Group,)g(and) 74 753 y(it)e(then)g(incremen)o(ts)g(its)g(lo)q(cal)h(coun)o(ter.)k(This)15 b(sc)o(heme)g(guaran)o(tees)f(system-wide)h(uniqueness)h(of)f(all)74 809 y(the)g(PGIDs)g(in)h(CCL.)74 897 y(The)j(implemen)o(tation)h(of)e(the)h (ab)q(o)o(v)o(e)g(mec)o(hanism)g(in)g(CCL)g(is)g(sligh)o(tly)h(di\013eren)o (t.)31 b(Because)19 b(most)74 953 y(existing)i(comm)o(unication)f(subsystems) g(ha)o(v)o(e)f(relativ)o(ely)i(large)f(start-up)f(costs)h(compared)f(to)g (the)74 1010 y(transfer)d(times,)h(CCL)g(app)q(ends)h(the)f(v)m(alue)h(of)e (the)h(lo)q(cal)h(coun)o(ter)f(of)f(eac)o(h)h(pro)q(cess)g(to)f(the)h(con)o (trol)74 1066 y(information)e(comm)o(unicated)f(during)i(the)e(setup)h(of)e (a)h(Pro)q(cess)h(Group.)k(This)c(eliminates)h(the)e(extra)74 1123 y(comm)o(unication)h(steps)g(needed)h(for)e(broadcasting)g(the)h(PID)g (and)f(the)h(coun)o(ter)f(of)h(the)f(pro)q(cess)h(with)74 1179 y(rank)f(0,)g(since)h(eac)o(h)f(pro)q(cess)g(has)g(all)h(the)f(information)h (needed)g(to)e(calculate)j(the)e(PGID)f(of)h(the)g(new)74 1236 y(Pro)q(cess)h(Group)g(to)g(whic)o(h)h(it)f(b)q(elongs.)74 1388 y Fk(2.4)56 b(Common)17 b(Group)h(Structure)g(Routines)74 1505 y Fn(Recen)o(tly)l(,)25 b(a)d(set)g(of)g(Common)g(Group)g(Structure)g (\(CGS\))g(routines)g(has)h(b)q(een)g(prop)q(osed)g(as)f(an)74 1562 y(extension)c(to)f(CCL)g([8].)25 b(The)18 b(CGS)f(routines)h(mak)o(e)e (use)i(of)f(pro)q(cess)h(group)f(routines)g(for)g(de\014ning)74 1618 y(pro)q(cess)i(groups)g(that)f(arise)h(in)h(algorithms)f(with)g(grid)h (and)f(h)o(yp)q(ercub)q(e)i(structures.)30 b(Once)20 b(these)74 1675 y(grid-structured)j(and)f(h)o(yp)q(ercub)q(e-structured)h(groups)f(ha)o (v)o(e)f(b)q(een)i(de\014ned,)i(the)d(standard)f(CCL)74 1731 y(routines)g(can)g(b)q(e)g(used)g(within)g(the)g(structured)f(groups)h(to)e (p)q(erform)i(collectiv)o(e)h(comm)o(unication)74 1788 y(op)q(erations.)27 b(The)18 b(CGS)f(routines)h(also)f(include)j(utilities)g(for)c(con)o(v)o (erting)i(b)q(et)o(w)o(een)g(1-dimensional)74 1844 y(and)e (higher-dimensional)i(addresses.)74 1932 y(As)k(an)h(example,)h(the)f Fg(F)o(ORM2DGRID)f Fn(routine)h(tak)o(es)e(an)h(existing)i(group,)f(views)g (it)g(as)f(an)74 1988 y Fj(X)14 b Fi(\002)e Fj(Y)27 b Fn(2D-grid)17 b(\(where)f Fj(X)k Fn(and)d Fj(Y)27 b Fn(are)17 b(sp)q(eci\014ed)i(b)o(y)d (the)h(user\),)g(and)f(partitions)h(it)g(in)o(to)g(a)f(set)h(of)74 2045 y(nono)o(v)o(erlapping)j(groups)f(corresp)q(onding)h(to)f(the)g(columns) h(of)f(a)g(t)o(w)o(o-dimensional)h(grid)f(and)h(also)74 2101 y(in)o(to)e(a)f(set)g(of)g(nono)o(v)o(erlapping)h(groups)g(corresp)q(onding)g (to)f(the)g(ro)o(ws)g(of)g(a)g(t)o(w)o(o-dimensional)i(grid.)74 2158 y(Th)o(us,)c(eac)o(h)g(pro)q(cess)h(receiv)o(es)g(t)o(w)o(o)e(new)h (PGIDs,)g(one)g(for)g(the)g(column)h(in)g(whic)o(h)g(it)g(is)f(lo)q(cated)h (and)74 2214 y(one)f(for)g(the)g(ro)o(w)g(in)h(whic)o(h)g(it)f(is)h(lo)q (cated)g(in)g(the)f(t)o(w)o(o-dimensional)h(grid.)74 2388 y Fm(3)69 b(Seman)n(tic)21 b(Issues)i(in)f(CCL)74 2521 y Fn(This)e(section)f (discusses)h(sev)o(eral)f(issues)h(related)f(to)f(the)h(seman)o(tics)g(of)f (p)q(oin)o(t-to-p)q(oin)o(t)i(comm)o(uni-)74 2577 y(cation)g(and)g(of)f (collectiv)o(e)j(comm)o(unication)e(op)q(erations)g(on)g(Pro)q(cess)f (Groups.)33 b(These)20 b(issues)h(are)960 2727 y(7)p eop %%Page: 8 8 bop 74 157 a Fn(fundamen)o(tal)16 b(to)e(the)i(design)g(of)f(CCL)g(and)g(of)g (similar)h(libraries.)74 310 y Fk(3.1)56 b(Send)18 b(and)i(Receiv)n(e)c (Seman)n(tics)74 427 y Fn(In)21 b(order)f(for)g(CCL)g(to)g(b)q(e)h(p)q (ortable,)g(it)g(is)g(imp)q(ortan)o(t)e(that)h(it)g(is)h(based)g(on)f(a)g (simple)i(mo)q(del)f(of)74 483 y(p)q(oin)o(t-to-p)q(oin)o(t)16 b(comm)o(unication)g(that)e(is)i(a)o(v)m(ailable)h(on)e(a)g(wide)h(range)f (of)g(mac)o(hines.)74 571 y(The)f(basic)h(mo)q(del)f(of)g(p)q(oin)o(t-to-p)q (oin)o(t)g(comm)o(unication)h(consists)f(of)f(a)h(single)h Fl(send)e Fn(op)q(eration)h(and)g(a)74 627 y(single)i Fl(receive)d Fn(op)q(eration.)20 b(It)14 b(is)h(assumed)f(that)g(a)g(pro)q(cess)g(can)h (send/receiv)o(e)g(to/from)e(an)o(y)h(other)74 684 y(pro)q(cess)g(in)g(the)g (parallel)h(mac)o(hine.)20 b(Namely)l(,)14 b(there)f(is)h(no)g(notion)f(of)g (a)h(top)q(ology)f(at)f(this)i(lev)o(el.)21 b(This)74 740 y(mo)q(del)14 b(is)f(called)i(a)d(fully-connected)j(mo)q(del.)20 b(In)14 b(addition)g(it)f(is)g(assumed)g(that)f(the)h(send)h(and)f(receiv)o(e)74 797 y(op)q(erations)i(ha)o(v)o(e)g(the)h(follo)o(wing)f(four)g(prop)q (erties:)74 884 y(\(1\))h(The)h(op)q(erations)g(are)f(blo)q(c)o(king,)i (namely)l(,)g(a)e Fh(blo)n(cking)h(send)j Fn(op)q(eration)d(and)g(a)g Fh(blo)n(cking)f(r)n(e)n(c)n(eive)74 941 y Fn(op)q(eration)e(are)g(used.)20 b(The)14 b(blo)q(c)o(king)i(send)f(op)q(eration)f(is)g(a)g Fl(send)g Fn(that)f(do)q(es)h(not)g(return)g(to)f(the)h(user)74 997 y(un)o(til)k(the)g(message)e(has)h(b)q(een)i(copied)f(out)f(of)f(the)i (user's)f(bu\013er.)25 b(It)17 b(should)i(b)q(e)e(p)q(oin)o(ted)h(out)f(that) 74 1054 y Fh(blo)n(cking)e(send)20 b Fn(is)c(di\013eren)o(t)g(from)f Fh(synchr)n(onous)h(send)f Fn(in)h(whic)o(h)h(the)f Fl(send)f Fn(do)q(es)h(not)f(return)g(to)g(the)74 1110 y(user)g(un)o(til)h(a)f(matc)o (hing)g Fl(receive)f Fn(op)q(eration)h(has)g(b)q(een)h(issued.)1217 1094 y Ff(1)1258 1110 y Fn(The)f(blo)q(c)o(king)h(receiv)o(e)g(op)q(eration) 74 1167 y(is)g(a)f Fl(receive)g Fn(that)g(do)q(es)g(not)h(return)f(to)g(the)g (user)h(un)o(til)h(the)e(requested)h(message)f(has)h(arriv)o(ed)f(and)74 1223 y(has)g(b)q(een)i(placed)f(in)g(the)f(user's)g(bu\013er.)74 1311 y(\(2\))20 b(The)g(receiv)o(e)i(op)q(eration)f(is)g(a)f Fh(r)n(e)n(c)n(eive-by-sour)n(c)n(e)p Fn(,)g(where)h(the)f(receiv)o(er)h(sp)q (eci\014es)i(the)d(desired)74 1367 y(source)e(of)f(the)h(message.)27 b(Only)19 b(a)f(message)f(from)g(the)h(sp)q(eci\014ed)i(source)d(can)h(b)q(e) h(deliv)o(ered)g(to)e(the)74 1424 y(receiv)o(er.)26 b(The)17 b(desired)h(source)f(m)o(ust)f(alw)o(a)o(ys)g(b)q(e)i(sp)q(eci\014ed)h(and)e (the)g(use)g(of)f(a)h(wildcard)h(v)m(alue)g(for)74 1480 y(the)d(source)h (parameter)e(is)i(not)f(allo)o(w)o(ed.)74 1568 y(\(3\))g(P)o(oin)o(t-to-p)q (oin)o(t)h(comm)o(unication)h(b)q(et)o(w)o(een)f(an)o(y)g(t)o(w)o(o)f(pro)q (cesses)h(is)h(required)g(to)f(b)q(e)g(FIF)o(O)g(suc)o(h)74 1624 y(that)j(m)o(ultiple)i(messages)e(sen)o(t)g(from)g(one)h(pro)q(cess)f (to)g(another)g(are)g(receiv)o(ed)i(in)f(the)g(same)f(order)74 1681 y(in)h(whic)o(h)h(they)e(w)o(ere)g(sen)o(t.)33 b(Ho)o(w)o(ev)o(er,)19 b(the)g Fl(send)g Fn(and)h Fl(receive)e Fn(op)q(erations)i(are)f(async)o (hronous,)74 1737 y(in)i(the)f(sense)h(that)e(no)h(b)q(ounds)h(are)e(assumed) h(on)g(the)g(relativ)o(e)h(sp)q(eeds)g(of)e(the)i(pro)q(cesses)f(or)f(the)74 1794 y(time)c(required)g(to)e(pass)h(messages)g(b)q(et)o(w)o(een)g(pro)q (cesses.)20 b(F)l(urthermore,)13 b(the)h(basic)h(mo)q(del)g(mak)o(es)f(no)74 1850 y(assumptions)j(regarding)g(whether)g(or)g(not)f(the)h(system)g(pro)o (vides)g(bu\013ering)g(of)g(messages)f(b)q(et)o(w)o(een)74 1906 y(a)j(sender)g(and)g(a)g(receiv)o(er.)31 b(Th)o(us,)20 b(a)e(blo)q(c)o(king)j(send)e(op)q(eration)g(ma)o(y)f(or)h(ma)o(y)f(not)g (blo)q(c)o(k)i(un)o(til)g(a)74 1963 y(matc)o(hing)15 b(receiv)o(e)i(is)e (issued.)74 2050 y(\(4\))g(Di\013eren)o(t)g Fl(send)g Fn(op)q(erations)h (issued)h(from)d(m)o(ultiple)k(source)d(pro)q(cesses)h(to)f(a)h(single)h (destination)74 2107 y(pro)q(cess)i(are)e Fh(non-interfering)p Fn(.)28 b(This)19 b(means)f(that)f(the)h(existence)i(of)d(messages)h(that)g (the)g(receiv)o(er)74 2163 y(do)q(es)e(not)f(wish)h(to)f(select)h(\(b)q (ecause)g(these)g(messages)f(do)g(not)g(matc)o(h)g(the)h(source)f(\014eld)i (sp)q(eci\014ed)h(b)o(y)74 2220 y(the)g Fl(receive)e Fn(op)q(eration\))i (cannot)f(prev)o(en)o(t)g(the)h(reception)g(of)f(a)h(message)f(that)f(the)i (receiv)o(er)g(do)q(es)74 2276 y(wish)d(to)g(receiv)o(e.)20 b(F)l(or)14 b(example,)i(if)f(pro)q(cesses)g(1)f(through)h Fj(n)9 b Fi(\000)h Fn(1)k(eac)o(h)h(sends)g(a)g(message)f(to)g(pro)q(cess)74 2333 y(0)h(and)h(if)g(pro)q(cess)g(0)f(issues)h(a)g Fl(receive)e Fn(in)j(order)e(to)g(receiv)o(e)h(a)f(message)g(from)g(pro)q(cess)h Fj(n)11 b Fi(\000)f Fn(1,)15 b(then)74 2389 y(this)g Fl(receive)e Fn(op)q(eration)i(will)h(get)d(the)i(message)f(sen)o(t)g(from)f(pro)q(cess)i Fj(n)8 b Fi(\000)h Fn(1)14 b(ev)o(en)h(if)f(all)i(of)e(the)g(other)74 2446 y(messages)20 b(w)o(ere)h(sen)o(t)f(earlier.)38 b(While)22 b(this)f(non-in)o(terference)h(prop)q(ert)o(y)e(seems)h(v)o(ery)g(natural,)g (it)74 2502 y(could)f(b)q(e)g(violated)f(b)o(y)g(a)g(system)f(that)g(stores)h (messages)f(in)i(bu\013ers)e(on)h(the)g(receiv)o(er)h(side,)g(since)p 74 2542 718 2 v 126 2569 a Fe(1)143 2584 y Fd(Some)14 b(researc)o(hers)g(use) f(the)g(term)g Fc(blo)n(cking)e(send)k Fd(to)e(indicate)i(what)e(w)o(e)g(ha)o (v)o(e)g(de\014ned)h(as)g Fc(synchr)n(onou)o(s)d(send)p Fd(.)960 2727 y Fn(8)p eop %%Page: 9 9 bop 74 157 a Fn(suc)o(h)21 b(bu\013ers)g(could)g(b)q(e)h(\014lled)g(b)o(y)f (messages)f(other)g(than)g(the)h(one)g(that)f(the)g(receiv)o(er)i(selects)f (to)74 214 y(receiv)o(e.)74 301 y(P)o(arallel)c(co)q(de)f(that)g(uses)g (message-passing)g(comm)o(unication)g(with)g(these)g(four)g(prop)q(erties)g (is)h Fh(r)n(ac)n(e-)74 358 y(fr)n(e)n(e)p Fn(:)g(there)10 b(is)h(a)f(unique)i(p)q(ossible)g(matc)o(hing)e(of)g(sends)h(to)e(receiv)o (es,)j(and)e(a)g(unique)i(p)q(ossible)g(successful)74 414 y(execution,)i (indep)q(enden)o(t)g(of)e(the)g(relativ)o(e)g(sp)q(eeds)h(of)f(the)g(pro)q (cesses)g(and)h(the)f(comm)o(unication)h(in)f(the)74 471 y(system.)18 b(Co)q(de)11 b(that)f(is)h(written)f(to)g(b)q(e)h(p)q(ortable)g(mak)o(es)f (no)g(assumption)h(on)g(the)f(amoun)o(t)g(of)g(bu\013ering)74 527 y(pro)o(vided)k(b)o(y)f(the)g(comm)o(unication)h(system;)f(it)h(needs)g (to)e(w)o(ork)g(ev)o(en)i(if)f(eac)o(h)h(send)f(op)q(eration)h(blo)q(c)o(ks) 74 583 y(un)o(til)k(a)f(matc)o(hing)h(receiv)o(e)g(is)f(executed)i(\(i.e.,)e (comm)o(unication)h(is)f(sync)o(hronous\).)26 b(The)17 b(execution)74 640 y(of)e(suc)o(h)g Fh(safe)f Fn(co)q(de)i(is)f(deterministic:)22 b(if)15 b(the)g(program)f(deadlo)q(c)o(ks)i(on)f(some)f(implemen)o(tation,)i (then)74 696 y(it)g(alw)o(a)o(ys)e(deadlo)q(c)o(ks.)74 784 y(Although)k(the)g(basic)h(mo)q(del)g(of)e(p)q(oin)o(t-to-p)q(oin)o(t)h(comm) o(unication)h(de\014ned)g(ab)q(o)o(v)o(e)e(is)i(su\016cien)o(t)f(for)74 840 y(implemen)o(ting)12 b(all)g(of)e(the)g(CCL)h(routines,)g(the)g(actual)f (implemen)o(tation)i(of)e(CCL)g(uses)h(t)o(w)o(o)e(additional)74 897 y(features)j(that)g(are)g(found)h(in)g(man)o(y)f(systems.)19 b(These)13 b(t)o(w)o(o)e(features)h(are)g(a)g Fl(send-receive)f Fn(op)q(eration)74 953 y(and)18 b(the)g(usage)g(of)f(message)g(tags)686 937 y Ff(2)705 953 y Fn(.)27 b(W)l(e)18 b(next)g(discuss)h(these)f(t)o(w)o(o) e(additional)k(features)d(and)h(their)74 1010 y(merits)d(in)h(implemen)o (ting)i(CCL.)74 1097 y(The)f Fl(send-receive)e Fn(op)q(eration)i(sim)o (ultaneously)h(sends)f(a)f(message)g(to)g(one)h(pro)q(cess)g(and)g(receiv)o (es)74 1154 y(a)22 b(message)g(from)f(another)h(pro)q(cess)h(without)f (requiring)i(additional)f(bu\013er)g(space)f(for)g(either)h(of)74 1210 y(the)17 b(messages.)25 b(CCL)17 b(uses)h(the)f Fl(send-receive)e Fn(op)q(eration)j(for)e(implemen)o(ting)j Fl(shift)d Fn(op)q(erations)74 1267 y(within)23 b(Pro)q(cess)f(Groups.)38 b(The)22 b Fl(shift)f Fn(op)q(eration)h(ma)o(y)f(create)g(a)g(cycle)i(of)e(pro)q(cesses,)i(eac)o(h) f(of)74 1323 y(whic)o(h)c(is)g(sending)h(a)e(message)g(to)g(its)h(successor)f (in)i(the)e(cycle)i(and)e(receiving)i(a)f(message)f(from)f(its)74 1380 y(predecessor)h(in)g(the)f(cycle.)24 b(If)17 b(a)f Fl(send-receive)e Fn(op)q(eration)i(is)h(not)f(used,)h(then)f(the)g Fl(send)g Fn(and)g(the)74 1436 y Fl(receive)f Fn(op)q(erations)h(m)o(ust)f(b)q(e)h (paired)g(carefully)h(to)e(a)o(v)o(oid)g(deadlo)q(c)o(ks.)22 b(\(A)16 b(deadlo)q(c)o(k)g(can)g(o)q(ccur)g(if)74 1493 y Fl(send)d Fn(op)q(erations)h(are)f(forced)h(to)f(w)o(ait)g(for)g(matc)o(hing)h Fl(receive)e Fn(op)q(erations)i(that)f(are)g(nev)o(er)h(issued.\))74 1549 y(F)l(or)i(example,)h(to)e(a)o(v)o(oid)h(deadlo)q(c)o(ks)h(in)g Fl(shift)f Fn(op)q(erations,)g(one)g(can)h(pair)f(the)h Fl(send)e Fn(and)i Fl(receive)74 1606 y Fn(op)q(erations,)k(if)g(the)f(cycle)i(is)e(of) g(ev)o(en)g(length,)i(as)e(follo)o(ws.)35 b(Pro)q(cesses)20 b(in)h(ev)o(en)f(p)q(ositions)h(in)g(the)74 1662 y(cycle)16 b(w)o(ould)f(p)q(erform)g(a)f Fl(send)g Fn(follo)o(w)o(ed)h(b)o(y)g(a)f Fl(receive)p Fn(,)f(while)k(pro)q(cesses)e(in)g(o)q(dd)g(p)q(ositions)h(in)f (the)74 1718 y(cycle)h(w)o(ould)f(p)q(erform)f(a)h Fl(receive)e Fn(follo)o(w)o(ed)i(b)o(y)g(a)f Fl(send)p Fn(.)19 b(\(Cycles)c(of)g(o)q(dd)g (lengths)g(can)g(b)q(e)g(handled)74 1775 y(similarly)l(.\))33 b(Although)19 b(suc)o(h)g(an)g(algorithm)g(is)g(p)q(ossible)i(in)f(CCL,)e (since)i(eac)o(h)f(pro)q(cess)g(kno)o(ws)f(its)74 1831 y(p)q(osition)c(in)f (the)g(cycle,)h(it)f(is)g(m)o(uc)o(h)f(simpler)i(and)f(more)f(e\016cien)o(t)h (to)f(use)h(a)f Fl(send-receive)f Fn(op)q(eration)74 1888 y(for)k(implemen)o (ting)i(the)e Fl(shift)p Fn(.)74 1975 y(Some)f(p)q(oin)o(t-to-p)q(oin)o(t)h (comm)o(unication)g(libraries)h(allo)o(w)e(the)g(use)h(of)f(wildcard)h(v)m (alues)g(in)g(the)f(source)74 2032 y(parameter)h(of)g(a)h(receiv)o(e)g(call.) 23 b(Ev)o(en)16 b(if)g(wildcard)h(v)m(alues)g(are)e(not)g(used)i(in)f(the)g (implemen)o(tation)h(of)74 2088 y(CCL,)h(there)h(is)g(still)h(the)f(risk)g (that)f(a)g(message)g(sen)o(t)g(as)g(part)g(of)h(the)f(implemen)o(tation)i (of)e(a)g(CCL)74 2145 y(call)i(will)g(b)q(e)g(mistak)o(enly)f(receiv)o(ed)h (b)o(y)f(a)f(user)h(receiv)o(e)h(with)f(a)f(wildcard)i(source.)31 b(An)19 b(additional)74 2201 y(mec)o(hanism)e(is)g(needed)h(to)e(distinguish) j(CCL)d(messages)g(from)g(user)h(messages.)23 b(Message)16 b(tags)f(can)74 2258 y(pro)o(vide)j(suc)o(h)g(mec)o(hanism.)27 b(These)18 b(are)f(v)m(alues)i(that)e(are)g(app)q(ended)i(to)e(the)g(con)o (ten)o(t)g(of)g(p)q(oin)o(t-to-)74 2314 y(p)q(oin)o(t)f(messages.)k(T)l(ags) 15 b(are)g(t)o(ypically)h(used)g(to)f(distinguish)j(b)q(et)o(w)o(een)d (di\013eren)o(t)h(t)o(yp)q(es)f(of)g(messages)74 2371 y(in)f(an)g (application,)h(as)e(w)o(ell)h(as)f(b)q(et)o(w)o(een)h(m)o(ultiple)h (messages)e(sen)o(t)g(to)f(a)h(destination)i(from)d(the)i(same)74 2427 y(source.)34 b(Eac)o(h)19 b Fl(send)g Fn(op)q(eration)h(sp)q(eci\014es)i (a)d(v)m(alue)i(in)g(the)f(tag)e(\014eld)j(of)f(the)f(message,)i(and)e(eac)o (h)74 2483 y Fl(receive)14 b Fn(op)q(eration)i(sp)q(eci\014es)h(a)e(desired)h (tag)e(v)m(alue)j(to)d(b)q(e)i(matc)o(hed)f(with)h(the)f(receiv)o(ed)h (message.)p 74 2523 718 2 v 126 2550 a Fe(2)143 2566 y Fd(Some)e(researc)o (hers)g(use)f(the)g(term)g Fc(typ)n(e)h Fd(instead)h(of)d Fc(tag)p Fd(.)960 2727 y Fn(9)p eop %%Page: 10 10 bop 74 157 a Fn(W)l(e)21 b(assume)f(that)f(a)h(mec)o(hanism)h(is)g(a)o(v)m (ailable)h(to)e(allo)q(cate)h(tag)f(ranges)g(to)f(v)m(arious)i(libraries,)i (so)74 214 y(that)16 b(user)i(messages)e(and)h(messages)g(generated)g(b)o(y)g (CCL)g(ha)o(v)o(e)f(disjoin)o(t)i(tag)e(ranges.)25 b(If)17 b(wildcard)74 270 y(tag)f(v)m(alues)i(are)f(allo)o(w)o(ed)g(in)h(receiv)o(e)g (op)q(erations,)f(one)g(also)g(need)h(to)f(mak)o(e)f(sure)h(that)f(user)i (receiv)o(e)74 327 y(op)q(erations)j(cannot)f(matc)o(h)f(tag)h(v)m(alues)h (used)g(b)o(y)f(CCL.)g(This)h(can)g(b)q(e)g(ac)o(hiev)o(ed)g(either)g(with)g (an)74 383 y(include/exclu)q(de)c(mec)o(hanism)d(for)f(wildcard)h(tag)f (ranges,)g(as)g(pro)o(vided)i(b)o(y)e(Express)h([30)o(])f(or)g(with)h(an)74 439 y(additional)j(con)o(text)d(mec)o(hanism,)i(as)f(pro)o(vided)h(b)o(y)f (Zip)q(co)q(de)i([28)o(].)74 592 y Fk(3.2)56 b(Implem)o(en)n(tation)16 b(Correctness)74 709 y Fn(There)i(are)f(man)o(y)g(issues)h(related)g(to)f(a)g (correct)g(implemen)o(tation)h(of)f(CCL.)h(These)f(issues)i(include,)74 766 y(for)f(example,)i(separating)f(user)g(p)q(oin)o(t-to-p)q(oin)o(t)g (messages)f(from)g(CCL)h(p)q(oin)o(t-to-p)q(oin)o(t)g(messages,)74 822 y(guaran)o(teeing)11 b(that)g(the)g(correct)g(messages)g(b)q(e)h(deliv)o (ered)h(for)d(collectiv)o(e)j(comm)o(unication)f(op)q(erations)74 879 y(in)g(o)o(v)o(erlapping)f(Pro)q(cess)f(Groups,)h(and)g(distinguishing)i (b)q(et)o(w)o(een)e(messages)f(that)g(b)q(elong)h(to)f(di\013eren)o(t)74 935 y(collectiv)o(e)17 b(comm)o(unication)f(op)q(erations)f(in)h(the)g(same)f (group.)74 1023 y(First,)h(the)h(seman)o(tics)g(of)f(CCL)g(assume)h(that)e (eac)o(h)i(CCL)f(routine)h(b)q(e)h(view)o(ed)f(as)f(a)g(barrier)h(b)o(y)f (the)74 1079 y(user.)26 b(Namely)l(,)18 b(a)e(pro)q(cess)i(can)f(return)g (from)f(a)h(CCL)g(routine)h(only)f(after)g(all)h(other)e(participating)74 1136 y(pro)q(cesses)k(called)i(this)e(routine.)35 b(Therefore,)20 b(in)h(a)f(correct)f(program,)h(an)o(y)f(CCL)h(op)q(eration)g(that)74 1192 y(in)o(v)o(olv)o(es)15 b(pro)q(cesses)h Fj(p)e Fn(and)h Fj(q)i Fn(should)f(b)q(e)f(called)h(in)g(the)f(same)f(order)h(b)o(y)g(pro)q (cesses)g Fj(p)f Fn(and)h Fj(q)r Fn(,)g(and)g(no)74 1248 y(p)q(oin)o(t-to-p)q (oin)o(t)h(comm)o(unication)g(b)q(et)o(w)o(een)f(pro)q(cesses)h Fj(p)f Fn(and)g Fj(q)i Fn(should)f(o)o(v)o(erlap)f(a)g(CCL)g(op)q(eration.)74 1336 y(The)j(correctness)f(of)g(the)h(CCL)f(implemen)o(tation)i(is)f(based)g (on)f(the)h(follo)o(wing)g(3)f(prop)q(erties)h(of)f(the)74 1393 y(p)q(oin)o(t-to-p)q(oin)o(t)f(comm)o(unication.)130 1517 y(1.)22 b(The)15 b(p)q(oin)o(t-to-p)q(oin)o(t)h(comm)o(unication)g(supp)q (orts)f(receiv)o(e-b)o(y-source)i(op)q(erations.)130 1611 y(2.)22 b(The)15 b(p)q(oin)o(t-to-p)q(oin)o(t)h(comm)o(unication)g(is)g(FIF)o(O)f (and)g(non-in)o(terfering.)130 1705 y(3.)22 b(A)13 b(collectiv)o(e)i(comm)o (unication)e(algorithm)g(in)h(CCL)f(is)g(kno)o(wn)g(to)f(all)i(participating) g(pro)q(cesses.)188 1762 y(In)k(particular,)f(for)g(a)g(sp)q(eci\014c)i (collectiv)o(e)f(comm)o(unication)g(op)q(eration)g(o)o(v)o(er)e(a)h(giv)o(en) g(pro)q(cess)188 1818 y(group)e(with)h(kno)o(wn)g(source)g(and)g(destination) h(pro)q(cesses)f(\(if)g(applicable\))i(and)e(message)f(size,)188 1874 y(a)h(\014xed)h(algorithm)f(is)h(c)o(hosen)g(that)e(is)i(kno)o(wn)f(to)g (all)h(the)f(pro)q(cesses.)24 b(Consequen)o(tly)l(,)17 b(at)f(an)o(y)188 1931 y(giv)o(en)f(p)q(oin)o(t)h(in)f(the)g(algorithm,)g(eac)o(h)g(pro)q(cess) g(kno)o(ws)g(exactly)g(to)f(whic)o(h)i(pro)q(cess)f(a)g(message)188 1987 y(should)j(b)q(e)h(sen)o(t)e(or)g(from)g(whic)o(h)h(pro)q(cess)g(a)f (message)g(should)i(b)q(e)f(receiv)o(ed.)28 b(Therefore,)17 b(the)188 2044 y(sequence)f(of)f(steps)g(for)g(implemen)o(ting)i(eac)o(h)e (CCL)g(op)q(eration)h(is)f(w)o(ell)i(de\014ned.)74 2169 y(The)j(foregoing)f (3)g(prop)q(erties)h(guaran)o(tee)f(that)f(the)i(messages)f(receiv)o(ed-b)o (y-source)i(b)o(y)e(an)o(y)g(giv)o(en)74 2225 y(pro)q(cess)e(are)f(receiv)o (ed)i(in)f(the)g(exact)f(same)h(order)f(for)g(an)o(y)g(giv)o(en)h(execution)h (of)e(a)g(CCL)h(algorithm.)74 2282 y(This)k(order)f(is)g(main)o(tained)h(in)g (the)f(presence)h(of)f(other)f(comm)o(unication,)j(lik)o(e)f(p)q(oin)o (t-to-p)q(oin)o(t)g(or)74 2338 y(other)15 b(CCL)h(calls.)22 b(The)16 b(k)o(ey)f(issue)i(is)f(that)f(the)h(FIF)o(O)f(prop)q(ert)o(y)g(b)q (et)o(w)o(een)h(source)g(and)g(destination)74 2395 y(is)j(main)o(tained)g(b)o (y)g(main)o(taining)g(the)f(order)g(b)q(et)o(w)o(een)h(CCL)f(calls)i(and)e (not)g(allo)o(wing)i(in)o(terference)74 2451 y(b)q(et)o(w)o(een)c(CCL)f (calls)h(and)g(p)q(oin)o(t-to-p)q(oin)o(t)f(calls.)949 2727 y(10)p eop %%Page: 11 11 bop 74 157 a Fk(3.3)56 b(Barrier)17 b(Seman)n(tics)74 274 y Fn(As)e(discussed)i(ab)q(o)o(v)o(e,)d(for)h(program)f(correctness,)h(eac)o(h) g(CCL)g(op)q(eration)g(should)h(b)q(e)g(view)o(ed)g(as)f(im-)74 331 y(p)q(osing)h(a)e(barrier)h(sync)o(hronization)h(on)f(its)g(participan)o (ts.)20 b(Ho)o(w)o(ev)o(er,)14 b(from)g(the)h(p)q(erformance)g(p)q(oin)o(t)74 387 y(of)g(view,)g(imp)q(osing)h(a)e(barrier)h(sync)o(hronization)h(on)f(all) h(the)e(participan)o(ts)i(in)f(CCL)g(op)q(erations)g(ma)o(y)74 443 y(ha)o(v)o(e)j(a)g(negativ)o(e)h(e\013ect)f(on)g(p)q(erformance.)30 b(T)l(o)18 b(address)g(this)h(problem,)g(CCL)g(allo)o(ws)f(t)o(w)o(o)f(mo)q (des)74 500 y(of)h(op)q(eration:)26 b Fh(b)n(arrier)19 b(mo)n(de)j Fn(and)d Fh(non-b)n(arrier)f(mo)n(de)p Fn(.)30 b(Selection)20 b(b)q(et)o(w)o(een)e(these)h(t)o(w)o(o)e(mo)q(des)h(is)74 556 y(done)h(globally)h(at)d(the)i(b)q(eginning)h(of)e(an)h(application.)31 b(F)l(or)18 b(ob)o(vious)g(reasons,)h(the)f(default)h(mo)q(de)74 613 y(is)f(the)f(barrier)h(mo)q(de.)26 b(When)18 b(a)f(CCL)g(op)q(eration)h (is)g(executed)g(in)g(barrier)g(mo)q(de,)g(no)f(pro)q(cess)g(can)74 669 y(complete)g(its)g(call)h(to)e(the)h(routine)g(un)o(til)g(all)h(the)f (pro)q(cesses)g(in)g(the)g(relev)m(an)o(t)g(Pro)q(cess)g(Group)f(ha)o(v)o(e) 74 726 y(executed)h(their)g(\(corresp)q(onding\))f(calls)h(to)f(the)g(same)g (routine.)22 b(Therefore,)16 b(when)h(an)f(op)q(eration)g(is)74 782 y(executed)j(in)f(barrier)g(mo)q(de,)f(there)h(m)o(ust)f(exist)h(some)f (p)q(oin)o(t)h(in)g(time)g(at)f(whic)o(h)h(all)h(the)e(pro)q(cesses)74 839 y(in)g(the)f(group)f(are)h Fh(simultane)n(ously)e Fn(executing)j(the)f (same)g(op)q(eration.)22 b(In)16 b(con)o(trast,)e(when)i(an)g(op)q(er-)74 895 y(ation)h(is)g(executed)h(in)f(non-barrier)g(mo)q(de,)g(eac)o(h)g(pro)q (cess)g(blo)q(c)o(ks)g(only)g(un)o(til)h(it)f(has)g(completed)g(its)74 952 y(participation)h(in)g(the)f(op)q(eration.)26 b(Notice)17 b(that)g(the)g(role)g(of)g(eac)o(h)g(pro)q(cess)g(ma)o(y)g(dep)q(end)h(not)f (only)74 1008 y(on)e(the)h(seman)o(tics)f(of)g(the)g(op)q(eration)g(but)h (also)f(on)g(the)g(sp)q(eci\014c)i(algorithm)f(used)f(to)g(implemen)o(t)h (it.)74 1096 y(The)d(di\013erences)h(b)q(et)o(w)o(een)f(the)f(t)o(w)o(o)g(mo) q(des)g(ma)o(y)g(imply)i(di\013erences)g(in)f(the)g(b)q(eha)o(vior)g(of)f(an) h(applica-)74 1152 y(tion)g(program.)18 b(In)c(a)f(non-barrier)g(mo)q(de,)g (pro)q(cesses)h(ma)o(y)e(return)h(more)g(quic)o(kly)h(from)e(calls)i(to)e (CCL)74 1209 y(routines,)19 b(resulting)g(in)h(a)d(b)q(etter)i(p)q (erformance.)29 b(Ho)o(w)o(ev)o(er,)17 b(there)i(is)f(also)h(some)f(danger)g (in)h(using)74 1265 y(non-barrier)14 b(mo)q(de)f(routines)h(as)f(in)h(some)f (of)g(CCL)g(routines)g(the)h(role)f(of)g(eac)o(h)g(pro)q(cess)h(dep)q(ends)h (also)74 1321 y(on)h(the)f(particular)i(algorithm)e(used)i(to)e(implemen)o(t) h(the)g(routine.)22 b(As)15 b(long)h(as)g(the)f(user's)h(program)74 1378 y(is)i(correct,)g(and)f(no)h(wildcard)h(p)q(oin)o(t-to-p)q(oin)o(t)f (comm)o(unication)h(is)f(used,)g(then)g(non-barrier)g(mo)q(de)74 1434 y(routines)h(will)i(op)q(erate)e(in)h(an)e(implemen)o(tation-indep)r (ende)q(n)o(t)j(manner)e(\(as)f(argued)h(in)h(the)f(ab)q(o)o(v)o(e)74 1491 y(subsection\).)h(Ho)o(w)o(ev)o(er,)14 b(if)g(wildcard)i(p)q(oin)o (t-to-p)q(oin)o(t)f(comm)o(unication)g(is)g(used,)g(then)g(the)f(b)q(eha)o (vior)74 1547 y(of)h(a)f(non-barrier)i(mo)q(de)f(routine)h(can)f(b)q(e)g (implemen)o(tation-dep)q(end)q(en)o(t.)22 b(As)15 b(an)g(example,)h(consider) 74 1604 y(the)e(follo)o(wing)h(piece)h(of)e(co)q(de,)h(in)g(whic)o(h)g Fl(G)f Fn(is)h(the)f(group)g(iden)o(ti\014er)i(of)e(a)g(Pro)q(cess)g(Group)g (consisting)74 1660 y(of)i(pro)q(cesses)h Fl(a)p Fn(,)f Fl(b)p Fn(,)g(and)g Fl(c)p Fn(.)23 b(In)17 b(this)f(example,)h(eac)o(h)g Fl(brecv)e Fn(is)i(a)f(blo)q(c)o(king)i(receiv)o(e)f(op)q(eration)f(with)74 1717 y(a)f(wildcard)i(source)e(parameter.)193 1805 y Fl(process)23 b(a)334 b(process)23 b(b)334 b(process)23 b(c)193 1862 y(---------)333 b(---------)g(---------)193 1918 y(bsend\(c\))906 b(brecv\(*\))193 1975 y(bcast\(G\))357 b(bcast\(G\))g(bcast\(G\))742 2031 y(bsend\(c\))g (brecv\(*\))74 2176 y Fn(Assume)17 b(that)e(the)h(source)g(of)g(the)g Fl(bcast)g Fn(op)q(eration)g(is)g(pro)q(cess)h Fl(b)p Fn(.)22 b(If)17 b(barrier)f(seman)o(tics)g(are)g(used)74 2233 y(for)d(the)h (broadcast)g(op)q(eration,)g(then)g(the)g(\014rst)g(receiv)o(e)g(of)g(pro)q (cess)g Fl(c)g Fn(will)h(receiv)o(e)g(the)f(message)g(sen)o(t)74 2289 y(b)o(y)i(pro)q(cess)g Fl(a)p Fn(,)g(and)g(the)g(second)g(receiv)o(e)h (of)e(pro)q(cess)h Fl(c)g Fn(will)i(receiv)o(e)f(the)f(message)f(send)h(b)o (y)g(pro)q(cess)74 2346 y Fl(b)p Fn(.)38 b(On)22 b(the)f(other)g(hand,)i(if)f (pro)q(cess)g Fl(b)f Fn(can)g(exit)h(the)f(broadcast)g(call)h(b)q(efore)g (pro)q(cess)g Fl(c)f Fn(starts)74 2402 y(executing)d(the)f(broadcast,)f(then) i(it)f(is)g(p)q(ossible)i(for)d(the)h(messages)g(to)f(b)q(e)i(receiv)o(ed)g (in)g(the)f(rev)o(erse)74 2458 y(order:)j(pro)q(cess)15 b Fl(c)f Fn(receiv)o(es)i(b)q(efore)f(the)g(broadcast)f(the)h(message)f(that)g Fl(b)h Fn(sen)o(t)g(after)f(the)h(broadcast,)74 2515 y(and)h(receiv)o(es)g (after)e(the)h(broadcast)g(the)g(message)g(that)f Fl(a)h Fn(sen)o(t)g(b)q (efore)h(the)f(broadcast!)74 2602 y(In)f(summary)l(,)f(non-barrier)h(mo)q(de) g(routines)f(ma)o(y)g(exhibit)i(a)e(di\013eren)o(t)h(b)q(eha)o(vior)f(than)h (barrier)f(mo)q(de)949 2727 y(11)p eop %%Page: 12 12 bop 74 157 a Fn(routines,)14 b(when)f(nondeterministic)i(comm)o(unication)f (op)q(erations)f(are)f(allo)o(w)o(ed.)20 b(F)l(urthermore,)12 b(ev)o(en)74 214 y(if)k(only)f(deterministic)i(comm)o(unication)f(op)q (erations)g(are)e(used,)i(a)e(co)q(de)i(that)f(w)o(ould)g(deadlo)q(c)o(k)h (with)74 270 y(barrier)23 b(mo)q(de)g(routines)g(ma)o(y)f(complete)h (successfully)i(on)e(some)f(executions)i(when)f(non-barrier)74 327 y(mo)q(de)17 b(routines.)26 b(Therefore,)16 b(barrier)i(mo)q(de)f(should) h(b)q(e)f(used)h(when)f(debugging)h(co)q(de)f(\(that)f(is,)i(in)74 383 y(iden)o(tifying)h(bugs)f(that)f(ma)o(y)f(not)h(b)q(e)h(caugh)o(t)f(in)h (non-barrier)g(mo)q(de\).)27 b(Once)18 b(a)f(co)q(de)h(is)g(kno)o(wn)f(to)74 439 y(op)q(erate)h(correctly)g(in)h(barrier)g(mo)q(de,)g(it)f(is)h(p)q (ossible)g(to)f(switc)o(h)g(to)g(non-barrier)h(mo)q(de)f(for)f(b)q(etter)74 496 y(p)q(erformance)22 b(\(pro)o(vided)g(that)e(the)i(barrier)f(mo)q(de)h (is)g(not)f(required)h(for)f(the)g(correctness)g(of)g(the)74 552 y(program\).)e(Finally)l(,)e(notice)e(that)g(it)h(is)f(alw)o(a)o(ys)g(p)q (ossible)i(to)e(use)g(an)g(explicit)j(sync)o(hronization)e(\(i.e.,)74 609 y(to)f(call)h(the)f(routine)h Fl(sync)p Fn(\))e(when)i(a)f(barrier)g(is)h (needed.)74 696 y(Some)d(CCL)g(op)q(erations)f(ha)o(v)o(e)h(seman)o(tics)g (that)f(imply)i(a)e(barrier,)h(while)h(others)e(do)h(not.)19 b(Op)q(erations)74 753 y(that)g(imply)h(a)f(barrier)h(are)e Fl(combine)p Fn(,)h Fl(concat)p Fn(,)g Fl(index)p Fn(,)g(and)h Fl(sync)p Fn(.)31 b(Th)o(us,)20 b(the)f(implemen)o(tation)74 809 y(of)g(these)g(op)q(erations)g(is)g(the)h(same)e(for)h(barrier)g(and)g (for)f(non-barrier)i(mo)q(des.)31 b(Other)19 b(op)q(erations)74 866 y(that)d(do)h(not)f(imply)i(a)e(barrier)h(are)f Fl(bcast)p Fn(,)g Fl(reduce)p Fn(,)g Fl(scatter)p Fn(,)f Fl(gather)p Fn(,)h Fl(prefix)p Fn(,)g(and)h Fl(shift)p Fn(.)23 b(In)74 922 y(non-barrier)15 b(mo)q(de,)g(the)g(latter)f(op)q(erations)h(\(except)g Fl(shift)p Fn(\))e(are)i(implemen)o(ted)h(with)f(a)f(fan-in)i(tree)74 979 y(to)i(a)g(particular)i(destination)f(or)f(a)g(fan-out)h(tree)f(from)g(a) g(particular)i(source.)30 b(F)l(or)18 b(implemen)o(ting)74 1035 y(these)k(op)q(erations)g(in)h(barrier)f(mo)q(de,)h(w)o(e)f(use)g(t)o(w) o(o)f(in)o(ternal)h(partial)h(sync)o(hronization)f(routines:)74 1092 y Fl(sync-in\(dest,G\))11 b Fn(whic)o(h)k(builds)g(a)e(fan-in)h(tree)g (in)g(the)f(group)g Fl(G)h Fn(that)e(con)o(v)o(erges)h(at)g(pro)q(cess)g Fl(dest)p Fn(,)74 1148 y(and)g Fl(sync-out\(src,G\))d Fn(whic)o(h)j(builds)h (a)e(fan-out)g(tree)g(in)h(the)f(group)g Fl(G)g Fn(that)g(div)o(erges)h(from) e(pro)q(cess)74 1205 y Fl(src)p Fn(.)22 b(A)16 b(barrier)g(mo)q(de)h(for)e (the)h(latter)g(op)q(erations)g(is)h(implemen)o(ted)h(b)o(y)e(p)q(erforming)g (a)g Fl(sync-in\(\))74 1261 y Fn(b)q(efore)f(a)f(fan-out)h(t)o(yp)q(e)f(op)q (eration)h(\(lik)o(e)g Fl(bcast)f Fn(and)h Fl(scatter)p Fn(\),)e(or)h(b)o(y)h (p)q(erforming)g(a)f Fl(sync-out\(\))74 1317 y Fn(after)d(a)h(fan-in)g(t)o (yp)q(e)g(op)q(eration)g(\(lik)o(e)h Fl(reduce)p Fn(,)e Fl(gather)p Fn(,)g(and)h Fl(prefix)p Fn(\).)18 b(Notice)12 b(that)f(for)g(the)h Fl(prefix)74 1374 y Fn(op)q(eration,)18 b(the)g(fan-out)f(tree)g(is)h(ro)q (oted)g(at)e(the)i(highest)g(rank)o(ed)g(pro)q(cess)g(in)g(the)g(group.)26 b(Finally)l(,)74 1430 y(regarding)d(the)f(Pro)q(cess)g(Group)g(creation)h(op) q(erations,)h Fl(partition\(\))c Fn(implies)25 b(a)d(barrier)g(mo)q(de)74 1487 y(within)17 b(the)e(paren)o(t)g(group)g(while)h Fl(group\(\))e Fn(do)q(es)i(not.)74 1640 y Fk(3.4)56 b(Dev)n(elop)17 b(and)j(Run)e(Mo)r(des) 74 1757 y Fn(In)h(addition)g(to)f(the)g(c)o(hoice)h(b)q(et)o(w)o(een)f (barrier)g(and)h(non-barrier)f(mo)q(des,)h(the)f(user)h(can)f(select)h(b)q (e-)74 1813 y(t)o(w)o(een)d Fh(develop)i Fn(and)e Fh(run)k Fn(mo)q(des.)i(In)16 b Fh(develop)g Fn(mo)q(de,)g(CCL)g(p)q(erforms)g(more)f (detailed)j(consistency)74 1869 y(c)o(hec)o(kings)f(of)e(the)h(parameters)f (that)h(the)g(user)g(sp)q(eci\014es)i(for)d(CCL)h(calls.)23 b(This)16 b(increases)h(the)f(o)o(v)o(er-)74 1926 y(head)h(of)g(the)g(CCL)g (op)q(erations)g(and)g(ma)o(y)f(require)h(more)g(comm)o(unication)g(b)q(et)o (w)o(een)h(pro)q(cesses.)25 b(In)74 1982 y Fh(run)17 b Fn(mo)q(de,)f(man)o(y) g(of)g(the)h(consistency)g(c)o(hec)o(kings)g(are)f(suppressed,)h(and)g(only)g (c)o(hec)o(kings)g(that)f(are)74 2039 y(required)22 b(b)o(y)e(the)g(seman)o (tics)h(of)f(the)g(op)q(erations)h(are)f(p)q(erformed.)35 b(The)21 b(default)g(in)g(CCL)g(is)f Fh(run)74 2095 y Fn(mo)q(de)d(and)g(the)g(mo)q (de)g(can)g(b)q(e)g(c)o(hanged)g(b)o(y)g(the)g(user.)24 b(Examples)18 b(of)e(consistency)i(c)o(hec)o(kings)f(that)74 2152 y(are)g(done)g(only)h(in) f Fh(develop)g Fn(mo)q(de)h(are)e(c)o(hec)o(king)i(the)f(correctness)g(of)f (the)h(source,)h(the)f(destination)74 2208 y(and)d(the)g(message)g(length)g (supplied)j(to)c(a)h(CCL)f(routine,)i(as)e(w)o(ell)i(as)f(c)o(hec)o(king)h (the)f(mem)o(b)q(ership)h(list)74 2265 y(of)g(a)g(Pro)q(cess)g(Group)g(to)f (b)q(e)i(v)m(alid.)74 2418 y Fk(3.5)56 b(No)18 b(T)-5 b(ags)20 b(for)e(CCL)i(Routines)74 2534 y Fn(In)i(certain)g(comm)o(unication)g (libraries,)i(the)d(predominan)o(t)g(mec)o(hanism)h(for)f(matc)o(hing)g (incoming)74 2591 y(messages)d(to)f(receiv)o(e)j(op)q(erations)e(is)h Fh(matching)g(by)g(tag)p Fn(,)g(rather)e(than)h Fh(matching)h(by)g(sour)n(c)n (e)p Fn(.)29 b(This)949 2727 y(12)p eop %%Page: 13 13 bop 74 157 a Fn(suggests)17 b(a)g(v)o(ersion)h(of)f(CCL)g(where)h(the)f(user) h(is)g(required)g(to)f(pro)o(vide)h(a)f(tag)g(for)g(eac)o(h)g(CCL)h(call;)74 214 y(this)d(approac)o(h)e(is)i(follo)o(w)o(ed,)f(for)f(example,)i(b)o(y)e (Express)h([30)o(].)19 b(The)c(requiremen)o(t)f(on)g(tags)f(is)h(that)g(all) 74 270 y(pro)q(cesses)i(that)e(execute)i(a)f(collectiv)o(e)i(comm)o (unication)f(call)g(pro)o(vide)g(the)f(same)g(tag)f(v)m(alue.)74 358 y(Assume)19 b(that)g(only)g(blo)q(c)o(king)i(p)q(oin)o(t-to-p)q(oin)o(t)e (comm)o(unications)h(are)f(used,)h(and)f(that)g(all)h(receiv)o(e)74 414 y(op)q(erations)e(matc)o(h)g(incoming)h(messages)e(b)o(y)h(tag,)f(with)i (no)f(wildcards.)29 b(F)l(urther)18 b(assume)f(that)h(no)74 471 y(t)o(w)o(o)c(messages)g(with)h(the)g(same)g(tag)f(can)h(b)q(e)h(sim)o (ultaneously)g(w)o(aiting)f(to)f(b)q(e)i(receiv)o(ed)g(b)o(y)f(the)g(same)74 527 y(receiv)o(er.)20 b(Then)14 b(the)f(co)q(de)h(is)g Fh(r)n(ac)n(e-fr)n(e)n (e)p Fn(.)k(It)13 b(follo)o(ws)g(that)g(CCL)g(can)h(b)q(e)g(implemen)o(ted)g (using)g(receiv)o(e-)74 583 y(b)o(y-tag,)h(pro)o(vided)h(that)f(the)h(user)g (guaran)o(tees)f(that)g(the)h(same)f(tag)g(v)m(alue)i(is)f(nev)o(er)g(sim)o (ultaneously)74 640 y(used)g(b)o(y)f(t)o(w)o(o)f(op)q(erations)h(that)g(in)o (v)o(olv)o(e)h(a)e(common)h(pro)q(cess.)74 728 y(This)d(condition)g(ma)o(y)e (b)q(e)h(a)o(wkw)o(ard)f(to)g(ful\014ll.)21 b(F)l(urthermore,)11 b(the)g(e\016cien)o(t)g(implemen)o(tation)h(of)f(some)74 784 y(collectiv)o(e)16 b(comm)o(unication)f(op)q(erations)g(\(suc)o(h)f(as)g(an)h (ascend)f(comm)o(unication)i(on)e(a)g(tree\))g(requires)74 840 y(matc)o(hing)i(b)o(y)g(more)f(than)h(tag;)f(e.g.,)g(using)h Fl(\(source,)23 b(tag\))15 b Fn(to)h(select)g(incoming)h(messages.)22 b(But,)74 897 y(then,)15 b(matc)o(hing-b)o(y-source)h(is)g(su\016cien)o(t,)f (and)h(the)f(tag)f(is)i(sup)q(er\015uous.)74 1071 y Fm(4)69 b(P)n(erformance)23 b(Issues)g(in)f(CCL)74 1204 y Fn(This)14 b(section)f(discusses)h(sev)o(eral)f(p)q(erformance)g(issues)h(and)f(describ) q(es)h(some)f(of)f(the)h(algorithms)g(used)74 1260 y(in)19 b(CCL.)e(The)h(design)h(of)e(comm)o(unication)i(algorithms)f(for)f(CCL)h (attempts)e(to)i(address)f(b)q(oth)h(the)74 1317 y(e\016ciency)j(and)e(the)h (p)q(ortabilit)o(y)g(of)f(the)g(comm)o(unication)h(op)q(erations.)32 b(In)20 b(the)g(discussion)h(of)d(the)74 1373 y(comm)o(unication)e (algorithms)f(in)h(this)g(section,)f(w)o(e)g(assume)g(one)h(pro)q(cess)f(p)q (er)h(pro)q(cessor.)74 1526 y Fk(4.1)56 b(Comm)n(unication)16 b(Mo)r(del)74 1643 y Fn(In)j(designing)g(algorithms)f(for)f(the)h(CCL)g(op)q (erations,)g(w)o(e)f(assume)h(a)f(mo)q(del)i(of)f(a)f(fully-connected)74 1699 y(message-passing)j(system)f(in)h(whic)o(h)g(eac)o(h)g(pro)q(cess)g(can) f(comm)o(unicate)h(directly)h(with)f(an)o(y)f(other)74 1756 y(pro)q(cess)d(and)g(ev)o(ery)g(pair)h(of)e(pro)q(cesses)h(are)g(equally)h (distan)o(t.)22 b(W)l(e)16 b(also)g(assume)g(that)f(eac)o(h)h(pro)q(cess)74 1812 y(can)f(send)h(one)f(message)g(and,)g(sim)o(ultaneously)l(,)h(receiv)o (e)g(another)f(message)g(in)h(the)f(same)g(comm)o(uni-)74 1869 y(cation)h(step.)j(\(This)d(is)g(usually)g(done)g(using)g(a)f Fl(send-receive)e Fn(op)q(eration)j(found)f(in)h(man)o(y)f(parallel)74 1925 y(systems.\))j(In)12 b(most)e(existing)j(message-passing)e(parallel)i (systems,)e(the)h(time)f(for)g(sending)h(an)g Fj(m)p Fn(-b)o(yte)74 1982 y(message)j(from)f(pro)q(cess)h Fj(p)g Fn(to)f(pro)q(cess)h Fj(q)r Fn(,)g(without)g(congestion,)g(can)g(b)q(e)g(mo)q(deled)i(as)d Fj(T)19 b Fn(=)13 b Fj(t)1709 1989 y Fb(s)1737 1982 y Fn(+)d Fj(mt)1838 1989 y Fb(c)1856 1982 y Fn(,)74 2038 y(where)i Fj(t)218 2045 y Fb(s)249 2038 y Fn(is)g(the)g(o)o(v)o(erhead)g(\(start-up)f(time\))h (asso)q(ciated)g(with)g(eac)o(h)g(send)h(and/or)e(receiv)o(e)i(op)q(eration,) 74 2095 y(and)g Fj(t)176 2102 y Fb(c)207 2095 y Fn(is)g(the)g(comm)o (unication)h(time)f(for)f(sending)j(eac)o(h)e(additional)h(b)o(yte)f(\(or)f (an)o(y)g(appropriate)h(data)74 2151 y(unit\).)74 2239 y(Suc)o(h)25 b(a)e(fully-connected)k(mo)q(del)d(addresses)g(emerging)h(trends)f(in)g(man)o (y)g(mo)q(dern)g(distributed-)74 2295 y(memory)g(parallel)i(computers)f(and)f (message-passing)h(comm)o(unication)g(en)o(vironmen)o(ts.)48 b(These)74 2352 y(trends)16 b(are)f(eviden)o(t)i(in)g(systems)e(suc)o(h)h(as) f(Thinking)j(Mac)o(hines')d(CM-5)h([29)o(],)f(In)o(tel's)h(P)o(aragon)e([31)o (],)74 2408 y(NCUBE's)19 b(nCUBE/2)h([27)o(],)f(MIT's)g(J-Mac)o(hine)i([14)o (],)f(IBM's)f(V)l(ulcan)i([7)o(,)e(32],)h(and)f(the)h(recen)o(tly)74 2465 y(announced)e(IBM's)e(Scalable)i(PO)o(WERparallel)g(System)f(1)f (\(SP1\),)f(and)i(in)g(en)o(vironmen)o(ts)g(suc)o(h)g(as)74 2521 y(Express)g([30)o(],)g(P)l(ARMA)o(CS)h([24)o(],)e(PICL)i([22)o(],)f(Zip) q(co)q(de)i([28)o(])d(and)i(V)l(en)o(us)g([2)o(].)25 b(These)18 b(systems)e(and)74 2577 y(en)o(vironmen)o(ts)k(generally)h(ignore)f(the)g(sp) q(eci\014c)i(structure)e(and)g(top)q(ology)f(of)h(the)g(comm)o(unication)949 2727 y(13)p eop %%Page: 14 14 bop 74 157 a Fn(net)o(w)o(ork)18 b(and)h(assume)g(a)g(fully-connected)j (collection)e(of)f(pro)q(cesses,)h(in)g(whic)o(h)g(eac)o(h)f(pro)q(cess)g (can)74 214 y(comm)o(unicate)d(directly)g(with)g(an)o(y)f(other)g(pro)q(cess) g(b)o(y)g(sending)i(and)e(receiving)i(messages.)j(The)15 b(fact)74 270 y(that)j(the)h(mo)q(del)h(do)q(es)f(not)f(assume)h(an)o(y)f(single)i(top) q(ology)e(mak)o(es)g(it)h(more)g(general)g(and)g(\015exible.)74 327 y(This)14 b(mo)q(del,)f(for)g(instance,)h(allo)o(ws)f(the)f(creation)i (of)e(algorithms)h(that)f(are)g(p)q(ortable)i(b)q(et)o(w)o(een)f(di\013er-)74 383 y(en)o(t)g(mac)o(hines,)g(that)f(can)h(op)q(erate)f(within)i(arbitrary)e (and)h(dynamic)g(subsets)g(of)f(pro)q(cesses,)h(and)g(that)74 439 y(can)18 b(op)q(erate)g(in)g(the)g(presence)h(of)e(faults)h(\(assuming)f (connectivit)o(y)i(is)f(main)o(tained\).)28 b(In)19 b(addition,)74 496 y(algorithms)14 b(dev)o(elop)q(ed)i(for)e(this)h(mo)q(del)g(can)f(also)h (b)q(e)f(helpful)j(in)e(designing)h(algorithms)e(for)g(sp)q(eci\014c)74 552 y(top)q(ologies.)74 640 y(T)l(o)k(examine)h(the)g(p)q(erformance)f(of)g (comm)o(unication)h(algorithms,)g(w)o(e)f(de\014ne)h(the)f(follo)o(wing)h (three)74 696 y(complexit)o(y)d(measures:)142 776 y Fi(\017)23 b Fj(C)220 783 y Ff(1)240 776 y Fn(:)18 b(the)13 b(n)o(um)o(b)q(er)f(of)g (comm)o(unication)h(steps)f(required)h(b)o(y)f(an)g(algorithm.)19 b Fj(C)1534 783 y Ff(1)1566 776 y Fn(is)13 b(an)f(imp)q(ortan)o(t)188 833 y(measure)k(when)h(the)f(comm)o(unication)h(start-up)f(time)g(is)h(high)g (relativ)o(e)g(to)e(the)i(transfer)e(time)188 889 y(of)d(one)h(unit)g(of)f (data)g(and)h(the)f(message)g(size)i(p)q(er)f(send/receiv)o(e)h(op)q(eration) e(is)h(relativ)o(ely)h(small.)142 977 y Fi(\017)23 b Fj(C)220 984 y Ff(2)240 977 y Fn(:)j(the)18 b(amoun)o(t)g(of)f(data)h(\(in)h(the)f (appropriate)g(unit)h(of)f(comm)o(unication:)27 b(b)o(ytes,)18 b(\015its,)h(or)188 1034 y(pac)o(k)o(ets\))d(transferred)h(in)i(sequence)f(b) o(y)g(an)o(y)f(pro)q(cess.)27 b Fj(C)1217 1041 y Ff(2)1255 1034 y Fn(is)18 b(an)f(imp)q(ortan)o(t)h(measure)f(when)188 1090 y(the)e(start-up)g(time)g(is)h(small)g(compared)f(to)g(the)g(message)g (size.)142 1178 y Fi(\017)23 b Fj(C)220 1185 y Ff(3)240 1178 y Fn(:)29 b(the)20 b(total)f(amoun)o(t)g(of)g(data)g(comm)o(unicated)h(o)o(v) o(er)f(the)h(net)o(w)o(ork)e(\(in)j(the)f(appropriate)188 1235 y(unit)c(of)f(comm)o(unication:)20 b(b)o(ytes,)15 b(\015its,)g(or)g(pac)o(k)o (ets\).)k(Measures)c Fj(C)1386 1242 y Ff(1)1422 1235 y Fn(and)g Fj(C)1542 1242 y Ff(2)1577 1235 y Fn(do)h(not)e(address)188 1291 y(the)f(issue)h(of)f(load)h(on)f(the)h(net)o(w)o(ork.)k Fj(C)878 1298 y Ff(3)911 1291 y Fn(also)c(considers)g(the)f(fact)g(that)g (comm)o(unicating)h(more)188 1347 y(data)g(o)o(v)o(er)h(the)g(net)o(w)o(ork)f (causes)h(the)h(net)o(w)o(ork)e(to)g(b)q(ecome)i(more)f(congested.)74 1458 y(Th)o(us,)g(under)h(the)g(fully-connected)i(mo)q(del,)e(an)f(algorithm) h(has)f(an)g(estimated)h(time)g(complexit)o(y)g(of)74 1514 y Fj(T)k Fn(=)14 b Fj(C)202 1521 y Ff(1)222 1514 y Fj(t)238 1521 y Fb(s)267 1514 y Fn(+)d Fj(C)345 1521 y Ff(2)365 1514 y Fj(t)381 1521 y Fb(c)399 1514 y Fn(.)21 b(The)16 b(term)g Fj(C)669 1521 y Ff(3)705 1514 y Fn(do)q(es)g(not)f(a\013ect)g(the)h (complexit)o(y)h(here,)e(but)h(ma)o(y)f(b)q(e)i(helpful)h(in)74 1570 y(estimating)d(the)g(congestion)f(b)q(eha)o(vior)h(of)g(a)f(real)h (parallel)h(mac)o(hine.)k(F)l(or)14 b(instance,)h(one)g(can)g(mo)q(del)74 1627 y Fj(T)27 b Fn(=)21 b Fj(C)216 1634 y Ff(1)236 1627 y Fj(t)252 1634 y Fb(s)284 1627 y Fn(+)14 b Fj(C)365 1634 y Ff(2)385 1627 y Fj(t)401 1634 y Fb(c)432 1627 y Fn(+)g Fj(f)5 b Fn(\()p Fj(C)558 1634 y Ff(3)578 1627 y Fn(\),)20 b(where)g(the)g(function)h Fj(f)k Fn(can)20 b(b)q(e)h(deriv)o(ed)g(for)f(a)f(giv)o(en)i(mac)o(hine)g(b)o (y)74 1683 y(curv)o(e-\014tting)16 b(exp)q(erimen)o(tal)g(results.)74 1771 y(It)f(should)h(b)q(e)f(noted)g(that)f(there)h(are)f(more)h(detailed)h (comm)o(unication)g(mo)q(dels,)f(suc)o(h)g(as)f(the)h(P)o(ostal)74 1827 y(mo)q(del)f([3)o(])f(and)f(the)h(LogP)g(mo)q(del)g([13],)f(whic)o(h)i (further)e(tak)o(e)g(in)o(to)h(accoun)o(t)f(that)g(a)h(receiving)h(pro)q (cess)74 1884 y(generally)22 b(completes)f(its)f Fl(receive)g Fn(op)q(eration)g(later)h(than)f(the)g(corresp)q(onding)i(sending)f(pro)q (cess)74 1940 y(\014nishes)c(its)e Fl(send)f Fn(op)q(eration.)20 b(Ho)o(w)o(ev)o(er,)14 b(designing)i(e\016cien)o(t)g(algorithms)f(for)f (these)i(mo)q(dels)f(seems)74 1997 y(to)f(b)q(e)h(more)f(complicated.)21 b(Another)14 b(imp)q(ortan)o(t)g(issue)i(is)f(the)f(uniformit)o(y)h(of)f(the) h(implemen)o(tation.)74 2053 y(F)l(or)20 b(example,)i(in)g(the)e(LogP)g(mo)q (del,)j(the)d(collectiv)o(e)i(comm)o(unication)g(algorithms)e(are)g(designed) 74 2110 y(based)c(on)g Fj(P)23 b Fn(\(the)15 b(n)o(um)o(b)q(er)i(of)e(pro)q (cessors\).)21 b(The)c(optimal)f(algorithms)g(for)f(t)o(w)o(o)g(distinct)i(v) m(alues)g(of)74 2166 y Fj(P)23 b Fn(are)17 b(sometimes)g(v)o(ery)f (di\013eren)o(t.)25 b(This)18 b(presen)o(ts)f(a)f(c)o(hallenge)i(if)g(the)f (goal)f(is)i(to)e(supp)q(ort)h(collec-)74 2223 y(tiv)o(e)i(comm)o(unication)g (algorithms)g(for)e(v)m(arious)i(sizes)h(pro)q(cess)f(groups)f(using)h(the)g (same)f(collectiv)o(e)74 2279 y(comm)o(unication)e(library)l(.)74 2429 y Fk(4.2)56 b(T)-5 b(unable)19 b(Algorithms)74 2546 y Fn(One)12 b(goal)f(in)h(the)f(design)h(of)f(the)g(algorithms)g(for)g(CCL)g(w) o(as)f(that)h(they)g(b)q(e)h(tunable,)g(that)f(is,)h(that)e(they)74 2602 y(exhibit)17 b(a)d(trade-o\013)g(b)q(et)o(w)o(een)h(the)g(di\013eren)o (t)g(comm)o(unication)h(complexit)o(y)f(measures.)20 b(This)15 b(goal)g(is)949 2727 y(14)p eop %%Page: 15 15 bop 74 157 a Fn(imp)q(ortan)o(t)15 b(for)g(suc)o(h)h(a)f(library)h(to)f(b)q (e)h(b)q(oth)g(p)q(ortable)g(and)g(e\016cien)o(t.)21 b(In)16 b(the)g(follo)o(wing)g(discussion,)74 214 y(w)o(e)f(use)g Fj(n)g Fn(to)f(denote)h(the)f(n)o(um)o(b)q(er)h(of)g(pro)q(cesses)g(\(pro)q (cessors\))f(in)o(v)o(olv)o(ed)h(in)h(a)e(CCL)h(op)q(eration,)f(and)74 270 y(w)o(e)k(use)h Fj(m)f Fn(to)g(denote)h(the)f(size)h(of)f(data)g(eac)o(h) g(pro)q(cess)h(has)f(initially)l(.)33 b(All)19 b(CCL)g(routines)f(can)h(b)q (e)74 327 y(implemen)o(ted)g(with)e Fj(C)482 334 y Ff(1)517 327 y Fn(=)f Fi(d)p Fn(log)647 337 y Ff(2)674 327 y Fj(n)p Fi(e)h Fn(comm)o(unication)h(steps,)e(whic)o(h)i(is)f(optimal)h(for)e(this)h (measure.)74 383 y(\(The)c Fl(shift)f Fn(routine)h(can)g(b)q(e)g(implemen)o (ted)h(optimally)g(in)f(one)g(comm)o(unication)h(step,)e(b)o(y)h(using)g(the) 74 439 y Fl(send-receive)g Fn(op)q(eration.\))19 b(Ho)o(w)o(ev)o(er,)13 b(when)h(designing)i(an)e(algorithm)g(to)g(minimize)i Fj(C)1672 446 y Ff(1)1692 439 y Fn(,)e(some)g(of)74 496 y(the)j(corresp)q(onding)g (terms)f(for)g Fj(C)678 503 y Ff(2)715 496 y Fn(or)g Fj(C)804 503 y Ff(3)840 496 y Fn(ma)o(y)g(not)g(b)q(e)i(optimal.)24 b(In)17 b(general,)g(there)g(are)f(tradeo\013s)74 552 y(b)q(et)o(w)o(een)g Fj(C)282 559 y Ff(1)317 552 y Fn(and)f Fj(C)437 559 y Ff(2)458 552 y Fn(,)f(or)h(b)q(et)o(w)o(een)h Fj(C)749 559 y Ff(1)784 552 y Fn(and)f Fj(C)904 559 y Ff(3)925 552 y Fn(.)74 640 y(As)k(an)h (example,)g(consider)h(the)e Fl(index)f Fn(op)q(eration.)33 b(A)19 b(straigh)o(tforw)o(ard)e(implemen)o(tation)k(of)d(the)74 696 y Fl(index)11 b Fn(op)q(eration)g(can)g(b)q(e)h(ac)o(hiev)o(ed)g(with)g Fj(C)852 703 y Ff(1)885 696 y Fn(=)h Fj(n)r Fi(\000)r Fn(1)e(comm)o (unication)h(steps)f(and)h Fj(C)1573 703 y Ff(2)1606 696 y Fn(=)h Fj(m)p Fn(\()p Fj(n)r Fi(\000)r Fn(1\))p Fj(=n)74 753 y Fn(units)22 b(of)g(data.)38 b(This)22 b(is)h(simply)f(b)o(y)g(sending)h (the)e(data)g(directly)i(from)e(source)h(to)f(destination.)74 809 y(Namely)l(,)15 b(eac)o(h)g(pro)q(cess)g(sends)h Fj(m=n)e Fn(units)i(of)e(data)g(to)h Fj(n)9 b Fi(\000)h Fn(1)15 b(other)f(pro)q (cesses.)20 b(Another)15 b(approac)o(h)74 866 y(is)g(b)o(y)f(using)h(a)e (di\013eren)o(t)i(radix)f(for)g(represen)o(ting)g(the)g(PIDs)g(of)g(the)g (pro)q(cesses)h(\(the)f(straigh)o(tforw)o(ard)74 922 y(approac)o(h)20 b(is)g(using)g(a)g(radix)g Fj(n)g Fn(represen)o(tation\).)33 b(In)21 b(the)f(case)g(of)f(a)g(radix)h(2)g(represen)o(tation)g(w)o(e)74 979 y(get)g Fj(C)187 986 y Ff(1)228 979 y Fn(=)h Fi(d)p Fn(log)8 b Fj(n)p Fi(e)20 b Fn(comm)o(unication)h(steps)f(and)g Fj(C)1003 986 y Ff(2)1044 979 y Fi(\031)h Fn(\()p Fj(m)8 b Fn(log)g Fj(n)p Fn(\))p Fj(=)p Fn(2)19 b(units)i(of)e(data.)34 b(In)20 b(general,)74 1035 y(b)o(y)e(c)o(ho)q(osing)g(an)g(appropriate)f(radix)h Fj(r)q Fn(,)g(the)g Fl(index)f Fn(op)q(eration)h(can)f(b)q(e)i(implemen)o (ted)g(with)f Fj(C)1796 1042 y Ff(1)1833 1035 y Fi(\031)74 1092 y Fn(\()p Fj(r)8 b Fi(\000)f Fn(1\))h(log)270 1103 y Fb(r)297 1092 y Fj(n)14 b Fn(comm)o(unication)h(steps)e(and)i(with)f Fj(C)988 1099 y Ff(2)1021 1092 y Fi(\031)f Fn(\()p Fj(m)p Fn(\()p Fj(r)7 b Fi(\000)g Fn(1\))h(log)1322 1103 y Fb(r)1349 1092 y Fj(n)p Fn(\))p Fj(=r)14 b Fn(units)h(of)e(data.)19 b(Hence,)74 1148 y(it)c(is)f(p)q(ossible)i(to)e(implemen)o(t)h(a)f(parameterized)g (algorithm)h(whic)o(h)g(can)f(b)q(e)h(tuned)f(according)h(to)e(the)74 1205 y(start-up)h(time)h Fj(t)367 1212 y Fb(s)386 1205 y Fn(,)f(p)q(er-b)o (yte)h(transfer)e(time)i Fj(t)879 1212 y Fb(c)897 1205 y Fn(,)f(the)h (message)f(size)h Fj(m)p Fn(,)f(and)h(p)q(ossibly)h(the)e(n)o(um)o(b)q(er)h (of)74 1261 y(\\parallel)i(p)q(orts")d(that)h(can)g(supp)q(ort)g(concurren)o (t)h(sends)f(and)h(receiv)o(es)g(e\013ectiv)o(ely)g(\(see)f([12)o(]\).)74 1349 y(As)20 b(a)f(second)i(example,)g(consider)g(the)f(broadcasting)g (problem.)34 b(The)20 b(algorithm)g(for)f(the)h Fl(bcast)74 1405 y Fn(routine)g(is)g(straigh)o(tforw)o(ard)d(when)j(the)g(size)g(of)f (the)h(data)f(is)h Fj(m)f Fn(=)i(1,)e(that)g(is,)i(when)f(the)f(source)74 1461 y(of)e(the)g(broadcast)g(has)g(one)g(item)g(to)g(broadcast.)25 b(In)18 b(this)f(case,)h(a)e(divide-and-conque)q(r)k(algorithm)74 1518 y(pro)o(vides)15 b(an)g(optimal)g(solution,)g(also)g(kno)o(wn)g(as)f Fh(r)n(e)n(cursive)h(doubling)p Fn(.)k(Ho)o(w)o(ev)o(er,)14 b(the)h(broadcasting)74 1574 y(problem)f(b)q(ecomes)f(m)o(uc)o(h)g(more)g (complicated)h(when)f(there)g(are)g Fj(m)f(>)h Fn(1)g(units)h(of)e(data)g(to) h(broadcast.)74 1631 y(In)19 b(this)f(case,)h(a)e(lo)o(w)o(er)h(b)q(ound)h (on)f(the)g(time)g(is)h Fi(d)p Fn(log)8 b Fj(n)p Fi(e)g Fj(t)1102 1638 y Fb(s)1133 1631 y Fn(+)k(\()p Fj(m)g Fn(+)g Fi(d)p Fn(log)d Fj(n)p Fi(e)j(\000)g Fn(1\))p Fj(t)1547 1638 y Fb(c)1564 1631 y Fn(.)29 b(The)18 b(common)74 1687 y(divide-and-conquer)f(algorithm)d(based) g(on)g(a)g(binomial)i(tree)e(tak)o(es)f Fi(d)p Fn(log)c Fj(n)p Fi(e)f Fn(\()p Fj(t)1464 1694 y Fb(s)1490 1687 y Fn(+)g Fj(mt)1589 1694 y Fb(c)1607 1687 y Fn(\))13 b(time,)i(whic)o(h)74 1744 y(ma)o(y)j(b)q(e)h(far)e(from)h(optimal.)29 b(When)19 b Fj(n)f Fn(is)h(a)f(p)q(o)o(w)o(er)g(of)g(t)o(w)o(o,)f(an)h(algorithm)g(based)h(on)f (log)9 b Fj(n)18 b Fn(edge-)74 1800 y(disjoin)o(t)e(spanning)h(trees)e(on)g (a)g(\(log)8 b Fj(n)p Fn(\)-cub)q(e)16 b(is)g(giv)o(en)g(in)g([26)o(,)f(25].) 20 b(This)c(algorithm)f(requires)h Fj(C)1800 1807 y Ff(2)1833 1800 y Fn(=)74 1857 y Fj(m)c Fn(+)h(log)8 b Fj(n)13 b Fi(\000)f Fn(1,)19 b(whic)o(h)g(is)g(optimal,)h(and)e(tak)o(es)g Fj(T)24 b Fn(=)18 b(\()1081 1823 y Fi(p)p 1119 1823 74 2 v 34 x Fj(mt)1175 1864 y Fb(c)1205 1857 y Fn(+)1252 1822 y Fi(p)p 1290 1822 129 2 v 35 x Fn(log)8 b Fj(nt)1399 1864 y Fb(s)1419 1857 y Fn(\))1437 1840 y Ff(2)1474 1857 y Fn(time.)30 b(F)l(or)18 b(arbitrary)74 1913 y(v)m(alues)g(of)f Fj(n)p Fn(,)h(an)f(algorithm)g(based)g(on)g(the)g (generalized)i(Fib)q(onacci)g(trees)e(w)o(as)f(giv)o(en)h(in)h([10)o(])f (with)74 1970 y Fj(C)106 1977 y Ff(2)149 1970 y Fi(\024)24 b Fj(m)14 b Fn(+)h(log)8 b Fj(n)15 b Fn(+)f(3)8 b(log)g(log)g Fj(n)15 b Fn(+)g(15.)38 b(More)21 b(recen)o(tly)l(,)i(an)f(algorithm)f(based) h(on)f(edge-disjoin)o(t)74 2026 y(spanning)g(trees)e(of)g(cascaded)g (decreasing-size)j(h)o(yp)q(ercub)q(es)f(w)o(as)d(giv)o(en)i([4)o(])f(with)h (nearly)g(optimal)74 2082 y Fj(C)106 2089 y Ff(2)139 2082 y Fn(=)13 b Fj(m)6 b Fn(+)g Fi(d)q Fn(log)i Fj(n)p Fi(e)q Fn(.)19 b(The)13 b Fl(reduce)g Fn(op)q(eration)g(is)h(usually)h(implemen)o(ted)g(in)f (a)f(similar)h(manner)g(to)e(the)74 2139 y Fl(bcast)j Fn(op)q(eration)g(b)o (y)g(rev)o(ersing)h(the)f(\015o)o(w)g(of)f(the)i(messages.)74 2227 y(As)11 b(another)f(example,)j(consider)e(the)g Fl(combine)f Fn(and)h Fl(prefix)f Fn(op)q(erations.)19 b(These)11 b(op)q(erations)g (exhibit)74 2283 y(an)17 b(in)o(teresting)g(trade-o\013)f(b)q(et)o(w)o(een)h (measures)f Fj(C)954 2290 y Ff(1)991 2283 y Fn(and)h Fj(C)1113 2290 y Ff(3)1133 2283 y Fn(.)24 b(On)17 b(one)g(hand,)g(these)g(op)q (erations)f(can)74 2339 y(b)q(e)22 b(implemen)o(ted)h(with)f Fj(C)559 2346 y Ff(1)602 2339 y Fn(=)h(2)8 b Fi(d)p Fn(log)g Fj(n)p Fi(e)22 b Fn(and)f Fj(C)972 2346 y Ff(3)1016 2339 y Fn(=)i Fj(O)q Fn(\()p Fj(mn)p Fn(\))e(using)h(a)f(reduction)h(tree)f(follo)o (w)o(ed)74 2396 y(b)o(y)c(a)f(broadcast)f(tree.)24 b(On)17 b(the)f(other)g(hand,)h(these)g(op)q(erations)g(can)f(b)q(e)h(implemen)o(ted) h(with)f Fj(C)1798 2403 y Ff(1)1833 2396 y Fn(=)74 2452 y Fi(d)p Fn(log)9 b Fj(n)p Fi(e)i Fn(and)g Fj(C)335 2459 y Ff(3)368 2452 y Fn(=)i Fj(O)q Fn(\()p Fj(mn)8 b Fn(log)f Fj(n)p Fn(\))k(using)h(a)e (butter\015y-t)o(yp)q(e)h(or)g(circulan)o(t-graph-t)o(yp)q(e)h(comm)o (unication)74 2509 y(pattern.)41 b(In)23 b(fact,)g(h)o(ybrid)g(algorithms)f (for)f(these)i(op)q(erations)f(exist.)42 b(F)l(or)21 b(instance,)k(a)d(h)o (ybrid)74 2565 y(algorithm)c(for)f(the)h Fl(combine)f Fn(op)q(eration)h (requires)h Fj(C)1034 2572 y Ff(1)1071 2565 y Fn(=)e Fi(d)p Fn(log)9 b Fj(n)p Fi(e)j Fn(+)g Fj(k)19 b Fn(comm)o(unication)g(steps)e(and) 949 2727 y(15)p eop %%Page: 16 16 bop 74 157 a Fn(comm)o(unicates)16 b Fj(C)401 164 y Ff(3)433 157 y Fn(=)d Fj(m)534 139 y Fb(n)p 526 146 37 2 v 526 175 a Ff(2)544 166 y Fa(k)568 157 y Fn(\(log)8 b Fj(n)i Fn(+)h(2)758 141 y Fb(k)q Ff(+1)834 157 y Fi(\000)f Fj(k)i Fi(\000)e Fn(2\))15 b(units)h(of)e(data)h(\(see)g([1)o(]\).)74 245 y(The)k Fl(scatter)e Fn(and)i Fl(gather)e Fn(op)q(erations)h(resem)o(ble)h(the)g Fl(bcast)e Fn(and)i Fl(reduce)e Fn(op)q(erations)i(in)g(their)74 301 y(functionalit)o(y)l(.)h(Ho)o(w)o(ev)o(er,)12 b(in)h(terms)e(of)h(their)g (p)q(erformance,)h(these)f(op)q(erations)g(can)g(b)q(e)h(implemen)o(ted)74 358 y(with)e Fj(C)205 365 y Ff(1)238 358 y Fn(=)h Fi(d)q Fn(log)c Fj(n)p Fi(e)j Fn(comm)o(unication)f(steps)h(and)f Fj(C)966 365 y Ff(3)999 358 y Fi(\031)j Fn(\()p Fj(m)8 b Fn(log)f Fj(n)p Fn(\))p Fj(=)p Fn(2)j(units)h(of)e(data,)i(or,)f(alternativ)o(ely)l(,)74 414 y(they)j(can)h(b)q(e)g(implemen)o(ted)h(with)e Fj(C)716 421 y Ff(1)749 414 y Fn(=)g Fj(n)6 b Fi(\000)g Fn(1)14 b(comm)o(unication)g (steps)f(and)g Fj(C)1453 421 y Ff(3)1486 414 y Fn(=)g Fj(m)p Fn(\()p Fj(n)6 b Fi(\000)g Fn(1\))p Fj(=n)13 b Fn(units)74 471 y(of)i(data.)74 558 y(Finally)l(,)i(consider)g(the)f Fl(concat)f Fn(op)q(eration.)21 b(The)16 b(next)g(subsection)h(outlines)g(an)f(algorithm) f(for)g(the)74 615 y Fl(concat)j Fn(op)q(eration)g(that)g(is)h(optimal)g(in)g (all)h(the)e(three)h(complexit)o(y)g(measures,)g(hence,)h(it)e(enables)74 671 y(go)q(o)q(d)d(p)q(erformance)h(o)o(v)o(er)e(a)h(v)m(ariet)o(y)g(of)g (parallel)i(mac)o(hines.)74 824 y Fk(4.3)56 b(An)19 b(Optimal)e(Algorithm)f (for)j(Concatenation)74 941 y Fn(Here,)f(w)o(e)g(outline)g(an)g(algorithm)f (for)g(the)h Fl(concat)f Fn(op)q(eration)h(whic)o(h)g(is)g(optimal)g(with)g (resp)q(ect)g(to)74 997 y(measures)13 b Fj(C)299 1004 y Ff(1)319 997 y Fn(,)g Fj(C)377 1004 y Ff(2)411 997 y Fn(and)g Fj(C)529 1004 y Ff(3)549 997 y Fn(.)19 b(In)14 b(the)f Fl(concat)f Fn(\(all-to-all)j (broadcast\))d(op)q(eration)h(among)f Fj(n)h Fn(pro)q(cesses,)74 1054 y(eac)o(h)k(pro)q(cess)g(has)f(a)g(\014xed-size)j(message)d(\(also)g (called)i(a)f Fh(data)h(blo)n(ck\))f Fn(that)f(it)h(needs)g(to)f(broadcast)74 1110 y(to)e(the)g(other)g Fj(n)8 b Fi(\000)h Fn(1)k(pro)q(cesses.)20 b(Th)o(us,)14 b(at)g(the)g(end,)h(eac)o(h)f(pro)q(cess)h(has)f(all)h Fj(n)f Fn(data)g(blo)q(c)o(ks.)20 b(Because)74 1167 y Fj(n)c Fn(need)g(not)f(necessarily)i(b)q(e)g(a)e(p)q(o)o(w)o(er)g(of)g(t)o(w)o(o,)f (the)h(simple)i(optimal)f(algorithm)g(for)f(concatenation,)74 1223 y(whic)o(h)h(is)g(based)f(on)h(a)e(butter\015y-t)o(yp)q(e)i(comm)o (unication)g(pattern,)e(cannot)h(b)q(e)h(directly)h(used.)74 1311 y(The)i(optimal)g(algorithm)f(for)g(the)h Fl(concat)e Fn(op)q(eration)i(is)g(based)g(on)f(the)h(structure)f(of)g(a)g(circulan)o(t) 74 1367 y(graph.)h(A)14 b Fh(cir)n(culant)h(gr)n(aph)j Fj(G)p Fn(\()p Fj(n;)8 b(S)s Fn(\))k(is)j(c)o(haracterized)g(b)o(y)f(t)o(w)o(o)e (parameters:)19 b(the)14 b(n)o(um)o(b)q(er)g(of)g(no)q(des)74 1424 y Fj(n)p Fn(,)h(and)g(a)f(set)h(of)f(in)o(teger)h(o\013sets)f Fj(S)s Fn(.)19 b(The)c(no)q(des)h(of)e(the)h(graph)g(are)f(lab)q(eled)j(from) d(0)g(through)h Fj(n)10 b Fi(\000)f Fn(1.)74 1480 y(Eac)o(h)20 b(no)q(de)h Fj(i)f Fn(is)g(connected)h(to)f(no)q(des)g(of)g(the)g(form)g (\(\()p Fj(i)12 b Fi(\006)i Fj(s)p Fn(\))e(mo)q(d)h Fj(n)p Fn(\))20 b(for)g(all)h Fj(s)g Fi(2)g Fj(S)h Fn(\(see)f([17)o(]\).)74 1537 y(Circulan)o(t)f(graphs)e(are)g(an)h(imp)q(ortan)o(t)f(class)h(of)g(net) o(w)o(orks)e(whic)o(h)j(can)f(b)q(e)g(used)g(as)g(fault-toleran)o(t)74 1593 y(net)o(w)o(orks)g(for)h(man)o(y)g(other)f(net)o(w)o(orks)h([9)o(,)g(16) o(].)35 b(W)l(e)20 b(use)h(a)f(circulan)o(t)h(graph)f(with)g(p)q(o)o(w)o (er-of-t)o(w)o(o)74 1649 y(o\013sets)c Fj(S)i Fn(=)d Fi(f)p Fn(1)p Fj(;)8 b Fn(2)p Fj(;)g Fn(4)p Fj(;)g Fi(\001)f(\001)g(\001)t Fj(;)h Fn(2)566 1633 y Fb(k)q Fq(\000)p Ff(1)632 1649 y Fi(g)p Fn(,)16 b(where)h Fj(k)g Fn(=)e Fi(d)q Fn(log)8 b Fj(n)p Fi(e)p Fn(,)17 b(as)g(a)f(structure)h(for)f(the)h Fl(concat)f Fn(algorithm)74 1706 y([11)o(])f(\(see)g(Figure)h(1\).)74 1794 y(The)g(algorithm)f(consists)g (of)g Fj(k)h Fn(steps.)k(In)c(step)f(0,)g(eac)o(h)g(pro)q(cess)g Fj(i)g Fn(sends)h(its)f(data)g(blo)q(c)o(k)h(to)e(pro)q(cess)74 1850 y(\()p Fj(i)c Fi(\000)g Fn(1\))i(mo)q(d)h Fj(n)p Fn(,)i(receiv)o(es)i(a) e(data)f(blo)q(c)o(k)i(from)f(pro)q(cess)h(\()p Fj(i)9 b Fn(+)i(1\))h(mo)q(d) h Fj(n)p Fn(,)i(and)g(app)q(ends)i(the)e(receiv)o(ed)74 1906 y(data)f(blo)q(c)o(k)h(to)f(its)g(curren)o(t)g(data.)19 b(In)c(general,)g(in) g(step)g Fj(j)s Fn(,)e(for)h(0)e Fi(\024)h Fj(j)i Fi(\024)e Fj(k)c Fi(\000)g Fn(2,)14 b(eac)o(h)g(pro)q(cess)h Fj(i)f Fn(sends)74 1963 y(all)j(its)f(curren)o(t)g(cum)o(ulated)h(2)606 1946 y Fb(j)640 1963 y Fn(data)f(blo)q(c)o(ks)g(to)g(pro)q(cess)g(\()p Fj(i)10 b Fi(\000)h Fn(2)1212 1946 y Fb(j)1230 1963 y Fn(\))i(mo)q(d)f Fj(n)p Fn(,)k(receiv)o(es)h(2)1607 1946 y Fb(j)1641 1963 y Fn(data)f(blo)q(c)o(ks)74 2019 y(from)f(pro)q(cess)i(\()p Fj(i)10 b Fn(+)h(2)455 2003 y Fb(j)473 2019 y Fn(\))h(mo)q(d)h Fj(n)p Fn(,)j(and)g(app)q(ends)h(the)g(receiv)o(ed)g(data)e(to)h(its)g(cum)o(ulated) h(data)e(blo)q(c)o(ks.)74 2076 y(In)h(the)f(last)g(step,)g(step)g Fj(k)10 b Fi(\000)h Fn(1,)j(eac)o(h)h(pro)q(cess)g Fj(i)g Fn(sends)h(only)f (the)g(last)g Fj(n)10 b Fi(\000)g Fn(2)1411 2059 y Fb(k)q Fq(\000)p Ff(1)1493 2076 y Fn(blo)q(c)o(ks)15 b(of)g(data)f(that)74 2132 y(it)j(cum)o(ulated)g(to)e(pro)q(cess)h(\()p Fj(i)11 b Fi(\000)g Fn(2)672 2116 y Fb(k)q Fq(\000)p Ff(1)738 2132 y Fn(\))h(mo)q(d)h Fj(n)p Fn(,)j(and)h(it)f(receiv)o(es)h(the)f(same)g(n)o(um)o(b)q(er)h(of)e (data)h(blo)q(c)o(ks)74 2189 y(from)f(pro)q(cess)g(\()p Fj(i)10 b Fn(+)g(2)452 2172 y Fb(k)q Fq(\000)p Ff(1)518 2189 y Fn(\))i(mo)q(d)h Fj(n)p Fn(.)20 b(Figure)c(2)f(presen)o(ts)g(an)g(example)h(of)f(this)g (algorithm.)74 2276 y(In)22 b(general,)i(all)e(the)g(homogeneous)g(op)q (erations)f(\(op)q(erations)h(for)f(whic)o(h)i(there)e(is)h(no)g(notion)g(of) 74 2333 y(a)16 b(distinct)i(source)f(and/or)f(destination,)h([19)o(]\))f(in)h (CCL,)f(can)h(b)q(e)g(implemen)o(ted)h(with)f(the)g(minimal)74 2389 y(n)o(um)o(b)q(er)11 b(of)f(start-ups)g(\()p Fi(d)p Fn(log)e Fj(n)p Fi(e)j Fn(comm)o(unication)g(steps\))f(using)i(the)e(same)g(circulan)o (t)i(graph)e(structure.)74 2446 y(Examples)19 b(of)e(other)h(suc)o(h)g(op)q (erations)h(are)e Fl(combine)p Fn(,)h Fl(index)p Fn(,)f(and)i Fl(sync)p Fn(.)28 b(An)18 b(algorithm)g(for)f(the)74 2502 y Fl(combine)f Fn(op)q(eration)h(with)g Fj(T)22 b Fn(=)16 b Fi(d)p Fn(log)8 b Fj(n)p Fi(e)g Fn(\()p Fj(t)843 2509 y Fb(s)873 2502 y Fn(+)k Fj(mt)976 2509 y Fb(c)993 2502 y Fn(\),)17 b(whic)o(h)h(is)f (optimal)h(with)f(resp)q(ect)g(to)g Fj(C)1741 2509 y Ff(1)1778 2502 y Fn(\(and)74 2559 y(also)e(to)g Fj(C)253 2566 y Ff(2)288 2559 y Fn(when)h Fj(m)c Fn(=)h(1\),)i(app)q(ears)g(in)h([5)o(].)949 2727 y(16)p eop %%Page: 17 17 bop 74 157 a Fk(4.4)56 b(Sp)r(ecialized)16 b(Algorithms)74 274 y Fn(F)l(or)g(certain)g(CCL)h(op)q(erations,)f(the)g(b)q(est-kno)o(wn)g (algorithms)h(for)e(the)h(case)h(when)f Fj(n)h Fn(is)f(a)g(p)q(o)o(w)o(er)g (of)74 331 y(2)f(ma)o(y)g(p)q(erform)f(b)q(etter)i(than)f(algorithms)g(for)f (the)i(general)f(case.)20 b(Since,)d(in)f(man)o(y)e(situations,)i Fj(n)f Fn(is)74 387 y(a)h(p)q(o)o(w)o(er)f(of)g(2,)h(it)g(is)g(w)o(orth)o (while)g(to)f(implemen)o(t)j(this)e(case)g(separately)l(.)22 b(F)l(or)15 b(instance,)h(the)g Fl(concat)74 443 y Fn(algorithm)f(for)g Fj(n)g Fn(that)f(is)i(a)f(p)q(o)o(w)o(er)f(of)h(t)o(w)o(o)f(can)h(use)g(a)g (w)o(ell-kno)o(wn)h(h)o(yp)q(ercub)q(e)h(recursiv)o(e)e(exc)o(hange)74 500 y(algorithm)j(whic)o(h)h(eliminates)h(shifting)g(lo)q(cal)f(arra)o(ys)e (at)g(the)i(end)f(of)g(the)h(op)q(eration.)29 b(As)18 b(another)74 556 y(example,)f(when)g Fj(n)g Fn(is)f(a)g(p)q(o)o(w)o(er)g(of)g(t)o(w)o(o,)f (the)h Fl(bcast)g Fn(algorithm)g(describ)q(ed)j(in)e([25)o(])f(is)h(substan)o (tially)74 613 y(simpler,)23 b(in)f(terms)e(of)g(lo)q(cal)i(data)e (structures)g(and)h(con)o(trol,)g(than)g(the)f(algorithm)h(for)f(arbitrary)74 669 y(v)m(alues)d(of)d Fj(n)i Fn(describ)q(ed)h(in)f([4)o(].)74 844 y Fm(5)69 b(Conclusions)74 976 y Fn(W)l(e)16 b(ha)o(v)o(e)f(describ)q(ed) i(the)e(main)h(issues)g(that)f(w)o(e)g(ha)o(v)o(e)g(encoun)o(tered)h(in)h (designing)f(and)g(implemen)o(t-)74 1033 y(ing)i(a)e(Collectiv)o(e)j(Comm)o (unication)e(Library)g(\(CCL\))g(for)f(the)h(recen)o(tly)h(announced)g(IBM)f (Scalable)74 1089 y(PO)o(WERparalel)d(System)f(1,)g(\(SP1\).)18 b(W)l(e)13 b(ha)o(v)o(e)f(fo)q(cused)i(on)f(three)g(no)o(v)o(el)g(asp)q(ects) g(in)h(the)e(design)i(and)74 1146 y(implemen)o(tation)21 b(of)d(CCL:)h(the)g (in)o(tro)q(duction)i(of)d(pro)q(cess)i(groups,)f(de\014nition)i(of)e(seman)o (tics)g(that)74 1202 y(ensures)f(correctness)f(and)h(the)f(design)i(of)e(no)o (v)o(el)g(algorithms)h(based)g(on)f(a)g(realistic)i(p)q(oin)o(t-to-p)q(oin)o (t)74 1259 y(comm)o(unication)c(mo)q(del.)21 b(Eac)o(h)14 b(of)g(these)h(no)o (v)o(el)f(asp)q(ects)h(suggests)f(in)o(teresting)h(a)o(v)o(en)o(ues)f(for)g (further)74 1315 y(learning)j(and)e(researc)o(h.)74 1468 y Fk(Ac)n(kno)n(wledgemen)n(ts)74 1585 y Fn(W)l(e)d(thank)g(the)g(follo)o(wing) h(p)q(eople:)19 b(Dan)12 b(F)l(ry)o(e)f(for)h(his)g(v)m(aluable)i(commen)o (ts)d(and)h(help)i(in)e(revising)h(the)74 1641 y(CCL)j(seman)o(tics)f(as)h (part)f(of)g(the)g(sp)q(ec)i(in)f([21)o(],)f(Magda)g(Konstan)o(tinidou)h(for) f(her)h(con)o(tributions)g(to)74 1698 y(the)c(initial)h(stage)e(of)g(the)h (CCL)f(design,)i(Eric)f(Leu)g(for)f(his)h(commen)o(ts)f(on)h(a)f(draft)g(of)g (this)h(pap)q(er,)g(man)o(y)74 1754 y(in)o(ternal)17 b(and)f(external)h (review)o(ers)f(of)f([21])g(for)h(their)g(v)m(aluable)i(commen)o(ts)d(to)h (the)g(CCL)g(seman)o(tics,)74 1811 y(and)g(Dragutin)e(P)o(etk)o(o)o(vic)h (for)g(his)h(constan)o(t)e(supp)q(ort)h(and)h(help.)949 2727 y(17)p eop %%Page: 18 18 bop 74 157 a Fm(App)r(endix:)30 b(List)23 b(of)g(CCL)f(routines)74 290 y Fn(CCL)14 b(consists)f(of)g(t)o(w)o(o)f(parts,)h(the)h(collectiv)o(e)h (comm)o(unication)f(routines)g(\()p Fl(CC)f Fn(part\))f(and)i(the)f(pro)q (cess)74 346 y(group)k(routines)h(\()p Fl(PG)e Fn(part)g(whic)o(h)i(is)g (called)h Fl(TG)e Fn(for)f(task)g(group)h(in)h(the)f(EUI)h(do)q(cumen)o(t.\)) 25 b(The)18 b Fl(CC)74 403 y Fn(routines)d(pro)o(vide)g(users)g(with)f(the)h (functionalit)o(y)g(of)f(op)q(erations)h(in)o(v)o(olving)h(collectiv)o(e)g (comm)o(unica-)74 459 y(tion.)23 b(The)17 b Fl(PG)f Fn(routines)g(pro)o(vide) h(users)f(with)h(the)f(functionalit)o(y)i(of)d(sp)q(ecifying)k(and)d (manipulating)74 516 y(groups.)74 603 y(The)g(a)o(v)m(ailable)g Fl(CC)f Fn(routines)h(are:)74 691 y Fl(bcast\(\))p Fn(:)23 b(Single)c(source)e(broadcast,)f(i.e.,)h(broadcast)g(a)f(message)h(from)f (one)i(pro)q(cess)f(to)f(all)i(tasks)74 747 y(in)e(a)f(group.)74 835 y Fl(reduce\(\))p Fn(:)k(Apply)c(an)g(asso)q(ciativ)o(e)g(\(but)f(not)g (necessarily)i(comm)o(utativ)o(e\))d(reduction)j(op)q(eration)f(on)74 891 y(all)h(the)g(pro)q(cesses)f(in)h(a)f(group,)g(and)g(place)h(the)f (reduction)i(result)e(in)h(one)g(user-sp)q(eci\014ed)h(pro)q(cess.)74 979 y Fl(combine\(\))p Fn(:)25 b(Apply)20 b(an)e(asso)q(ciativ)o(e)h(\(but)f (not)h(necessarily)g(comm)o(utativ)o(e\))f(reduction)h(op)q(eration)74 1035 y(on)e(all)h(the)f(pro)q(cesses)g(in)h(a)e(group,)h(and)g(place)h(the)f (reduction)h(result)f(in)h(eac)o(h)f(of)f(the)h(pro)q(cesses)h(in)74 1092 y(the)d(group.)74 1179 y Fl(scatter\(\))p Fn(:)k(Distribute)d(distinct)g (messages)f(from)f(a)h(single)i(source)e(to)f(eac)o(h)i(pro)q(cess)f(in)h(a)f (group.)74 1267 y Fl(gather\(\))p Fn(:)23 b(Gather)17 b(distinct)i(messages)e (from)f(eac)o(h)i(pro)q(cess)f(in)i(a)e(group)g(to)g(a)g(single)i (destination)74 1323 y(pro)q(cess.)74 1411 y Fl(concat\(\))p Fn(:)25 b(Concatenate)18 b(to)f(all)i(pro)q(cesses)g(in)g(a)f(group.)29 b(Eac)o(h)18 b(pro)q(cess)g(in)h(a)f(group)g(p)q(erforms)g(a)74 1467 y Fl(bcast)d Fn(within)h(the)f(group.)74 1555 y Fl(index\(\))p Fn(:)k(Eac)o(h)c(task)g(in)h(a)e(group)h(p)q(erforms)g(a)g Fl(scatter)f Fn(op)q(eration.)74 1643 y Fl(prefix\(\))p Fn(:)19 b(P)o(arallel)d(pre\014x)g(op)q(eration.)k(It)15 b(is)h(sometimes)f(called)i Fl(scan)p Fn(.)74 1730 y Fl(shift\(\))p Fn(:)i(Shift)d(data)e(up)i(or)f(do)o (wn)g(some)g(n)o(um)o(b)q(er)g(of)g(steps)g(in)h(a)f(group.)74 1818 y Fl(sync\(\))p Fn(:)k(Barrier)c(sync)o(hronization)i(in)f(a)f(group.)74 1905 y(The)h Fl(PG)e Fn(routines)i(are)f(summarized)h(b)q(elo)o(w.)74 1993 y Fl(group\(\))p Fn(:)h(Create)11 b(a)g(\(static\))g(pro)q(cess)h(group) f(b)o(y)h(explicitly)i(sp)q(ecifying)g(the)d(pro)q(cesses)h(participating)74 2049 y(in)k(the)f(group.)74 2137 y Fl(partition\(\))p Fn(:)23 b(De\014ne)c(a)e(\(dynamic\))h(pro)q(cess)g(group)f(b)o(y)h(partitioning)h (an)e(existing)i(group)e(based)74 2193 y(on)e(a)g(lo)q(cally)i(supplied)h(in) o(teger)d(lab)q(el.)74 2281 y Fl(getsize\(\))p Fn(:)k(Get)14 b(the)i(size)g(\(the)f(n)o(um)o(b)q(er)g(of)g(pro)q(cesses\))g(of)g(an)g (existing)h(pro)q(cess)g(group.)74 2369 y Fl(getmembers\(\))p Fn(:)i(Get)d(the)g(ordered)h(arra)o(y)e(of)g(pro)q(cess)i(ids)g(of)f(an)g (existing)h(pro)q(cess)f(group.)74 2456 y Fl(getrank\(\))p Fn(:)k(Get)14 b(the)i(rank)f(of)f(a)h(pro)q(cess)h(in)g(a)f(pro)q(cess)g (group.)74 2544 y Fl(getpid\(\))p Fn(:)k(Get)c(the)g(pro)q(cess)g(id)h(of)f (the)g(pro)q(cess)h(with)f(a)g(certain)h(rank)f(in)h(a)f(pro)q(cess)g(group.) 949 2727 y(18)p eop %%Page: 19 19 bop 74 157 a Fl(getlabel\(\))p Fn(:)20 b(Get)c(the)g(lab)q(el)i(\(whic)o(h)e (w)o(as)g(supplied)i(b)o(y)e(a)g(user)g(when)g(the)h(group)e(w)o(as)h (formed\))f(of)74 214 y(an)g(existing)h(pro)q(cess)g(group.)949 2727 y(19)p eop %%Page: 20 20 bop 74 157 a Fm(References)97 277 y Fn([1])22 b(A.)14 b(Aggarw)o(al)g(and)g (S.)h(Kipnis,)h(\\Message-Time)f(T)l(radeo\013)f(for)g(Com)o(bining)h(Data)f (in)h(Message-)168 334 y(P)o(assing)g(Systems",)f Fh(IBM)i(R)n(ese)n(ar)n(ch) f(R)n(ep)n(ort)p Fn(,)g(R)o(C-18349,)e(Septem)o(b)q(er)j(1992.)97 428 y([2])22 b(V.)d(Bala)h(and)f(S.)h(Kipnis,)i(\\Pro)q(cess)d(Groups:)29 b(a)19 b(mec)o(hanism)h(for)f(the)h(co)q(ordination)g(of)f(and)168 484 y(comm)o(unication)c(among)e(pro)q(cesses)i(in)g(the)g(V)l(en)o(us)g (collectiv)o(e)h(comm)o(unication)f(library",)f Fh(Pr)n(o-)168 541 y(c)n(e)n(e)n(dings)g(of)i(the)h(7th)g(International)e(Par)n(al)r(lel)g (Pr)n(o)n(c)n(essing)f(Symp)n(osium)p Fn(,)h(IEEE,)g(April)h(1993.)97 634 y([3])22 b(A.)d(Bar-No)o(y)f(and)i(S.)f(Kipnis,)j(\\Designing)e (broadcasting)f(algorithms)h(in)g(the)f(p)q(ostal)h(mo)q(del)168 691 y(for)f(message-passing)g(systems",)h(to)f(app)q(ear)h(in)h Fh(Mathematic)n(al)f(Systems)f(The)n(ory)p Fn(.)g(Also)h(ap-)168 747 y(p)q(eared)15 b(in)g Fh(Pr)n(o)n(c)n(e)n(e)n(dings)e(of)j(the)g(4th)g(A) o(nnual)f(A)o(CM)f(Symp)n(osium)i(on)f(Par)n(al)r(lel)g(A)o(lgorithms)h(and) 168 804 y(A)o(r)n(chite)n(ctur)n(es)p Fn(,)d(June)k(1992,)c(pp.)j(11{22.)97 898 y([4])22 b(A.)34 b(Bar-No)o(y)g(and)h(S.)f(Kipnis,)41 b(\\Broadcasting)35 b(m)o(ultiple)h(messages)e(in)i(sim)o(ultaneous)168 954 y(send/receiv)o(e)18 b(systems",)e(to)g(app)q(ear)h(in)h Fh(Discr)n(ete)f(Applie)n(d)h (Mathematics)p Fn(.)e(Also)h(app)q(eared)h(as)168 1010 y Fh(IBM)d(R)n(ese)n (ar)n(ch)h(R)n(ep)n(ort)p Fn(,)e(R)o(C-18352,)g(Septem)o(b)q(er)i(1992.)97 1104 y([5])22 b(A.)15 b(Bar-No)o(y)l(,)f(S.)i(Kipnis,)h(and)e(B.)g(Sc)o(hieb) q(er,)i(\\An)f(optimal)g(algorithm)f(for)g(computing)h(census)168 1161 y(functions)h(in)g(message-passing)g(systems",)e(to)h(app)q(ear)h(in)g Fh(Par)n(al)r(lel)g(Pr)n(o)n(c)n(essing)e(L)n(etters)p Fn(.)g(Also)168 1217 y(app)q(eared)g(as)g Fh(IBM)h(R)n(ese)n(ar)n(ch)f(R)n(ep)n(ort)p Fn(,)g(R)o(C-18527,)f(No)o(v)o(em)o(b)q(er)g(1992.)97 1311 y([6])22 b(A.)17 b(Beguelin,)i(J.)f(Dongarra,)e(A.)h(Geist,)g(R.)h(Manc)o (hek,)f(and)h(V.)f(Sunderam,)h(\\A)f(user's)g(guide)168 1367 y(to)e(PVM)g(P)o(arallel)i(Virtual)f(Mac)o(hine",)g Fh(ORNL)g(T)m(e)n(chnic)n (al)e(R)n(ep)n(ort)p Fn(,)i(ORNL/TM-11826,)e(Ma)o(y)168 1424 y(1992.)97 1518 y([7])22 b(J.)16 b(Bruc)o(k,)g(R.)g(Cypher,)g(L.)g(Gra)o(v)m (ano,)f(A.)h(Ho,)f(C.)h(T.)f(Ho,)h(S.)g(Kipnis,)h(S.)f(Konstan)o(tinidou,)h (M.)168 1574 y(Snir)g(and)g(E.)g(Upfal,)g(\\Surv)o(ey)g(of)f(routing)h (issues)g(for)f(the)h(V)l(ulcan)h(parallel)h(computer",)d Fh(IBM)168 1631 y(R)n(ese)n(ar)n(ch)f(R)n(ep)n(ort)p Fn(,)g(RJ-8839,)g(June)h(1992.)97 1724 y([8])22 b(J.)c(Bruc)o(k,)h(R.)g(Cypher,)g(P)l(.)f(Elustondo,)i(A.)e (Ho,)h(and)f(C.T.)g(Ho,)h(\\A)f(Prop)q(osal)g(for)g(Common)168 1781 y(Group)13 b(Structures)g(in)i(a)e(Collectiv)o(e)i(Comm)o(unication)e (Library",)h Fh(IBM)g(R)n(ese)n(ar)n(ch)g(R)n(ep)n(ort)p Fn(,)g(RJ-)168 1837 y(9421,)f(Marc)o(h)i(1993.)97 1931 y([9])22 b(J.)g(Bruc)o(k,)i(R.)e (Cypher,)i(and)f(C.T.)e(Ho,)i(\\E\016cien)o(t)g(fault-toleran)o(t)f(mesh)g (and)h(h)o(yp)q(ercub)q(es)168 1988 y(arc)o(hitectures",)g Fh(Pr)n(o)n(c)n(e)n(e)n(dings)d(of)j(the)g(1992)g(International)e(Symp)n (osium)i(on)f(F)m(ault-T)m(oler)n(ant)168 2044 y(Computing)p Fn(,)14 b(pp.)i(162{169.)74 2138 y([10])22 b(J.)16 b(Bruc)o(k,)h(R.)g (Cypher,)g(and)g(C.T.)f(Ho,)g(\\Multiple)j(message)d(broadcasting)h(with)g (generalized)168 2194 y(Fib)q(onacci)f(trees",)e Fh(Pr)n(o)n(c)n(e)n(e)n (dings)f(of)j(the)g(4th)g(IEEE)f(Symp)n(osium)h(on)f(Par)n(al)r(lel)g(and)h (Distribute)n(d)168 2251 y(Pr)n(o)n(c)n(essing)p Fn(,)c(Decem)o(b)q(er)k (1992,)d(pp.)j(424{431.)74 2345 y([11])22 b(J.)16 b(Bruc)o(k,)g(C.T.)f(Ho,)g (and)h(S.)g(Kipnis,)i(\\Concatenating)e(data)f(optimally)i(in)g (message-passing)168 2401 y(systems",)d Fh(IBM)h(R)n(ese)n(ar)n(ch)h(R)n(ep)n (ort)p Fn(,)e(RJ-9191,)h(Jan)o(uary)g(1993.)74 2495 y([12])22 b(J.)g(Bruc)o(k,)i(R.)e(Cypher,)i(C.T.)d(Ho,)i(and)g(S.)f(Kipnis,)k (\\E\016cien)o(t)c(algorithms)g(for)g(the)g(index)168 2551 y(op)q(eration)15 b(in)h(message-passing)f(systems",)f Fh(IBM)i(R)n(ese)n(ar) n(ch)f(R)n(ep)n(ort)p Fn(,)g(in)h(preparation.)949 2727 y(20)p eop %%Page: 21 21 bop 74 157 a Fn([13])22 b(D.)16 b(Culler,)k(R.)d(Karp,)h(D.)f(P)o(atterson,)g (A.)g(Saha)o(y)l(,)g(K.E.)h(Sc)o(hauser,)g(E.)f(San)o(tos,)g(R.)h(Subramo-) 168 214 y(nian,)g(and)g(T.)e(v)o(on)i(Eic)o(k)o(en,)g(\\LogP:)e(to)o(w)o (ards)g(a)h(realistic)i(mo)q(del)f(of)f(parallel)i(computation",)168 270 y Fh(Pr)n(o)n(c)n(e)n(e)n(dings)c(of)k(the)f(4th)h(SIGPLAN)d(Symp)n (osium)j(on)f(Principles)f(and)h(Pr)n(actic)n(es)f(of)h(Par)n(al)r(lel)168 327 y(Pr)n(o)n(gr)n(amming)p Fn(,)c(A)o(CM,)g(Ma)o(y)g(1993.)74 420 y([14])22 b(W.)14 b(J.)h(Dally)l(,)g(A.)g(Chien,)g(S.)g(Fisk)o(e,)g(W.)f (Horw)o(at,)f(J.)i(Keen,)h(M.)e(Lariv)o(ee,)h(R.)g(Lethin,)h(P)l(.)f(Nuth,) 168 477 y(S.)d(Wills,)j(P)l(.)d(Carric)o(k,)h(and)g(G.)f(Fyler)h(\\The)g (J-Mac)o(hine:)20 b(a)12 b(\014ne-grain)i(concurren)o(t)f(computer",)168 533 y Fh(Information)j(Pr)n(o)n(c)n(essing)d(89)p Fn(,)j(Elsevier)g(Science)h (Publishers,)g(IFIP)l(,)e(1989,)f(pp.)h(1147{1153.)74 627 y([15])22 b(J.)11 b(Dongarra,)g(R.)h(Hemp)q(el,)i(A.)d(Hey)l(,)i(and)f(D.)f(W)l(alk)o (er,)h(\\A)g(Prop)q(osal)g(for)f(a)h(user-lev)o(el,)i(message-)168 684 y(passing)23 b(in)o(terface)g(in)h(a)f(distributed)h(memory)e(en)o (vironmen)o(t",)j Fh(ORNL)e(T)m(e)n(chnic)n(al)e(R)n(ep)n(ort)p Fn(,)168 740 y(ORNL/TM-12231,)14 b(Octob)q(er)h(1992.)74 834 y([16])22 b(S.)f(Dutt)f(and)i(J.)f(P)l(.)g(Ha)o(y)o(es,)h(\\Designing)g (fault-toleran)o(t)f(systems)g(using)h(automorphisms",)168 890 y Fh(Journal)16 b(of)g(Par)n(al)r(lel)g(and)g(Distribute)n(d)h(Computing) p Fn(,)d(V)l(ol.)i(12,)e(1991,)g(pp.)h(249{268.)74 984 y([17])22 b(B.)11 b(Elspas)i(and)f(J.)g(T)l(urner,)h(\\Graphs)e(with)i(circulan)o(t)g (adjacency)f(matrices",)g Fh(Journal)i(of)f(Com-)168 1041 y(binatorial)j(The) n(ory)p Fn(,)e(No.)h(9,)f(1970,)g(pp.)h(297{307.)74 1134 y([18])22 b(B.)h(G.)f(Fitc)o(h)i(and)f(M.)g(E.)g(Giampapa,)h(\\The)g(V)l(ulcan)g(Op)q (eration)h(En)o(vironmen)o(t:)36 b(a)23 b(brief)168 1191 y(o)o(v)o(erview)14 b(and)h(status)e(rep)q(ort",)h Fh(Pr)n(o)n(c)n(e)n(e)n(dings)f(of)j(the)g (5th)g(Workshop)g(on)g(Use)f(of)h(Par)n(al)r(lel)e(Pr)n(o-)168 1247 y(c)n(essors)h(in)g(Mete)n(or)n(olo)n(gy)p Fn(,)f(ECMWF,)g(No)o(v)o(em)o (b)q(er)h(1992.)74 1341 y([19])22 b(G.)13 b(F)l(o)o(x)h(and)h(W.)e(F)l (urmanski,)i(\\Optimal)g(comm)o(unication)g(algorithms)g(for)e(regular)i (decomp)q(o-)168 1398 y(sitions)e(on)f(the)g(h)o(yp)q(ercub)q(e",)i Fh(Pr)n(o)n(c)n(e)n(e)n(dings)e(of)h(the)h(3r)n(d)g(Confer)n(enc)n(e)e(on)h (Hyp)n(er)n(cub)n(e)g(Concurr)n(ent)168 1454 y(Computers)j(and)g(Applic)n (ations)p Fn(,)e(A)o(CM,)g(1988,)g(pp.)h(648{713.)74 1548 y([20])22 b(G.)17 b(F)l(o)o(x,)g(M.)g(Johnson,)i(G.)e(Lyzenga,)h(S.)g(Otto,)g(J.)f (Salmon,)i(and)f(D.)f(W)l(alk)o(er,)h Fh(Solving)f(Pr)n(ob-)168 1604 y(lems)e(on)g(Concurr)n(ent)g(Pr)n(o)n(c)n(essors,)f(V)m(olume)h(I:)g (Gener)n(al)g(T)m(e)n(chniques)f(and)h(R)n(e)n(gular)h(Pr)n(oblems)p Fn(,)168 1661 y(Pren)o(tice-Hall,)g(Englew)o(o)q(o)q(d)g(Cli\013s,)f(New)g (Jersey)l(,)h(1988.)74 1755 y([21])22 b(D.)c(F)l(ry)o(e,)h(R.)f(Bry)o(an)o (t,)h(C.T.)f(Ho,)h(P)l(.)f(de)h(Jong,)h(R.)f(La)o(wrence,)g(and)g(M.)f(Snir,) j(\\An)d(External)168 1811 y(User)c(In)o(terface)g(for)f(Scalable)j(P)o (arallel)f(Systems:)k(F)o(OR)l(TRAN)c(In)o(terface",)f Fh(T)m(e)n(chnic)n(al) f(R)n(ep)n(ort)p Fn(,)168 1867 y(IBM)i(Highly)h(P)o(arallel)h(Sup)q (ercomputing)f(Systems)f(Lab)q(oratory)l(,)g(No)o(v)o(em)o(b)q(er)f(1992.)74 1961 y([22])22 b(G.)g(A.)h(Geist,)h(M.)f(T.)f(Heath,)j(B.)e(W.)f(P)o(eyton,)i (and)g(P)l(.)e(H.)h(W)l(orley)l(,)i(\\A)e(user's)g(guide)h(to)168 2018 y(PICL:)c(a)h(P)o(ortable)f(Instrumen)o(ted)h(Comm)o(unication)g (Library",)g Fh(ORNL)g(T)m(e)n(chnic)n(al)e(R)n(ep)n(ort)p Fn(,)168 2074 y(ORNL/TM-11616,)14 b(Octob)q(er)h(1990.)74 2168 y([23])22 b(M.)11 b(E.)g(Giampapa,)h(B.)g(G.)f(Fitc)o(h,)h(G.)f(R.)h(Irwin,)h (and)f(D.)f(G.)g(Shea,)i(\\V)l(ulcan)g(op)q(erating)f(en)o(viron-)168 2224 y(men)o(t:)27 b(programmer's)17 b(reference",)j Fh(Internal)e(Memor)n (andum)p Fn(,)i(IBM)g(T.J.)e(W)l(atson)g(Researc)o(h)168 2281 y(Cen)o(ter,)c(Marc)o(h)h(1991.)74 2375 y([24])22 b(R.)d(Hemp)q(el,)i(\\The)d (ANL/GMD)h(macros)f(\(P)l(ARMA)o(CS\))g(in)i(F)o(OR)l(TRAN)g(for)e(p)q (ortable)i(par-)168 2431 y(allel)h(programming)e(using)h(the)f(message)g (passing)h(programming)f(mo)q(del,)i(user's)e(guide)h(and)168 2488 y(reference)c(man)o(ual",)g Fh(T)m(e)n(chnic)n(al)e(Memor)n(andum)p Fn(,)j(Gesellsc)o(haft)f(f)q(\177)-24 b(ur)16 b(Mathematik)f(und)i(Daten-)168 2544 y(v)o(erab)q(eitung)e(m)o(bH,)g(W)l(est)g(German)o(y)l(.)949 2727 y(21)p eop %%Page: 22 22 bop 74 157 a Fn([25])22 b(C.T.)e(Ho,)j(\\Optimal)g(broadcasting)f(on)g(SIMD)g (h)o(yp)q(ercub)q(es)h(without)f(indirect)i(addressing)168 214 y(capabilit)o(y",)c Fh(Journal)f(of)h(Par)n(al)r(lel)e(and)i(Distribute)n (d)f(Computing)p Fn(,)g(V)l(ol.)g(13,)f(No.)g(2,)h(Octob)q(er)168 270 y(1991,)13 b(pp.)j(246{255.)74 364 y([26])22 b(S.)13 b(L.)h(Johnsson)g (and)g(C.T.)f(Ho,)g(\\Spanning)i(graphs)e(for)h(optim)o(um)f(broadcasting)h (and)g(p)q(erson-)168 420 y(alized)19 b(comm)o(unication)g(in)g(h)o(yp)q (ercub)q(es",)g Fh(IEEE)f(T)m(r)n(ansactions)f(on)i(Computers)p Fn(,)f(V)l(ol.)g(C-38,)168 477 y(No.)c(9,)h(Septem)o(b)q(er)h(1989,)d(pp)j (1249{1268.)74 571 y([27])22 b(J.)f(F.)g(P)o(almer)g(\\The)h(NCUBE)f(family)h (of)f(parallel)i(sup)q(ercomputers",)g Fh(Pr)n(o)n(c)n(e)n(e)n(dings)d(of)i (the)168 627 y(International)15 b(Confer)n(enc)n(e)f(on)i(Computer)h(Design)p Fn(,)d(IEEE,)h(1986.)74 721 y([28])22 b(A.)e(Skjellum)i(and)e(A.)g(P)l(.)g (Leung,)i(\\Zip)q(co)q(de:)32 b(a)20 b(p)q(ortable)h(m)o(ulticomputer)g(comm) o(unication)168 777 y(library)e(atop)e(the)h(Reactiv)o(e)h(Kernel",)h Fh(Pr)n(o)n(c)n(e)n(e)n(dings)d(of)i(the)g(5th)h(Distribute)n(d)f(Memory)g (Com-)168 834 y(puting)d(Confer)n(enc)n(e)p Fn(,)d(IEEE,)i(April)h(1990,)e (pp.)h(328{337.)74 928 y([29])22 b Fh(Conne)n(ction)j(Machine)h(CM-5)h(T)m(e) n(chnic)n(al)d(Summary)p Fn(,)30 b(Thinking)e(Mac)o(hines)g(Corp)q(oration,) 168 984 y(1991.)74 1078 y([30])22 b Fh(Expr)n(ess)15 b(3.0)i(Intr)n(o)n (ductory)f(Guide)p Fn(,)f(P)o(arasoft)f(Corp)q(oration,)g(1990.)74 1172 y([31])22 b Fh(Par)n(agon)16 b(XP/S)f(Overview)p Fn(,)g(In)o(tel)h(Corp) q(oration,)e(1991.)74 1266 y([32])22 b(\\V)l(ulcan)12 b(system)e(summary",)h Fh(Internal)f(Memor)n(andum)p Fn(,)i(IBM)f(T.J.)g(W)l(atson)f(Researc)o(h)h (Cen)o(ter.)949 2727 y(22)p eop %%Page: 23 23 bop 353 679 a 19537183 19800310 7762247 25786449 27299430 45586759 startTexFig 353 679 a %%BeginDocument: Figs/circ16.ps 50 dict begin /arrowHeight 8 def /arrowWidth 4 def /none null def /numGraphicParameters 17 def /stringLimit 65535 def /Begin { save numGraphicParameters dict begin } def /End { end restore } def /SetB { dup type /nulltype eq { pop false /brushRightArrow idef false /brushLeftArrow idef true /brushNone idef } { /brushDashOffset idef /brushDashArray idef 0 ne /brushRightArrow idef 0 ne /brushLeftArrow idef /brushWidth idef false /brushNone idef } ifelse } def /SetCFg { /fgblue idef /fggreen idef /fgred idef } def /SetCBg { /bgblue idef /bggreen idef /bgred idef } def /SetF { /printSize idef /printFont idef } def /SetP { dup type /nulltype eq { pop true /patternNone idef } { /patternGrayLevel idef patternGrayLevel -1 eq { /patternString idef } if false /patternNone idef } ifelse } def /BSpl { 0 begin storexyn newpath n 1 gt { 0 0 0 0 0 0 1 1 true subspline n 2 gt { 0 0 0 0 1 1 2 2 false subspline 1 1 n 3 sub { /i exch def i 1 sub dup i dup i 1 add dup i 2 add dup false subspline } for n 3 sub dup n 2 sub dup n 1 sub dup 2 copy false subspline } if n 2 sub dup n 1 sub dup 2 copy 2 copy false subspline patternNone not brushLeftArrow not brushRightArrow not and and { ifill } if brushNone not { istroke } if 0 0 1 1 leftarrow n 2 sub dup n 1 sub dup rightarrow } if end } dup 0 4 dict put def /Circ { newpath 0 360 arc patternNone not { ifill } if brushNone not { istroke } if } def /CBSpl { 0 begin dup 2 gt { storexyn newpath n 1 sub dup 0 0 1 1 2 2 true subspline 1 1 n 3 sub { /i exch def i 1 sub dup i dup i 1 add dup i 2 add dup false subspline } for n 3 sub dup n 2 sub dup n 1 sub dup 0 0 false subspline n 2 sub dup n 1 sub dup 0 0 1 1 false subspline patternNone not { ifill } if brushNone not { istroke } if } { Poly } ifelse end } dup 0 4 dict put def /Elli { 0 begin newpath 4 2 roll translate scale 0 0 1 0 360 arc patternNone not { ifill } if brushNone not { istroke } if end } dup 0 1 dict put def /Line { 0 begin 2 storexyn newpath x 0 get y 0 get moveto x 1 get y 1 get lineto brushNone not { istroke } if 0 0 1 1 leftarrow 0 0 1 1 rightarrow end } dup 0 4 dict put def /MLine { 0 begin storexyn newpath n 1 gt { x 0 get y 0 get moveto 1 1 n 1 sub { /i exch def x i get y i get lineto } for patternNone not brushLeftArrow not brushRightArrow not and and { ifill } if brushNone not { istroke } if 0 0 1 1 leftarrow n 2 sub dup n 1 sub dup rightarrow } if end } dup 0 4 dict put def /Poly { 3 1 roll newpath moveto -1 add { lineto } repeat closepath patternNone not { ifill } if brushNone not { istroke } if } def /Rect { 0 begin /t exch def /r exch def /b exch def /l exch def newpath l b moveto l t lineto r t lineto r b lineto closepath patternNone not { ifill } if brushNone not { istroke } if end } dup 0 4 dict put def /Text { ishow } def /idef { dup where { pop pop pop } { exch def } ifelse } def /ifill { 0 begin gsave patternGrayLevel -1 ne { fgred bgred fgred sub patternGrayLevel mul add fggreen bggreen fggreen sub patternGrayLevel mul add fgblue bgblue fgblue sub patternGrayLevel mul add setrgbcolor eofill } { eoclip originalCTM setmatrix pathbbox /t exch def /r exch def /b exch def /l exch def /w r l sub ceiling cvi def /h t b sub ceiling cvi def /imageByteWidth w 8 div ceiling cvi def /imageHeight h def bgred bggreen bgblue setrgbcolor eofill fgred fggreen fgblue setrgbcolor w 0 gt h 0 gt and { l b translate w h scale w h true [w 0 0 h neg 0 h] { patternproc } imagemask } if } ifelse grestore end } dup 0 8 dict put def /istroke { gsave brushDashOffset -1 eq { [] 0 setdash 1 setgray } { brushDashArray brushDashOffset setdash fgred fggreen fgblue setrgbcolor } ifelse brushWidth setlinewidth originalCTM setmatrix stroke grestore } def /ishow { 0 begin gsave fgred fggreen fgblue setrgbcolor /fontDict printFont findfont printSize scalefont dup setfont def /descender fontDict begin 0 [FontBBox] 1 get FontMatrix end transform exch pop def /vertoffset 0 descender sub printSize sub printFont /Courier ne printFont /Courier-Bold ne and { 1 add } if def { 0 vertoffset moveto show /vertoffset vertoffset printSize sub def } forall grestore end } dup 0 3 dict put def /patternproc { 0 begin /patternByteLength patternString length def /patternHeight patternByteLength 8 mul sqrt cvi def /patternWidth patternHeight def /patternByteWidth patternWidth 8 idiv def /imageByteMaxLength imageByteWidth imageHeight mul stringLimit patternByteWidth sub min def /imageMaxHeight imageByteMaxLength imageByteWidth idiv patternHeight idiv patternHeight mul patternHeight max def /imageHeight imageHeight imageMaxHeight sub store /imageString imageByteWidth imageMaxHeight mul patternByteWidth add string def 0 1 imageMaxHeight 1 sub { /y exch def /patternRow y patternByteWidth mul patternByteLength mod def /patternRowString patternString patternRow patternByteWidth getinterval def /imageRow y imageByteWidth mul def 0 patternByteWidth imageByteWidth 1 sub { /x exch def imageString imageRow x add patternRowString putinterval } for } for imageString end } dup 0 12 dict put def /min { dup 3 2 roll dup 4 3 roll lt { exch } if pop } def /max { dup 3 2 roll dup 4 3 roll gt { exch } if pop } def /arrowhead { 0 begin transform originalCTM itransform /taily exch def /tailx exch def transform originalCTM itransform /tipy exch def /tipx exch def /dy tipy taily sub def /dx tipx tailx sub def /angle dx 0 ne dy 0 ne or { dy dx atan } { 90 } ifelse def gsave originalCTM setmatrix tipx tipy translate angle rotate newpath 0 0 moveto arrowHeight neg arrowWidth 2 div lineto arrowHeight neg arrowWidth 2 div neg lineto closepath patternNone not { originalCTM setmatrix /padtip arrowHeight 2 exp 0.25 arrowWidth 2 exp mul add sqrt brushWidth mul arrowWidth div def /padtail brushWidth 2 div def tipx tipy translate angle rotate padtip 0 translate arrowHeight padtip add padtail add arrowHeight div dup scale arrowheadpath ifill } if brushNone not { originalCTM setmatrix tipx tipy translate angle rotate arrowheadpath istroke } if grestore end } dup 0 9 dict put def /arrowheadpath { newpath 0 0 moveto arrowHeight neg arrowWidth 2 div lineto arrowHeight neg arrowWidth 2 div neg lineto closepath } def /leftarrow { 0 begin y exch get /taily exch def x exch get /tailx exch def y exch get /tipy exch def x exch get /tipx exch def brushLeftArrow { tipx tipy tailx taily arrowhead } if end } dup 0 4 dict put def /rightarrow { 0 begin y exch get /tipy exch def x exch get /tipx exch def y exch get /taily exch def x exch get /tailx exch def brushRightArrow { tipx tipy tailx taily arrowhead } if end } dup 0 4 dict put def /midpoint { 0 begin /y1 exch def /x1 exch def /y0 exch def /x0 exch def x0 x1 add 2 div y0 y1 add 2 div end } dup 0 4 dict put def /thirdpoint { 0 begin /y1 exch def /x1 exch def /y0 exch def /x0 exch def x0 2 mul x1 add 3 div y0 2 mul y1 add 3 div end } dup 0 4 dict put def /subspline { 0 begin /movetoNeeded exch def y exch get /y3 exch def x exch get /x3 exch def y exch get /y2 exch def x exch get /x2 exch def y exch get /y1 exch def x exch get /x1 exch def y exch get /y0 exch def x exch get /x0 exch def x1 y1 x2 y2 thirdpoint /p1y exch def /p1x exch def x2 y2 x1 y1 thirdpoint /p2y exch def /p2x exch def x1 y1 x0 y0 thirdpoint p1x p1y midpoint /p0y exch def /p0x exch def x2 y2 x3 y3 thirdpoint p2x p2y midpoint /p3y exch def /p3x exch def movetoNeeded { p0x p0y moveto } if p1x p1y p2x p2y p3x p3y curveto end } dup 0 17 dict put def /storexyn { /n exch def /y n array def /x n array def n 1 sub -1 0 { /i exch def y i 3 2 roll put x i 3 2 roll put } for } def Begin [ 0.9 0 0 0.9 0 0 ] concat /originalCTM matrix currentmatrix def Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -31.2139 101.893 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 110.786 -40.107 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 70.107 61.2139 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 26.107 89.3209 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 99.3209 15 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 -1 -31.2139 1101.89 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 -1 70.107 1142.57 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 -1 26.107 1114.47 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 -1 99.3209 1188.79 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ -1 0 0 1 490.786 -40.107 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ -1 0 0 1 531.465 61.2139 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ -1 0 0 1 575.465 89.3209 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ -1 0 0 1 502.251 15 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ -1 0 0 -1 531.465 1142.57 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ -1 0 0 -1 575.465 1114.47 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ -1 0 0 -1 502.251 1188.79 ] concat 332 642 4 4 Elli End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 296.428 762 ] concat [ (0) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 363 744 ] concat [ (1) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 407 717 ] concat [ (2) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 435.107 671.893 ] concat [ (3) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 452 606.572 ] concat [ (4) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 443 549 ] concat [ (5) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 407 496 ] concat [ (6) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 363 469 ] concat [ (7) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 296.428 449.786 ] concat [ (8) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 230 469 ] concat [ (9) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 177 496 ] concat [ (10) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 151 540 ] concat [ (11) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 133 606.428 ] concat [ (12) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 151 673 ] concat [ (13) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 177 717 ] concat [ (14) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 221 744 ] concat [ (15) ] Text End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 313 729 370 716 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 370 716 414 688 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 414 688 444 641 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 443 643 455 588 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 455 588 443 533 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 443 533 412 486 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 412 487 370 458 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 370 458 313 446 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 313 446 256 458 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 256 458 211 486 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 211 486 182 532 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 182 533 171 588 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 170 588 182 643 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 182 643 211 689 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 211 689 255 718 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 313 729 455 587 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 369 716 443 531 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 413 688 413 487 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 443 643 369 459 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 455 588 312 447 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 443 533 255 459 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 414 486 210 487 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 370 458 182 533 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 313 446 171 588 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 255 459 182 643 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 211 485 211 688 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 183 532 256 718 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 256 718 313 730 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 172 588 313 730 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 183 643 370 717 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 212 689 413 688 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 256 717 443 643 Line End End %I eop showpage end %%EndDocument 353 679 a endTexFig 349 2031 a Fn(Figure)15 b(1:)20 b(A)15 b(circulan)o(t)h(graph)f(with)h(16)e (no)q(des)i(and)f(o\013sets)f(1)h(and)h(4.)949 2727 y(23)p eop %%Page: 24 24 bop -92 619 a 33564236 36585014 1184071 1512980 35719495 39469056 startTexFig -92 619 a %%BeginDocument: Figs/concat.ps /Times-Roman findfont 12 scalefont setfont 407.929993 448.610016 moveto (After round 2) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (After round 2) show 190.970001 448.610016 moveto (After round 1) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (After round 1) show 407.929993 575.169983 moveto (After round 0) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (After round 0) show 0.400000 setlinewidth newpath 109.609993 674.609985 moveto 109.609993 584.210022 lineto 127.689995 584.210022 lineto 127.689995 674.609985 lineto closepath stroke newpath 109.609993 656.530029 moveto closepath stroke newpath 109.609993 656.530029 moveto 127.689995 656.530029 lineto stroke newpath 109.609993 638.450012 moveto 127.689995 638.450012 lineto stroke newpath 109.609993 620.369995 moveto 127.689995 620.369995 lineto stroke newpath 109.609993 602.289978 moveto 127.689995 602.289978 lineto stroke newpath 145.770004 674.609985 moveto 145.770004 584.210022 lineto 163.850006 584.210022 lineto 163.850006 674.609985 lineto closepath stroke newpath 145.770004 656.530029 moveto closepath stroke newpath 145.770004 656.530029 moveto 163.850006 656.530029 lineto stroke newpath 145.770004 638.450012 moveto 163.850006 638.450012 lineto stroke newpath 145.770004 620.369995 moveto 163.850006 620.369995 lineto stroke newpath 145.770004 602.289978 moveto 163.850006 602.289978 lineto stroke newpath 181.930008 602.289978 moveto 200.010010 602.289978 lineto stroke newpath 181.930008 620.369995 moveto 200.010010 620.369995 lineto stroke newpath 181.930008 638.450012 moveto 200.010010 638.450012 lineto stroke newpath 181.930008 656.530029 moveto 200.010010 656.530029 lineto stroke newpath 181.930008 656.530029 moveto closepath stroke newpath 181.930008 674.609985 moveto 181.930008 584.210022 lineto 200.010010 584.210022 lineto 200.010010 674.609985 lineto closepath stroke newpath 218.089996 674.609985 moveto 218.089996 584.210022 lineto 236.169998 584.210022 lineto 236.169998 674.609985 lineto closepath stroke newpath 218.089996 656.530029 moveto closepath stroke newpath 218.089996 656.530029 moveto 236.169998 656.530029 lineto stroke newpath 218.089996 638.450012 moveto 236.169998 638.450012 lineto stroke newpath 218.089996 620.369995 moveto 236.169998 620.369995 lineto stroke newpath 218.089996 602.289978 moveto 236.169998 602.289978 lineto stroke newpath 254.250000 602.289978 moveto 272.330017 602.289978 lineto stroke newpath 254.250000 620.369995 moveto 272.330017 620.369995 lineto stroke newpath 254.250000 638.450012 moveto 272.330017 638.450012 lineto stroke newpath 254.250000 656.530029 moveto 272.330017 656.530029 lineto stroke newpath 254.250000 656.530029 moveto closepath stroke newpath 254.250000 674.609985 moveto 254.250000 584.210022 lineto 272.330017 584.210022 lineto 272.330017 674.609985 lineto closepath stroke 118.650002 683.650024 moveto (P0) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (P0) show 154.809998 683.650024 moveto (P1) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (P1) show 190.970001 683.650024 moveto (P2) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (P2) show 227.130005 683.650024 moveto (P3) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (P3) show 263.290009 683.650024 moveto (P4) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (P4) show /Times-Roman findfont 8 scalefont setfont 118.650002 665.570007 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 154.809998 665.570007 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 190.970001 665.570007 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 227.130005 665.570007 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 263.290009 665.570007 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show newpath 326.570007 674.609985 moveto 326.570007 584.210022 lineto 344.649994 584.210022 lineto 344.649994 674.609985 lineto closepath stroke newpath 326.570007 656.530029 moveto closepath stroke newpath 326.570007 656.530029 moveto 344.649994 656.530029 lineto stroke newpath 326.570007 638.450012 moveto 344.649994 638.450012 lineto stroke newpath 326.570007 620.369995 moveto 344.649994 620.369995 lineto stroke newpath 326.570007 602.289978 moveto 344.649994 602.289978 lineto stroke newpath 362.730011 674.609985 moveto 362.730011 584.210022 lineto 380.809998 584.210022 lineto 380.809998 674.609985 lineto closepath stroke newpath 362.730011 656.530029 moveto closepath stroke newpath 362.730011 656.530029 moveto 380.809998 656.530029 lineto stroke newpath 362.730011 638.450012 moveto 380.809998 638.450012 lineto stroke newpath 362.730011 620.369995 moveto 380.809998 620.369995 lineto stroke newpath 362.730011 602.289978 moveto 380.809998 602.289978 lineto stroke newpath 398.890015 602.289978 moveto 416.970001 602.289978 lineto stroke newpath 398.890015 620.369995 moveto 416.970001 620.369995 lineto stroke newpath 398.890015 638.450012 moveto 416.970001 638.450012 lineto stroke newpath 398.890015 656.530029 moveto 416.970001 656.530029 lineto stroke newpath 398.890015 656.530029 moveto closepath stroke newpath 398.890015 674.609985 moveto 398.890015 584.210022 lineto 416.970001 584.210022 lineto 416.970001 674.609985 lineto closepath stroke newpath 435.049988 674.609985 moveto 435.049988 584.210022 lineto 453.130005 584.210022 lineto 453.130005 674.609985 lineto closepath stroke newpath 435.049988 656.530029 moveto closepath stroke newpath 435.049988 656.530029 moveto 453.130005 656.530029 lineto stroke newpath 435.049988 638.450012 moveto 453.130005 638.450012 lineto stroke newpath 435.049988 620.369995 moveto 453.130005 620.369995 lineto stroke newpath 435.049988 602.289978 moveto 453.130005 602.289978 lineto stroke newpath 471.209991 602.289978 moveto 489.290009 602.289978 lineto stroke newpath 471.209991 620.369995 moveto 489.290009 620.369995 lineto stroke newpath 471.209991 638.450012 moveto 489.290009 638.450012 lineto stroke newpath 471.209991 656.530029 moveto 489.290009 656.530029 lineto stroke newpath 471.209991 656.530029 moveto closepath stroke newpath 471.209991 674.609985 moveto 471.209991 584.210022 lineto 489.290009 584.210022 lineto 489.290009 674.609985 lineto closepath stroke 335.610016 665.570007 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 371.769989 665.570007 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 407.929993 665.570007 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 444.089996 665.570007 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 480.250000 665.570007 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 335.610016 647.489990 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 371.769989 647.489990 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 407.929993 647.489990 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 444.089996 647.489990 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 480.250000 647.489990 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show newpath 109.609993 548.049988 moveto 109.609993 457.649994 lineto 127.689995 457.649994 lineto 127.689995 548.049988 lineto closepath stroke newpath 109.609993 529.970032 moveto closepath stroke newpath 109.609993 529.970032 moveto 127.689995 529.970032 lineto stroke newpath 109.609993 511.890015 moveto 127.689995 511.890015 lineto stroke newpath 109.609993 493.809998 moveto 127.689995 493.809998 lineto stroke newpath 109.609993 475.730011 moveto 127.689995 475.730011 lineto stroke newpath 145.770004 548.049988 moveto 145.770004 457.649994 lineto 163.850006 457.649994 lineto 163.850006 548.049988 lineto closepath stroke newpath 145.770004 529.970032 moveto closepath stroke newpath 145.770004 529.970032 moveto 163.850006 529.970032 lineto stroke newpath 145.770004 511.890015 moveto 163.850006 511.890015 lineto stroke newpath 145.770004 493.809998 moveto 163.850006 493.809998 lineto stroke newpath 145.770004 475.730011 moveto 163.850006 475.730011 lineto stroke newpath 181.930008 475.730011 moveto 200.010010 475.730011 lineto stroke newpath 181.930008 493.809998 moveto 200.010010 493.809998 lineto stroke newpath 181.930008 511.890015 moveto 200.010010 511.890015 lineto stroke newpath 181.930008 529.970032 moveto 200.010010 529.970032 lineto stroke newpath 181.930008 529.970032 moveto closepath stroke newpath 181.930008 548.049988 moveto 181.930008 457.649994 lineto 200.010010 457.649994 lineto 200.010010 548.049988 lineto closepath stroke newpath 218.089996 548.049988 moveto 218.089996 457.649994 lineto 236.169998 457.649994 lineto 236.169998 548.049988 lineto closepath stroke newpath 218.089996 529.970032 moveto closepath stroke newpath 218.089996 529.970032 moveto 236.169998 529.970032 lineto stroke newpath 218.089996 511.890015 moveto 236.169998 511.890015 lineto stroke newpath 218.089996 493.809998 moveto 236.169998 493.809998 lineto stroke newpath 218.089996 475.730011 moveto 236.169998 475.730011 lineto stroke newpath 254.250000 475.730011 moveto 272.330017 475.730011 lineto stroke newpath 254.250000 493.809998 moveto 272.330017 493.809998 lineto stroke newpath 254.250000 511.890015 moveto 272.330017 511.890015 lineto stroke newpath 254.250000 529.970032 moveto 272.330017 529.970032 lineto stroke newpath 254.250000 529.970032 moveto closepath stroke newpath 254.250000 548.049988 moveto 254.250000 457.649994 lineto 272.330017 457.649994 lineto 272.330017 548.049988 lineto closepath stroke 118.650002 539.010010 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 154.809998 539.010010 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 190.970001 539.010010 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 227.130005 539.010010 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 263.290009 539.010010 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 118.650002 520.929993 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 154.809998 520.929993 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 190.970001 520.929993 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 227.130005 520.929993 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 263.290009 520.929993 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 118.650002 502.849976 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 118.650002 484.769989 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 154.809998 502.849976 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 154.809998 484.769989 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 190.970001 502.849976 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 190.970001 484.769989 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 227.130005 502.849976 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 227.130005 484.769989 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 263.290009 502.849976 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 263.290009 484.769989 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show newpath 326.570007 548.049988 moveto 326.570007 457.649994 lineto 344.649994 457.649994 lineto 344.649994 548.049988 lineto closepath stroke newpath 326.570007 529.970032 moveto closepath stroke newpath 326.570007 529.970032 moveto 344.649994 529.970032 lineto stroke newpath 326.570007 511.890015 moveto 344.649994 511.890015 lineto stroke newpath 326.570007 493.809998 moveto 344.649994 493.809998 lineto stroke newpath 326.570007 475.730011 moveto 344.649994 475.730011 lineto stroke newpath 362.730011 548.049988 moveto 362.730011 457.649994 lineto 380.809998 457.649994 lineto 380.809998 548.049988 lineto closepath stroke newpath 362.730011 529.970032 moveto closepath stroke newpath 362.730011 529.970032 moveto 380.809998 529.970032 lineto stroke newpath 362.730011 511.890015 moveto 380.809998 511.890015 lineto stroke newpath 362.730011 493.809998 moveto 380.809998 493.809998 lineto stroke newpath 362.730011 475.730011 moveto 380.809998 475.730011 lineto stroke newpath 398.890015 475.730011 moveto 416.970001 475.730011 lineto stroke newpath 398.890015 493.809998 moveto 416.970001 493.809998 lineto stroke newpath 398.890015 511.890015 moveto 416.970001 511.890015 lineto stroke newpath 398.890015 529.970032 moveto 416.970001 529.970032 lineto stroke newpath 398.890015 529.970032 moveto closepath stroke newpath 398.890015 548.049988 moveto 398.890015 457.649994 lineto 416.970001 457.649994 lineto 416.970001 548.049988 lineto closepath stroke newpath 435.049988 548.049988 moveto 435.049988 457.649994 lineto 453.130005 457.649994 lineto 453.130005 548.049988 lineto closepath stroke newpath 435.049988 529.970032 moveto closepath stroke newpath 435.049988 529.970032 moveto 453.130005 529.970032 lineto stroke newpath 435.049988 511.890015 moveto 453.130005 511.890015 lineto stroke newpath 435.049988 493.809998 moveto 453.130005 493.809998 lineto stroke newpath 435.049988 475.730011 moveto 453.130005 475.730011 lineto stroke newpath 471.209991 475.730011 moveto 489.290009 475.730011 lineto stroke newpath 471.209991 493.809998 moveto 489.290009 493.809998 lineto stroke newpath 471.209991 511.890015 moveto 489.290009 511.890015 lineto stroke newpath 471.209991 529.970032 moveto 489.290009 529.970032 lineto stroke newpath 471.209991 529.970032 moveto closepath stroke newpath 471.209991 548.049988 moveto 471.209991 457.649994 lineto 489.290009 457.649994 lineto 489.290009 548.049988 lineto closepath stroke 335.610016 539.010010 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 371.769989 539.010010 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 407.929993 539.010010 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 444.089996 539.010010 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 480.250000 539.010010 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 335.610016 520.929993 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 371.769989 520.929993 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 407.929993 520.929993 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 444.089996 520.929993 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 480.250000 520.929993 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 335.610016 502.849976 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 335.610016 484.769989 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 371.769989 502.849976 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 371.769989 484.769989 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 407.929993 502.849976 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 407.929993 484.769989 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 444.089996 502.849976 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 444.089996 484.769989 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 480.250000 502.849976 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 480.250000 484.769989 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 335.610016 466.690002 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 371.769989 466.690002 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 407.929993 466.690002 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 444.089996 466.690002 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 480.250000 466.690002 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show /Times-Roman findfont 12 scalefont setfont 335.610016 683.650024 moveto (P0) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (P0) show 371.769989 683.650024 moveto (P1) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (P1) show 407.929993 683.650024 moveto (P2) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (P2) show 444.089996 683.650024 moveto (P3) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (P3) show 480.250000 683.650024 moveto (P4) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (P4) show newpath 109.609993 421.489990 moveto 109.609993 331.089996 lineto 127.689995 331.089996 lineto 127.689995 421.489990 lineto closepath stroke newpath 109.609993 403.410004 moveto closepath stroke newpath 109.609993 403.410004 moveto 127.689995 403.410004 lineto stroke newpath 109.609993 385.330017 moveto 127.689995 385.330017 lineto stroke newpath 109.609993 367.250000 moveto 127.689995 367.250000 lineto stroke newpath 109.609993 349.170013 moveto 127.689995 349.170013 lineto stroke /Times-Roman findfont 8 scalefont setfont 118.650002 412.450012 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 118.650002 394.369995 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 118.650002 376.290009 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 118.650002 358.209991 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 118.650002 340.130005 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 154.809998 340.130005 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 154.809998 358.209991 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 154.809998 376.290009 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 154.809998 394.369995 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 154.809998 412.450012 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show newpath 145.770004 349.170013 moveto 163.850006 349.170013 lineto stroke newpath 145.770004 367.250000 moveto 163.850006 367.250000 lineto stroke newpath 145.770004 385.330017 moveto 163.850006 385.330017 lineto stroke newpath 145.770004 403.410004 moveto 163.850006 403.410004 lineto stroke newpath 145.770004 403.410004 moveto closepath stroke newpath 145.770004 421.489990 moveto 145.770004 331.089996 lineto 163.850006 331.089996 lineto 163.850006 421.489990 lineto closepath stroke newpath 181.930008 421.489990 moveto 181.930008 331.089996 lineto 200.010010 331.089996 lineto 200.010010 421.489990 lineto closepath stroke newpath 181.930008 403.410004 moveto closepath stroke newpath 181.930008 403.410004 moveto 200.010010 403.410004 lineto stroke newpath 181.930008 385.330017 moveto 200.010010 385.330017 lineto stroke newpath 181.930008 367.250000 moveto 200.010010 367.250000 lineto stroke newpath 181.930008 349.170013 moveto 200.010010 349.170013 lineto stroke 190.970001 412.450012 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 190.970001 394.369995 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 190.970001 376.290009 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 190.970001 358.209991 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 190.970001 340.130005 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 227.130005 340.130005 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 227.130005 358.209991 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 227.130005 376.290009 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 227.130005 394.369995 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 227.130005 412.450012 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show newpath 218.089996 349.170013 moveto 236.169998 349.170013 lineto stroke newpath 218.089996 367.250000 moveto 236.169998 367.250000 lineto stroke newpath 218.089996 385.330017 moveto 236.169998 385.330017 lineto stroke newpath 218.089996 403.410004 moveto 236.169998 403.410004 lineto stroke newpath 218.089996 403.410004 moveto closepath stroke newpath 218.089996 421.489990 moveto 218.089996 331.089996 lineto 236.169998 331.089996 lineto 236.169998 421.489990 lineto closepath stroke newpath 254.250000 421.489990 moveto 254.250000 331.089996 lineto 272.330017 331.089996 lineto 272.330017 421.489990 lineto closepath stroke newpath 254.250000 403.410004 moveto closepath stroke newpath 254.250000 403.410004 moveto 272.330017 403.410004 lineto stroke newpath 254.250000 385.330017 moveto 272.330017 385.330017 lineto stroke newpath 254.250000 367.250000 moveto 272.330017 367.250000 lineto stroke newpath 254.250000 349.170013 moveto 272.330017 349.170013 lineto stroke 263.290009 412.450012 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 263.290009 394.369995 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 263.290009 376.290009 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 263.290009 358.209991 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 263.290009 340.130005 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show /Times-Roman findfont 16 scalefont setfont 299.450012 629.409973 moveto (==>) stringwidth pop 2.0 div -1 mul -5.600000 rmoveto (==>) show 299.450012 502.849976 moveto (==>) stringwidth pop 2.0 div -1 mul -5.600000 rmoveto (==>) show 516.409973 629.409973 moveto (==>) stringwidth pop 2.0 div -1 mul -5.600000 rmoveto (==>) show 516.409973 502.849976 moveto (==>) stringwidth pop 2.0 div -1 mul -5.600000 rmoveto (==>) show /Times-Roman findfont 12 scalefont setfont 190.970001 575.169983 moveto (Initial) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (Initial) show 190.970001 322.050018 moveto (After local shift) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (After local shift) show showpage %%EndDocument -92 619 a endTexFig 286 2090 a Fn(Figure)15 b(2:)20 b(Example)c(of)e(the)i(concatenation)f (algorithm)g(with)h(5)f(pro)q(cessors.)949 2727 y(24)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-comm@CS.UTK.EDU Wed Apr 21 16:18:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08358; Wed, 21 Apr 93 16:18:14 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19305; Wed, 21 Apr 93 16:16:14 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 21 Apr 1993 16:16:12 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from hub.meiko.co.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19245; Wed, 21 Apr 93 16:16:00 -0400 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk with SMTP id AA06389 (5.65c/IDA-1.4.4 for mpi-comm@cs.utk.edu); Wed, 21 Apr 1993 21:15:48 +0100 Date: Wed, 21 Apr 1993 21:15:48 +0100 From: James Cownie Message-Id: <199304212015.AA06389@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA04899; Wed, 21 Apr 93 20:11:18 GMT To: mpi-comm@cs.utk.edu Cc: ajgh@ecs.soton.ac.uk Subject: Meeting in Europe ? Content-Length: 944 Gentle people, At the last MPI meeting we agreed to have a meeting in August (and possibly even September as well). After this decision had been taken, various people approached me and asked whether we could have one of these meetings in Europe. (Not only Rolf and Lyndon !). I have since spoken to Professor Hey, who has kindly offered to host the meeting at Southampton and handle local arrangements if we were to decide to hold a meeting in Europe. Since the arrangements for these meetings are still fluid, now is the time to make this decision, therefore would you care to think about this now. I guess we should aim to make a decision at the next meeting. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Wed Apr 21 16:34:03 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08835; Wed, 21 Apr 93 16:34:03 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20629; Wed, 21 Apr 93 16:32:23 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 21 Apr 1993 16:32:18 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Mail.Think.COM by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20616; Wed, 21 Apr 93 16:32:16 -0400 Received: from Electryon.Think.COM by mail.think.com; Wed, 21 Apr 93 16:32:14 -0400 From: Adam Greenberg Received: by electryon.think.com (4.1/Think-1.2) id AA24033; Wed, 21 Apr 93 16:32:08 EDT Date: Wed, 21 Apr 93 16:32:08 EDT Message-Id: <9304212032.AA24033@electryon.think.com> To: jim@meiko.co.uk Cc: mpi-comm@cs.utk.edu, ajgh@ecs.soton.ac.uk In-Reply-To: <199304212015.AA06389@hub.meiko.co.uk> "jim@meiko.co.uk" Subject: Meeting in Europe ? Date: Wed, 21 Apr 1993 21:15:48 +0100 From: James Cownie Gentle people, At the last MPI meeting we agreed to have a meeting in August (and possibly even September as well). After this decision had been taken, various people approached me and asked whether we could have one of these meetings in Europe. (Not only Rolf and Lyndon !). I have since spoken to Professor Hey, who has kindly offered to host the meeting at Southampton and handle local arrangements if we were to decide to hold a meeting in Europe. Since the arrangements for these meetings are still fluid, now is the time to make this decision, therefore would you care to think about this now. I guess we should aim to make a decision at the next meeting. Heh. I'm fully willing to go to Europe. However, even if Europe is not the eventual venue, somewhere (anywhere?) other than a hotel isolated in North Dallas is to be desired. moose From owner-mpi-comm@CS.UTK.EDU Wed Apr 21 16:42:21 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08999; Wed, 21 Apr 93 16:42:21 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21293; Wed, 21 Apr 93 16:40:53 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 21 Apr 1993 16:40:50 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ocfmail.ocf.llnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21278; Wed, 21 Apr 93 16:40:48 -0400 Received: from [134.9.250.226] (nessett.ocf.llnl.gov) by ocfmail.ocf.llnl.gov (4.1/SMI-4.0) id AA09476; Wed, 21 Apr 93 13:40:32 PDT Message-Id: <9304212040.AA09476@ocfmail.ocf.llnl.gov> Date: Wed, 21 Apr 1993 13:50:13 -0800 To: James Cownie From: nessett@ocfmail.ocf.llnl.gov (Dan Nessett) X-Sender: nessett@ocfmail.llnl.gov Subject: Re: Meeting in Europe ? Cc: mpi-comm@cs.utk.edu Because of budgetary cuts, foreign travel has been restricted recently for us at LLNL. So, I couldn't make it to a meeting held outside the US. Dan From owner-mpi-comm@CS.UTK.EDU Wed Apr 21 17:27:29 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10298; Wed, 21 Apr 93 17:27:29 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24631; Wed, 21 Apr 93 17:26:14 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 21 Apr 1993 17:26:13 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from gw1.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24623; Wed, 21 Apr 93 17:26:11 -0400 Received: by gw1.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA16165; Wed, 21 Apr 93 21:26:08 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA26045; Wed, 21 Apr 93 15:24:44 MDT Date: Wed, 21 Apr 93 15:24:44 MDT From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9304212124.AA26045@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: Meeting in Europe ? > Heh. I'm fully willing to go to Europe. However, even if Europe is not the > eventual venue, somewhere (anywhere?) other than a hotel isolated in North > Dallas is to be desired. > > moose I'll second this. If this is a subcommittee meeting, you've got two votes from this organization! Tom Henderson NOAA Forecast Systems Laboratory From owner-mpi-comm@CS.UTK.EDU Wed Apr 21 23:13:08 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA15052; Wed, 21 Apr 93 23:13:08 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14845; Wed, 21 Apr 93 23:10:44 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 21 Apr 1993 23:10:42 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from THIALFI.CS.CORNELL.EDU by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14828; Wed, 21 Apr 93 23:10:40 -0400 Received: from CLOYD.CS.CORNELL.EDU by thialfi.cs.cornell.edu (5.67/I-1.99D) id AA15605; Wed, 21 Apr 93 23:10:34 -0400 Received: from ELLI.CS.CORNELL.EDU by cloyd.cs.cornell.edu (5.67/I-1.99D) id AA15802; Wed, 21 Apr 93 23:10:42 -0400 From: elster@cs.cornell.edu (Anne C. Elster) Date: Wed, 21 Apr 93 23:10:31 -0400 Message-Id: <9304220310.AA21130@elli.cs.cornell.edu> Received: by elli.cs.cornell.edu (5.67/N-0.13) id AA21130; Wed, 21 Apr 93 23:10:31 -0400 To: mpi-comm@cs.utk.edu Subject: Re: Meeting in Europe ? James Cownie writes: >At the last MPI meeting we agreed to have a meeting in August (and >possibly even September as well). > >After this decision had been taken, various people approached me and >asked whether we could have one of these meetings in Europe. (Not only >Rolf and Lyndon !). > Hear, hear! If MPI is supposed to be a truely INTERNATIONAL forum, I think it is indeed appropriate to schedule at least one of the meetings outside the U.S. Now, if the organizers (Jack/David) would still sponsor at least some of the airfare for those of us (academics) they now support, there is no way you can keep me away from the meeting! :-) Adam Greenberg follows up: > Heh. I'm fully willing to go to Europe. However, even if Europe is not the > eventual venue, somewhere (anywhere?) other than a hotel isolated in North > Dallas is to be desired. > > moose No kidding! Hmmm .. maybe the HPF peope picked the local to make sure people didn't feel like taking some sessions off to explore less technical things ... ;-) (devious smile) Anne ------------------------------------------------------------------------------- Anne C. Elster | (607) 254-8653 [off] 255-6236 [FAX] Space Plasma Physics | School of Electrical Engineering | (607) 272-2986 [home/answ.mach] Cornell University, 351 Theory Ctr.| Ithaca, NY 14853 | e-mail: elster@cs.cornell.edu USA | na.elster@na-net.ornl.gov From owner-mpi-comm@CS.UTK.EDU Thu Apr 22 01:46:33 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA16636; Thu, 22 Apr 93 01:46:33 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23870; Thu, 22 Apr 93 01:45:06 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 22 Apr 1993 01:45:05 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from gmdzi.gmd.de by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23857; Thu, 22 Apr 93 01:45:02 -0400 Received: from f1neuman.gmd.de (f1neuman) by gmdzi.gmd.de with SMTP id AA07638 (5.65c/IDA-1.4.4 for ); Thu, 22 Apr 1993 07:44:53 +0200 Received: by f1neuman.gmd.de id AA15494; Thu, 22 Apr 1993 07:44:49 GMT Date: Thu, 22 Apr 1993 07:44:49 GMT From: Rolf.Hempel@gmd.de Message-Id: <9304220744.AA15494@f1neuman.gmd.de> To: mpi-comm@cs.utk.edu Subject: Re: Meeting in Europe ? Cc: gmap10@f1neuman.gmd.de It's an excellent idea to hold one of the meetings in Europe. If you need an additional reason to accept it, here it is: in August and probably also in September it's a lot cooler in Europe (especially in England) than at Dallas. Rolf Hempel From owner-mpi-comm@CS.UTK.EDU Thu Apr 22 04:53:44 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21419; Thu, 22 Apr 93 04:53:44 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09250; Thu, 22 Apr 93 04:52:12 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 22 Apr 1993 04:52:11 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09242; Thu, 22 Apr 93 04:52:06 -0400 Date: Thu, 22 Apr 93 09:51:46 BST Message-Id: <6797.9304220851@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Meeting in Europe ? To: James Cownie , mpi-comm@cs.utk.edu In-Reply-To: James Cownie's message of Wed, 21 Apr 1993 21:15:48 +0100 Reply-To: lyndon@epcc.ed.ac.uk Cc: ajgh@ecs.soton.ac.uk Jim writes: > > At the last MPI meeting we agreed to have a meeting in August (and > possibly even September as well). > > After this decision had been taken, various people approached me and > asked whether we could have one of these meetings in Europe. (Not only > Rolf and Lyndon !). I solid proposal if ever I heard one indeed! My support can be assumed. > I have since spoken to Professor Hey, who has kindly offered to host > the meeting at Southampton and handle local arrangements if we were to > decide to hold a meeting in Europe. Southampton would be fine. We could probably hold one here in Edinburgh (much prettier town than Southampton, imho :-). Of course I'd just love that. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ >From owner-mpi-comm@CS.UTK.EDU Wed May 12 04:35:20 1993 Received: from convex.convex.com by mozart.convex.com (5.64/1.28) id AA17462; Wed, 12 May 93 04:35:17 -0500 Received: from CS.UTK.EDU by convex.convex.com (5.64/1.35) id AA12711; Wed, 12 May 93 04:36:28 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12047; Wed, 12 May 93 05:14:29 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 12 May 1993 05:14:28 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from THIALFI.CS.CORNELL.EDU by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12017; Wed, 12 May 93 05:14:25 -0400 Received: from CLOYD.CS.CORNELL.EDU by thialfi.cs.cornell.edu (5.67/I-1.99D) id AA18254; Wed, 12 May 93 05:14:13 -0400 Received: from ELLI.CS.CORNELL.EDU by cloyd.cs.cornell.edu (5.67/I-1.99D) id AA14805; Wed, 12 May 93 05:14:13 -0400 From: elster@cs.cornell.edu (Anne C. Elster) Date: Wed, 12 May 93 05:14:09 -0400 Message-Id: <9305120914.AA19664@elli.cs.cornell.edu> Received: by elli.cs.cornell.edu (5.67/N-0.13) id AA19664; Wed, 12 May 93 05:14:09 -0400 To: mpi-comm@cs.utk.edu Subject: MPI_LOAD_INFO sugestion Cc: elster@cs.cornell.edu Status: R As much as I hate to expand the number of fuctions in MPI, I think this function is a MUST the way I see parallel computing in teh future. Below is a latex-version of the write-up I will be bringing to the meeting today. (Yes, yes, I know I should have discussed this long time ago, but ...). I will bring hard-copies to the meeting, so that people can start thinking about it. Hope you like the idea! - ------------------------------------------------------------------------------- Anne C. Elster | (607) 254-8653 [off] 255-6236 [FAX] Xerox Design Research Institute | (607) 272-2986 [home/answ.mach] Cornell University, 502 Theory Ctr.| Ithaca, NY 14853 | e-mail: elster@cs.cornell.edu USA | na.elster@na-net.ornl.gov \documentstyle[fullpage,12pt]{article} \baselineskip=12pt \oddsidemargin -0.6in \begin{document} \setlength{\parskip}{1ex} % \maketitle %\pagestyle{empty} \begin{center} {\large MPI\_LOAD\_INFO: Implementing load-balanced applications on distributed-memory systems} Anne C. Elster\\ \small Cornell University, Ithaca NY 14853 \date{\today} \end{center} \footnotesize \subsection*{Motivation} Currently, most parallelized algorithms for distributed memory machines assume that they will run on a homogeneous set of processors. (I.e. that each node in the parallel system has approximately the same computing power and I/O speed.) The main focus in parallel computing has thus far been to minimize communication over-head and achieve a load-balance for the particular algorithm or application. However, as each computational node a parallel system become more complex, their individual processor/node performance characteristics will tend to deviate more as some nodes perform I/O while others concentrate on system tasks, etc. This behavior has so far been addressed through multi-tasking on large shared-memory systems such as a network of IBM 3090s, however, no simple, straight-forward solutions exist currently for the distributed memory cases. It is, however, this authors belief that load-balancing application at the ``load-info'' level (i.e. based on each node's dynamic load at run-time) will become very important in the future -- especially if there were mechanisms built into MPI that would facilitate such implementations. \subsection*{Current options} The current options for getting load-info read into a Fortran or C application on Unix-based systems are not wonderful: System load is typically from the OS-side done by {\bf reading kernel memory (-kmem)}. This will, however, usually require {\em setgid} permissions and grubbing through kernel symbol tables. This brute-force approach might fork and exec something with the required kmem permissions. Another alternative is to use {\bf rstat} which does not require such permission, but unfortunately, is not available on all systems. Rstat typically gives you statistics from a remote kernel that should be current disregarding network delay. The third approach I can think of is to use the output from {\bf uptime}. In C, this could possibly be done though fscanf on the return value of popen. \subsection*{MPI suggestion} Since I fond none of the previous methods very programmer-friendly, my suggestion is to add a {\bf MPI\_LOAD\_INFO} that would return the desired information. In the point-to-point case, this would be similar to what one could get from $rstat$, whereas in the collective sense, one could be gathering information on a whole process group (or nodes within a context?) . I see the latter as VERY useful. Clearly the specific format of the info. to be received via such an MPI call needs to be discussed. If the feed-back on this suggestion is positive, this author would gladly write up a more detailed proposal for such a feature. Anne C. Elster Cornell Univ. \end{document} >From weeks@mozart.convex.com Wed Jun 9 14:59:20 1993 Return-Path: Received: from ens.ens-lyon.fr by lip.ens-lyon.fr (4.1/SMI-4.1) id AA15025; Wed, 9 Jun 93 14:59:18 +0200 Received: from convex.convex.com by ens.ens-lyon.fr (4.1/SMI-4.1) id AA12536; Wed, 9 Jun 93 14:58:20 +0200 Received: from mozart.convex.com by convex.convex.com (5.64/1.35) id AA01415; Wed, 9 Jun 93 07:57:31 -0500 Received: by mozart.convex.com (5.64/1.28) id AA27552; Wed, 9 Jun 93 07:59:43 -0500 Date: Wed, 9 Jun 93 07:59:43 -0500 From: weeks@mozart.convex.com (Dennis Weeks) Message-Id: <9306091259.AA27552@mozart.convex.com> To: Jack.Dongarra@lip.ens-lyon.fr Subject: May 12 mpi mail #4 Status: R Another one from Steve Otto (initial verbosity deleted so I could print the LaTeX file --DW): \documentstyle[twoside,11pt]{report} %\pagestyle{headings} %%\markright{ {\em Draft Document of the MPI Standard,\/ \today} } \marginparwidth 0pt \oddsidemargin=.25in \evensidemargin .25in \marginparsep 0pt \topmargin=-.5in \textwidth=6.0in \textheight=9.0in \parindent=2em % ---------------------------------------------------------------------- % mpi-macs.tex --- man page macros, % discuss, missing, mpifunc macros % % ---------------------------------------------------------------------- % a couple of commands from Marc Snir, modified S. Otto \newlength{\discussSpace} \setlength{\discussSpace}{.7cm} \newcommand{\discuss}[1]{\vspace{\discussSpace} {\small {\bf Discussion:} #1} \vspace{\discussSpace} } \newcommand{\missing}[1]{\vspace{\discussSpace} {\small {\bf Missing:} #1} \vspace{\discussSpace} } \newlength{\codeSpace} \setlength{\codeSpace}{.3cm} \newcommand{\mpifunc}[1]{\vspace{\codeSpace} {\bf #1} \vspace{\codeSpace} } % ----------------------------------------------------------------------- % A few commands to help in writing MPI man pages % \def\twoc#1#2{ \begin{list} {\hbox to95pt{#1\hfil}} {\setlength{\leftmargin}{120pt} \setlength{\labelwidth}{95pt} \setlength{\labelsep}{0pt} \setlength{\partopsep}{0pt} \setlength{\parskip}{0pt} \setlength{\topsep}{0pt} } \item {#2} \end{list} } \outer\long\def\onec#1{ \begin{list} {} {\setlength{\leftmargin}{25pt} \setlength{\labelwidth}{0pt} \setlength{\labelsep}{0pt} \setlength{\partopsep}{0pt} \setlength{\parskip}{0pt} \setlength{\topsep}{0pt} } \item {#1} \end{list} } \def\manhead#1{\noindent{\bf{#1}}} \hyphenation{RE-DIS-TRIB-UT-ABLE sub-script mul-ti-ple} \begin{document} \setcounter{page}{1} \pagenumbering{roman} \title{ {\em D R A F T} \\ Proposal VIII for contexts and groups} \author{Mark P. Sears, {\em Sandia National Laboratories} } \maketitle \hfuzz=5pt %\tableofcontents %\begin{abstract} %We don't have an abstract yet. %\end{abstract} \setcounter{page}{1} \pagenumbering{arabic} \label{sec:context} %======================================================================% % BEGIN "Proposal VIII" % % written by Mark P. Sears % Sandia National Laboratories % March 30, 1993 % % I defined these because the discuss macro wont let me % use \verb or verbatim environment. \def\dishdr{\vspace{\discussSpace} \begingroup\setlength{\parindent}{0pt} \setlength{\parskip}{0pt}\leftskip=.6cm \small {\bf Discussion: }} \def\disend{\par\vspace{\discussSpace} \endgroup\par } % end of macro definitions \chapter{Contexts and Groups -- Proposal VIII} \begin{center} {Mark P. Sears, Sandia National Laboratories} \end{center} \section{Introduction and motivation.} \dishdr March 30 draft -- handed out at MPI meeting April 13 draft -- modifications to context allocation scheme, added discussion of limitations, interaction with pt2pt draft, some reorganization. May 11 draft -- added getgroupcontext function which allows the allocation of a dynamic context within a group. On reading this proposal, anything in the discussion sections is considered to be commentary, is open to debate and is probably not part of the proposal as it would go into the MPI document as a whole. \disend The MPI committee has determined that message passing common practice needs to be extended to satisfy two objectives. The first of these is to add facilities which allow development and use of independent software modules (libraries and other third party software). The primary need here is seen to be enabling the establishment of independent communication environment, or context, for independent software components. The second objective is to add capabilities, especially for collective communications, which allow the software developer to use subsets, or groups, of the processes in the parallel task in a convenient way. The purpose of this chapter is to define these concepts more precisely. The development of these ideas follows primarily from the principle that context and group concepts should be as orthogonal as possible, in order to make them easy to understand and implement. The remainder of this chapter defines the terms {\bf process}, {\bf parallel task}, {\bf process address}, {\bf context}, {\bf group }, and {\bf topology} as they are used in this document. In addition this chapter describes the MPI functions needed to assign, create, and manipulate contexts and groups, and describes the overall syntax needed to define point to point and collective communications. Later chapters in the document contain precise descriptions of point to point and collective communication functions syntax and semantics. In the following, a {\bf global quantity} is one which is defined and can be used in the same way on every process of the parallel task. For example, it can be sent as part of a message from one process to another and used on the second process in the same way as on the first. A {\bf local quantity} is defined only on the process which creates it and cannot be sent (in a meaningful way) to another process. Process addresses, tag values, and context values are global quantitites; groups are local quantities. Similarly a {\bf global call} or {\bf group call} is a function call executed (loosely) synchronously on all processes or all processes which are members of a group. A {\bf local call} is a function call executed on a single process independent of any execution on other processes. \dishdr This proposal differs from the other context-group proposals in that context and group are held to be orthogonal and distinct entities. It is closest in flavor to Proposal III, where context is a low level concept, but here the definition of group is unrelated to the definition of context. In this proposal, groups have no context or tags associated with them, default or otherwise. \begin{enumerate} \item The purpose of contexts is to promote software modularity by allowing the construction of independent tag spaces. A context is an integer-valued extension to the tag field which must match exactly between sender and receiver of point-to-point communications. \item In this proposal, contexts are global and there is no concept of a process ``belonging'' to a context. Contexts are also anticipated to be a scarce resource. Context allocation is a rare event, typically limited to program startup. \item The purpose of groups is to provide tools for management of subsets of processes in a parallel task. A group is defined abstractly as an ordered set of unique integers (the elements). Group as a type is a pointer to an object which defines the mapping between rank and the elements of the group, and is a locally defined opaque type. Note that a group can be considered as a ``collection of processes'' only insofar as the elements of the group are process addresses. \item As defined in this proposal, groups contain no context or tag information, nor is a facility for caching user defined information provided. This implies that context and tag must be specified in addition to group for collective communication functions. This accords with the goal of MPI as specifying a lower level implementation that can be used to create different kinds of higher level implementations. \end{enumerate} This proposal is distinguished from proposal III by a much more complete orthogonalization of the concepts of group and context, as well as specifying group at a lower, more fundamental level than proposal III. The advantages of this lower level definition are primarily ease of implementation and understanding and additional flexibility in creating higher level functionality. In addition, because groups use no context or tag resources, they are limited only by available memory and CPU. \disend \section{Parallel task and process models.} An MPI program or {\bf parallel task} consists of a set of {\bf processes}, each executing independent code in an independent address space. The model of a process is similar to that of UNIX. A process can be single threaded or multithreaded, and in a multithreaded environment, MPI makes no distinction between the several threads of a process. Messages can be sent and received between processes, but there is no mechanism to specify the receiving thread or obtain the sending thread of a message. The set of processes in a parallel task is fixed. Each process is addressed by its rank in the task, which is fixed and assigned in an implementation defined way when the parallel task is created. The assignment of processes to processors is also implementation defined, and MPI defines a method (described below) to return a description of the process assignment, or {\bf global topology}. The process address of a process is an integer defined globally within the parallel task, which can be sent in messages to any other process and used with the same meaning on any other process. In addition to the processes in a parallel task, an MPI implementation may also allow communication with a {\bf host process}, if it exists. The host process is not considered part of the parallel task, but point to point communications with it are allowed. The host process may send to or receive messages from any process in the parallel task. The process address of the host process is an integer defined globally within the parallel task, but is not a valid rank in the parallel task. A routine, \verb+MPI_gethost+, returns the host address if available, or returns the constant \verb+MPI_NOHOST+ if there is no host process. An MPI implementation may support a limited subset of point-to-point communications with a host process. The creation and termination of processes and parallel tasks is not defined within MPI. \dishdr This section does not differ greatly from other proposals. This section would modify section 1.3 of the draft Point to Point communication document, mostly in regard to the question of how processes are addressed. I have expanded on the host process stuff a little from the discussion at the last meeting. The committee agreed (I think) to include the possibility of a host process. I feel it is unfortunate but necessary to postpone any discussion of communication between parallel tasks to MPI 2.0. Some of the possible properties that such communication might have are: \begin{itemize} \item A process may belong to at most one parallel task. \item Process addressing must be extended to allow for the inclusion of a parallel task identifier. \item Inter-task communication requires a session layer which intra-task communication does not. \end{itemize} but all this remains open to debate. % i want to think about this a little: % % Inter-task communication can be added to the above proposal as follows. % Each parallel task has a system-wide unique identifier. The definition % of a process address is widened to include both a task id and the process % rank within the task. In other words process-address = (task-id, rank). % The result must be encodable in an integer. No changes would be made % to the syntax or semantics of point-to-point communications, except that % % The concept of dynamic parallel tasks, tasks where the number of processes belonging to the task changes over time, is also full of difficulties and should also be postponed to MPI 2.0, along with any discussion of the means by which parallel tasks are created or destroyed. Note that it would be quite useful to define these things. \disend \section{Contexts.} The purpose of contexts is to promote software modularity by allowing the construction of independent tag spaces. A context is an integer-valued extension to the tag field which must match exactly between sender and receiver of point-to-point communications. By specifying a unique context in addition to tag and other features of a message, the communication between sender and receiver is made independent of other communications which use different contexts. In the following, context values are assigned or allocated by various software components, but they are not pushed or popped during program execution. This more static model avoids the state associated with such a stack mechanism. Contexts as described here do not provide a mechanism for a software component to protect contexts that it has allocated or assigned from misuse. The only protection is that software components which use different contexts are free to use tags within those contexts with a guarantee of no interference. A {\bf context} or {\bf context value} is an integer in a range \verb+[0..MPI_MAXCONTEXT]+ which is part of the message envelope. The sender and receiver of the message must specify exactly matching context values; no wildcard filtering on the context part of a message envelope is allowed. Filtering of messages based on sender address and/or tag is discussed further in the section on point to point communications. The number of available context values, \verb+MPI_NUMCONTEXTS+, is an implementation-defined quantity. This proposal recommends a minimum number of 256, although implementations allowing a greater or lesser number of context values are allowed. \dishdr There has been considerable discussion on the network about the number of available context values. One proposal (by Rik Littlefield) is that MPI should go ahead and specify that context and tag are both 32 bit qtys and that the implementor should provide for the possibility of contexts and tags anywhere in this space. I think these limits are too generous. The other side (e.g. Peter Rigsbee) is to not specify any hard limits and let the marketplace decide. I don't quite agree with this either, since other standards (e.g. Fortran 77) {\it do} specify limits for some things. Perhaps the committee could decide that there is a lower limit (say 256, my original estimate of 16 was seen by many as too small and I have altered it (see above)) for compliance, and a recommended number of $2^{16}$ or $2^{32}$. Then we could identify compliant implementations (satisfies the lower limit) and strongly compliant implementations (satisfies the recommended limit). I propose that this point be left up for grabs and discussed further by the environmental subcommittee. The above statement specifies that there exists an implementation defined limit on the size of the context space. But it does not specify that there is a limit on the number of contexts in the space that can be used at any one time. Should we specify the existence of such a limit? I don't think so. \disend There exists a predefined and allocated context value, \verb+MPI_DEFAULTCONTEXT+ which does not need to be allocated or opened and cannot be freed or closed. This context value is intended for initialization purposes and is free for use by all software components. Context values may either be assigned statically or allocated dynamically, and so we refer to them as static contexts and dynamic contexts. Since context values are integers, it is not appropriate to speak of contexts being created or destroyed. Static context values are chosen by the programmer and should lie in the range \verb+[0, MPI_STATICCONTEXTLIMIT]+, where \verb+MPI_STATICCONTEXTLIMIT+ is an implementation-defined constant. Dynamic context values will be allocated outside this range, within the limit \verb+MPI_CONTEXTLIMIT+ mentioned previously. \dishdr A problem could arise when a static context value is used which would normally be allocated by the dynamic allocation routines. Here we only specify an administrative solution, because that might be the intended program behavior. Generally, we adhere to the notion of allowing any reasonable use of context values rather than specifing strong limitations. \disend Static context values must be opened before use and can be closed when no longer needed. The routine \begin{verbatim} void MPI_opencontext(int context) \end{verbatim} is a local operation that specifies that the context can be used in sending and receiving messages. The routine \begin{verbatim} void MPI_closecontext(int context) \end{verbatim} is a local operation that specifies that the context is no longer needed and no further messages will be sent or received using the context value. Dynamic context values can be allocated with the routine \begin{verbatim} int MPI_getcontext() \end{verbatim} which returns a context value and which must be called synchronously by all processes in the parallel task before messages can be sent or received with the context. Dynamic context values can be freed with the routine \begin{verbatim} void MPI_freecontext(int context-value) \end{verbatim} which is a local operation and which signals to MPI that the given context value will no longer be used. In addition, dynamic context values can be allocated within a group with the routine \begin{verbatim} int MPI_getgroupcontext(Group g) \end{verbatim} This routine must be called synchronously by all processes which are members of the group. It will allocate a context value which has not been previously allocated on any of the processes in the group, so the context value is guaranteed to be unique within the group. Note that if there are two groups which do not intersect then the same context value may be allocated on each. Note also that it is possible for a context value to be allocated in a group and sent to a process not in the group which then opens the context and uses it for communications. There is no guarantee of uniqueness on the second process. Finally, there is no coupling of the context with the group. It is not possible to get the group that was used to allocate the context, nor is the context associated with the group in any way. \dishdr The getgroupcontext function is a collective communication operation. Here it has been specified without reference to a predecessor context or a set of tags, which means that it must be implemented using context and tags internal to the implementation. The semantics needs to be refined to handle the possibility of multithreading (programmer must mutex other threads). Both the getcontext and getgroupcontext functions give the user a lot of flexibility and a lot of room to hang himself or herself. \disend \dishdr The original version of this proposal only allowed for dynamic global context values. After discussion, I have modified the proposal to allow for statically and dynamically allocated contexts, as well as contexts which are allocated only within a group. Internally, MPI implementations may need to set up private data structures or other resources (queues, buffers, etc.) in order to send and receive messages with a particular context value. The \verb+MPI_getcontext()+ routine is the appropriate entry point for such setup to occur. Some implementations will simply set up resources for all possible context values at program initiation. % Deleted: % Obviously, there are many possible schemes for context allocation and % deallocation. The scheme proposed here avoids the need and overhead % of a complex server. Another option is to avoid the allocation problem % altogether, have only static allocation % (as for Fortran unit numbers or TCP/IP ports), and have a routine % \verb+MPI_opencontext(int context-value)+ which initializes use of the % context. % Fancier context allocation could be built on top of such a proposal. It may be necessary for some MPI implementations to preallocate other context values for internal use. Such context values would not be accessible by user code. \disend \section{Groups.} The purpose of groups is to provide tools for management of subsets of processes in a parallel task. A group is defined abstractly as an ordered set of unique integers (the elements); this abstract definition allows groups to be constructed in many different ways and allows the definition of a group as an explicit list (e.g. 7,4,13) or a computational rule (e.g. processes with odd addresses). Groups as program objects may be defined by MPI or may be developed as third party software. A {\bf group} is a one-to-one mapping from the integers $(0, 1, ..., N-1)$ to another set of integers, the {\bf elements} of the group. The number of elements $N$ is the {\bf order} of the group. The {\bf range} of the group is an upper limit on the size of any element. Note that a group is not an ``array of integers'', although that is one possible implementation. An ordered collection of processes is said to be a group if each element of the group defines a process address, or rank in the parallel task. A process belongs to a group if its address is an element of the group. In the following, the opaque type {\tt MPIGroup} is intended to be thought of as a pointer to one of many possible group structures. This {\bf group identifier} is local to a process and may not be legitimately passed from one process to another. \dishdr It would not be too difficult to define functions \verb+MPI_getgroupdsc()+ and \linebreak[4] \verb+MPI_bldgroupfromdsc()+ which would compute an ASCII description of a group or build a group from its ASCII description. Such functions would ease the problem of sending groups in messages. \disend Groups are defined only by construction in a process, so the correct method of passing a group from one process to another is to pass the information needed to construct it. Group construction and destruction is therefore local to a process, and is independent of whether the process belongs to the group or not. There may be many classes of groups, some defined by MPI and some defined as user extensions. Any group class must define the following methods: \begin{verbatim} int MPI_grouporder(MPIGroup g) -- compute order of a group. int MPI_grouprange(MPIGroup g) -- compute range of a group. int MPI_groupelement(MPIGroup g, int rank) -- compute element given rank. int MPI_grouprank(MPIGroup g, int element) -- compute rank of element. int MPI_groupiselement(MPIGroup g, int element) -- membership predicate. MPIGroup MPI_copygroup(MPIGroup g) -- make a copy of a group. void MPI_freegroup(MPIGroup g) -- group destructor. \end{verbatim} \dishdr The public definition of a group would contain a class member defining all of these operations, so, for example, \verb+MPI_grouporder+ would simply execute the function \verb+group->class->order+. In order for the group concept to be user extensible, some common definition of the basic group object and class object needs to be established. This definition is properly done by the binding committee. Thus, although {\bf MPIGroup} is an opaque type it is not entirely hidden -- a sophisticated user can get at it and create new group types. \disend In addition, each class of groups must define a group constructor. The constructor takes as parameters the information needed to construct the group and returns the opaque group identifier. >From the point of view of this proposal, group construction and use are inexpensive operations, limited only by memory and CPU resources. \dishdr For example, let us define a simple class of groups, which we call {\bf linear} groups. For such a group, we define the mapping from rank to element to be the function $(start + delta * rank)$, so the parameters needed to create the group are the order and the constants $start, delta$. The group constructor would then be defined as \begin{verbatim} MPIGroup MPI_makelineargroup(int order, int start, int delta). \end{verbatim} Another simple class would be a {\bf list} group, for which the construction parameters are the order of the group and a list of the elements. The constructor is \begin{verbatim} MPIGroup MPI_makelistgroup(int order, int *elements). \end{verbatim} Note that in the first case, there is probably no need for any communication to create the group. In the second case, the list might have to be distributed using communications, but the group construction is purely local to each process. \disend \section{Using groups in point-to-point communications.} Groups can be used to simplify point-to-point communication. A process in a group can be defined either by its process address (equivalent to the rank in the task) or by its rank in the group. A typical call to send a message using a (group,rank) address might have the form \begin{verbatim} MPI_send(buffer, length, MPI_groupelement(group, rank), tag, context) \end{verbatim} Note that in order for a process to send a message to a process in another group using the second process' group rank, it must construct a local copy of the group. Since a group defines an order for its elements, it is easy to compute the preceding and following processes of a particular process in a group. The preceding and following processes (elements) of the current process are given by \begin{verbatim} my_rank = MPI_grouprank(group, my_pid) pid_precede = MPI_groupelement(group, my_rank-1) pid_follow = MPI_groupelement(group, my_rank+1). \end{verbatim} modulo boundary conditions. \dishdr Two and three dimensional generalizations are straightforward but require additional functions specific to particular types of groups (e.g. Cartesian groups). \disend \section{Groups and collective communication.} Groups are the primary method of organizing collective communication operations. All collective communication operations have the form \begin{verbatim} void MPI_xxx(group, context, ...) \end{verbatim} Such functions must be called by all of the processes in the group in order to complete successfully. \dishdr Previously I proposed that collective communication routines be passed both a context and one or more tags. But a much cleaner solution is to pass the routine just a context under the assumption that the routine may make any use of tags within the context that it likes. Groups may contain private (internal to MPI) information about methods and data to accelerate common collective communication operations. For example, a group implementation may include a private copy of a spanning tree to be used for fast broadcasting. Such information is implementation dependent and not normally accessible to a user. The definition of a group should contain a hook for the collective comm routines to hang such data. \disend \section{Topology.} There are several meanings for the term topology as used in MPI. The first meaning is associated with the global topology or assignment of processes to processors in a particular network. Such a topology might be classified as a hypercube, mesh, or general network topology. Information about the global topology is returned by the routine \begin{verbatim} char *MPI_globaltopology() \end{verbatim} which returns a string containing a description of the global topology. The form of this string is generally {\it topology-class parameter ...}. \dishdr For example, various implementations might return strings like \begin{verbatim} N 564 -- a random (Ethernet) network with 564 processes. H 5 -- a 5 dimensional hypercube. R 2 16 13 -- a two dimensional mesh, 16 by 13. \end{verbatim} \disend The second type of topology is that internal to a group. Since, in this proposal, groups are ordered sets and therefore have an implicitly defined (one dimensional) topology, there is no need for independent topology routines. A different ordering can be selected by building a different group. For example, to select a Gray ordering instead of a natural ordering (useful on a hypercube for some applications), one would build a permutation group implementing the Gray ordering. Alternatively, one could impose a Gray ordering on an existing group by a group composition operation. Third, we define a {\bf virtual topology} as a method of referring to elements of a group other than by straightforward rank. For example, we might want to think of a group as a rectangle or higher order Cartesian object, and reference elements by x-rank, y-rank, etc. Another example might be a tree virtual topology, where we would refer to elements in terms of parent, sibling and child. Virtual topologies can be implemented by layering on top of MPI. \section{Contexts, groups, and third party software.} >From the point of view of this proposal, there are two kinds of third party or library software, i.e. software written to use MPI but intended for use in turn by other applications. The first type is similar to the MPI collective communication operations as described above. This type uses the contexts and tag space provided by whatever software calls it. The second type, typically associated with much more complex functions, establishes and manages its own context and tag spaces. MPI must be careful to support both possibilities. Library software of the first kind, including MPI collective communications, might benefit from the specification of some sort of ``tag manager'' so that only context would need to be provided. This proposal does not attempt to answer this question. \section{Cache facility.} This proposal does not define a cache facility for adding information to groups or contexts. Such a facility, although it has attractive features, is more appropriately layered on top of MPI. \section{Modifications to other documents.} The discussions in the current point-to-point document (sections 1.3 and\ 1.4 of the May 9 draft would be deleted). All of the collective communication routines would be modified to conform to the guideline that a collective communication routine is passed a group and context value separately. \section{Subsets of MPI.} Because this proposal maintains a strong orthogonality between the context and group concepts, it is easy to eliminate all collective communication operations (indeed all group operations) from MPI in order to generate a useful subset. The mechanisms for managing context are simple and do not require a server concept. \end{document} >From weeks@mozart.convex.com Wed Jun 9 15:02:26 1993 Return-Path: Received: from ens.ens-lyon.fr by lip.ens-lyon.fr (4.1/SMI-4.1) id AA15054; Wed, 9 Jun 93 15:02:23 +0200 Received: from convex.convex.com by ens.ens-lyon.fr (4.1/SMI-4.1) id AA12599; Wed, 9 Jun 93 15:01:00 +0200 Received: from mozart.convex.com by convex.convex.com (5.64/1.35) id AA01533; Wed, 9 Jun 93 08:00:15 -0500 Received: by mozart.convex.com (5.64/1.28) id AA27965; Wed, 9 Jun 93 08:02:23 -0500 Date: Wed, 9 Jun 93 08:02:23 -0500 From: weeks@mozart.convex.com (Dennis Weeks) Message-Id: <9306091302.AA27965@mozart.convex.com> To: Jack.Dongarra@lip.ens-lyon.fr Subject: May 12 mpi mail #5 Status: R Another LaTeX source with initial verbosity deleted ... \documentstyle[twoside]{report} \pagestyle{headings} %\markright{ {\em Draft Document of the MPI Standard,\/ \today} } \marginparwidth 0pt \oddsidemargin=.25in \evensidemargin .25in \marginparsep 0pt \topmargin=-.5in \textwidth=6.0in \textheight=9.0in \parindent=2em % ---------------------------------------------------------------------- % mpi-macs.tex --- man page macros, % discuss, missing, mpifunc macros % % ---------------------------------------------------------------------- % a couple of commands from Marc Snir, modified S. Otto \newlength{\discussSpace} \setlength{\discussSpace}{.7cm} \newcommand{\discuss}[1]{\vspace{\discussSpace} {\small {\bf Discussion:} #1} \vspace{\discussSpace} } \newcommand{\missing}[1]{\vspace{\discussSpace} {\small {\bf Missing:} #1} \vspace{\discussSpace} } \newcommand{\implement}[1]{\vspace{\discussSpace} {\small {\bf Implementation note:} #1} \vspace{\discussSpace} } \newlength{\codeSpace} \setlength{\codeSpace}{.3cm} \newcommand{\mpifunc}[1]{\vspace{\codeSpace} {\bf #1} \vspace{\codeSpace} } % ----------------------------------------------------------------------- % A few commands to help in writing MPI man pages % \def\twoc#1#2{ \begin{list} {\hbox to95pt{#1\hfil}} {\setlength{\leftmargin}{120pt} \setlength{\labelwidth}{95pt} \setlength{\labelsep}{0pt} \setlength{\partopsep}{0pt} \setlength{\parskip}{0pt} \setlength{\topsep}{0pt} } \item {#2} \end{list} } \outer\long\def\onec#1{ \begin{list} {} {\setlength{\leftmargin}{25pt} \setlength{\labelwidth}{0pt} \setlength{\labelsep}{0pt} \setlength{\partopsep}{0pt} \setlength{\parskip}{0pt} \setlength{\topsep}{0pt} } \item {#1} \end{list} } \def\manhead#1{\noindent{\bf{#1}}} \hyphenation{RE-DIS-TRIB-UT-ABLE sub-script mul-ti-ple} \begin{document} \setcounter{page}{1} \pagenumbering{roman} \title{ {\em D R A F T} \\ Document for a Standard Message-Passing Interface} \author{Scott Berryman, {\em Yale Univ} \\ James Cownie, {\em Meiko Ltd} \\ Jack Dongarra, {\em Univ. of Tennessee and ORNL} \\ Al Geist, {\em ORNL} \\ Bill Gropp, {\em ANL} \\ Rolf Hempel, {\em GMD} \\ Bob Knighten, {\em Intel} \\ Rusty Lusk, {\em ANL} \\ Steve Otto, {\em Oregon Graduate Inst} \\ Tony Skjellum, {\em Missisippi State Univ} \\ Marc Snir, {\em IBM T. J. Watson} \\ David Walker, {\em ORNL} \\ Steve Zenith, {\em Kuck \& Associates} } \date{May 9, 1993 \\ This work was supported by ARPA and NSF under contract number \#\#\#, by the National Science Foundation Science and Technology Center Cooperative Agreement No. CCR-8809615. } \maketitle \hfuzz=5pt %\tableofcontents %\begin{abstract} %We don't have an abstract yet. %\end{abstract} \setcounter{page}{1} \pagenumbering{arabic} \chapter{Point to Point Communication} \label{sec:pt2pt} \begin{center} Marc Snir \\ William Gropp and Ewing Lusk \end{center} \section{Introduction} This section is a draft of the current proposal for point-to-point communication. It does not yet include a description of the Fortran 77 and C bindings. I have tried to indicate, wherever appropriate, gaps and unresolved issues, using small type. \discuss{ The following sections of the introduction contain general information on the design of MPI procedures. The material should be moved to a general introduction for the entire document. The actual binding of these objects in Fortran and C will be discussed in the language binding subcommittee. } \section{Data Types} \subsection{Handle} MPI procedures use at various places {\em handles}. Handles are used to access opaque objects. Such object can be created, updated and destroyed only by by calling suitable MPI procedures, and providing the handle as parameter. Opaque objects hide from the user the internal representation used for various MPI objects, thus allowing to have similar calls in C and Fortran, allowing to overcome problems with the typing rules in these languages, and allowing for future extension of their functionality. The mechanism for opaque objects used here loosely follows the POSIX Fortran binding standard. An opaque object can be {\em persistent} or {\em ephemeral}. A persistent object persists until destroyed by an explicit operation. An ephemeral object is good for a single use; thus an ephemeral object associated with a communication operation disappears once this operation is completed (or once this object is not needed anymore for the completion of the operation). Opaque object are created by calls that are specific to each object type. They are destroyed by a call to {\tt MPI\_FREE}. Additional MPI functions are available to access and update specific opaque objects. \mpifunc{MPI\_FREE(handle)} \begin{description} \item[IN handle] handle to object \end{description} Mark the handle for destruction. The object can destroyed as soon as there there is no pending operation that is using this object. This is equivalent to changing the persistence of the handle from {\tt persistent} to {\tt ephemeral}. For example, if a communication object is freed, after the communication started but before it completed, then the pending operation is not affected; one can use the handle in calls to {\tt MPI\_WAIT} or {\tt MPI\_STATUS}. However, once the operation completes, it is erroneous to reuse the freed handle for a new communication. \discuss{ An alternative design point was discussed, whereby it is the user responsibility to free a handle only if there are no pending operations on this handle. The current design was deemed more elegant. Note that it is not necessary to destroy the handle as soon as it is not needed anymore. A valid implementation may just mark freed handles and garbage collect them periodically. On the other hand, some objects, such as buffer descriptor objects, can be used by several distinct concurrent communication operations. Thus, to find out whether such object can be freed, one would need a reference count of pending operations. This complication was not observed at our last meeting, and may be a good argument to reconsider the current definition of {\tt MPI\_FREE}. We do not have yet a mechanism to control where from storage is allocated for opaque objects. Some believe it is desirable to allow the user to control this allocation. I now have a generic {\tt MPI\_FREE} function for all kinds of opaque objects. Any reason to have specialized functions for each kind of object, as for {\tt CREATE}? } \mpifunc{MPI\_ASSOCIATED(handle, type)} \begin{description} \item[IN handle] handle to object \item[OUT type] state \end{description} Returns the type of the object the handle is currently associated with, if such exists. Returns the special type {\tt MPI\_NULL} if the handle is not currently associated with any object. MPI may provide predefined opaque objects and predefined, static handles to these objects. Such objects may not be destroyed. \subsection{Array of handles} An MPI call may need a parameter that is an {\em array of handles}. The array-of-handles object is not self-delimiting; it does not carry information on the number of entries in the array. This information has to be provided by an additional {\tt len} parameter, whenever a list-of-handles parameter is used. \discuss{ The reason for not having self-delimiting arrays is that records are not first-class citizens in Fortran and that having an array where one entry encodes length, whereas the other entries encode handles was deemed to be too error-prone. This line of argumentation really implies that we do not view array-of-handles as an opaque object, and that we expect the user to build these objects explicitly, not via MPI calls. The alternative is for array-of-handles to be an opaque object, manipulated via s set of MPI calls. } \subsection{State} MPI procedures use at various places arguments with {\em state} types. The values of such data type are all identified by names, and no operation is defined on them. For example, the {\tt MPI\_APPEND} routine has a state type parameter with values {\tt MPI\_INT, MPI\_REAL,} etc. \subsection{Named constants} MPI procedures sometimes assign a special meaning to a special value of a basic type parameter; e.g. {\tt tag} is an integer valued parameter of point-to-point communication operations, with a special {\tt DONTCARE} value. Such parameters will have a range of regular values, which is a proper subrange of the range of values of the corresponding basic type; special values (such as DONTCARE) will be outside the regular range. The range of regular values can be queried, and sometimes set, using environment inquiry or environment setting functions (Section~\ref{sec:inquiry}). \discuss{ Need to agree on a language mechanism for named constants. Implementers should detect, whenever possible, illegal uses of ``special values''. Thus, the use of the {\tt DONTCARE} value to tag a message sent will be flagged as an error. } \subsection{Choice} MPI functions sometimes use parameters with a {\em choice} (or union) data type. I.e., distinct calls to the same routine may pass by reference actual parameters of different types. The mechanism for providing such parameters will differ from language to language. \discuss{ The Fortran 77 standard specifies that the type of actual arguments need to agree with the type of dummy arguments; no construct equivalent to C pointers is available. Thus, it would seem that there is no standard conforming mechanism to support choice parameters. However, most Fortran compiler either don't check type consistency of calls to external routines, or support a special mechanism to link foreign (e.g., C) routines. I suggest that we accept this nonconformity with Fortran 77 standard. I.e., we accept that the same routine may be passed an actual parameter of a different type at distinct calls. Generic routines can be used in Fortran 90 to provide a standard conforming solution. This solution will be consistent with our nonstandard conforming Fortran 77 solution. } \section{Processes} \discuss{This material belongs to an introduction or environment section} An MPI program is executed by several autonomous processes that each execute their own code, in a MIMD style. The codes executed by each process need not be identical. The processes communicate via calls to MPI communication primitives. Typically, each processor executes in its own address space, although shared-memory implementations of MPI are possible. This document specifies the behavior of a parallel program assuming that only MPI calls are used for communication. The interaction of an MPI program with other possible means of communication (e.g., shared memory) is not specified. In particular, it is assumed that message buffers at distinct processors are disjoint. MPI does not specify the execution model for each process. A process can be sequential, or can be multithreaded, with threads possibly executing concurrently. Care has been taken to make MPI ``thread-safe'', by avoiding the use of implicit global states. The initial allocation of processes to an MPI computation and their binding to physical processors is not specified by the program itself. It is expected that vendors will provide mechanisms to do so either at load time or at run time. Such mechanisms will allow the specification of the initial number of required processes, the code to be executed by each initial process, and the allocation of processes to processors. Also, the current proposal does not provide for dynamic creation or deletion of processes during program execution (the total number of processes is fixed), although it is intended to be consistent with such extension. Finally, the current proposal does not specify a naming scheme for processes. We propose to always identify processes according to their relative rank in a context (group), so that, effectively, processes are identified by consecutive integers. \subsection{Error Handling} \discuss{ Material of this section should be moved to a general introduction section. This material has not yet been approved by MPIF} MPI provides the user with reliable message transmission: A message sent is always received correctly, and the user does not need to check for transmission errors, time-outs, or other error conditions. In other words, MPI does not provide mechanisms for dealing with failures in the communication system. Where the MPI implementation is built on an unreliable underlying mechanism, then it is the job of the implementer of the MPI subsystem to insulate the user from this unreliability, or to reflect unrecoverable errors as global program failures. Similarly MPI itself provides no mechanisms for handling node failures. Of course, MPI programs may still be erroneous. A {\bf program error} can occur when an MPI call is called with an incorrect parameter (non-existing destination in a send operation, buffer too small in a receive operation, etc.) This type of error would occur in any implementation. In addition, a {\bf resource error} may occur when a program exceeds the amount of available system resources (number of pending messages, system buffers, etc.). The occurrence of this type of error depends on the amount of available resources in the system and the resource allocation mechanism used; this may differ from system to system. The recommended implementation profile provides several mechanisms to alleviate the portability problem this represents. One can also write {\bf safe} programs, that are not subject to resource errors. All MPI procedure calls return an error parameter that indicates successful completion of the operation, or the error condition that occurred, otherwise. The recommended implementation profile in a POSIX environment is for any MPI routine that encounters a recoverable error to generate an {\em MPI error signal}, using a special signal value. The default handler for this signal terminates the execution of all involved processes, with a suitable error message being returned to the user. However, the user can provide his or her own signal handling routine. In particular, the user can specify a ``noop'' signal handler, thus relegating all error handling to the user code, using the error parameters returned by the MPI calls. MPI calls may initiate operations that continue asynchronously after the call returned. Thus, the operation may return with a code indicating successful completion, yet later cause an error exception to be raised. If there is a subsequent call that relates to the same operation (e.g., a call that verifies that an asynchronous operation has completed) then the error parameter associated with this call will be used to indicate the nature of the error. In a few cases, the error may occur after all calls that relate to the operation have completed, so that no error parameter can be used to indicate the nature of the error (e.g., an error in a send with the ready mode). In such cases, an error will be undetected, if the user disabled the MPI error signal. \discuss{ The alternative choice is to have fatal and non-fatal signals. One might want different signals for different modules. The details of such proposal need be elaborated in an appropriate ``profile'' subcommittee. } \section{Messages} A message consists of an {\em envelope} and {\em data}. \subsection{Data} The data part of a message consists of a sequence of values, each of a basic datatype in the host language. Thus, in Fortran 77, a message consists of a sequence of values that are each of type {\tt INTEGER}, {\tt REAL}, {\tt DOUBLE PRECISION}, {\tt COMPLEX}, {\tt DOUBLE PRECISION COMPLEX}, {\tt LOGICAL}, or (length 1) {\tt CHARACTER}. A message may also contain a value of type {\tt BYTE}, which consists of 8 binary digits. A byte is different from a character. Different machines may have different representations for the same character, or may use more than one byte to represent a character. On the other hand, a byte has the same binary value on any machine. A message may mix values of different types. \missing{ Need to agree on the C types (including handling of signed/unsigned), and on Fortran 90 types (including handling of kinds). } \subsection{Envelope} \label{subsec:envelope} The following information is associated with each message: \begin{description} \item[source] The rank of the sending process \item[destination] The rank of the receiving process \item[tag] User defined \item[context] handle \end{description} The range of valid values for the {\bf source} and {\bf destination} fields is {\tt 0 ... n-1}, where {\tt n} is the number of processes in the specified context. {\tt source = destination} is allowed: a process can send a message to itself. The ranges of valid values for {\tt tag} is implementation dependent, and can be found by calling a suitable query function, as described in Section~\ref{sec:inquiry}. The {\tt tag} field can be arbitrarily set by the application, and can be used to distinguish different messages. {\tt Context} should be a context shared by both source and destination. The actual mechanism used to associate an envelope with a message is implementation dependent; some of the information (e.g., {\bf sender} or {\bf receiver}) may be implicit, and need not be explicitly carried by a message. \section{Communication Buffers} \label{subsec:buffers} The basic point to point communication operations are {\bf send} and {\bf receive}. A {\bf send} operation creates a message; the message data is taken from the {\bf send buffer}. A {\bf receive} operation consumes a message; the message data is put into the {\bf receive buffer}. The specification of send or receive buffers uses the same syntax. \subsection{Communication buffer components} A buffer consists of a sequence of {\bf buffer components}. Each component consists of a sequence of (not necessarily distinct) variables of the same basic datatype. The datatype of such a variable is specified by a state parameter. The possible values for Fortran are {\tt MPI\_INT}, {\tt MPI\_REAL}, {\tt MPI\_DOUBLE}, {\tt MPI\_COMPLEX}, {\tt MPI\_DCOMPLEX}, {\tt MPI\_LOGICAL}, {\tt MPI\_CHAR} and {\tt MPI\_BYTE}, corresponding to datatypes {\tt INTEGER}, {\tt REAL}, {\tt DOUBLE PRECISION}, {\tt COMPLEX}, {\tt DOUBLE PRECISION COMPLEX}, {\tt LOGICAL}, {\tt CHARACTER} and to untyped {\tt BYTE}. The possible values for C are {\tt MPI\_CHAR}, {\tt MPI\_SHORT}, {\tt MPI\_INT}, {\tt MPI\_LONG}, {\tt MPI\_FLOAT}, {\tt MPI\_DOUBLE} and {\tt MPI\_BYTE}. \discuss{ May need to take care of kinds in Fortran 90 and of signed/unsigned in C. More elaborate definitions belong to the language binding. } There are five kinds of buffer components: \subsubsection{Contiguous component} A sequence of contiguous values of the same basic type, specified by \begin{description} \item[start] Initial element \item[count] Number of elements ($\tt count \ge 0$) \item[datatype] Type of elements \end{description} Thus, if {\tt A} is an array of integers, then {\tt A(1), 4, MPI\_INT} is the buffer component with entries {\tt A(1), A(2), A(3), A(4)}. \subsubsection{Vector component} A sequence of equally spaced and equally sized blocks of elements of the same basic type, specified by \begin{description} \item[start] Initial element \item[count] Number of elements ($\tt count \ge 0$) \item[stride] Number of elements between the start of each block \item[lenblk] Number of elements in each block ($\tt lenblk > 0$) \item[datatype] Type of elements \end{description} Assume that an element of type {\tt datatype} occupies {\tt m} bytes. Then, in a byte addressable machine, the buffer component described by these five parameters consists of the {\tt count} variables at addresses {\tt start, start+m, ... , start+m*(lenblk-1), start+m*stride,} \\ {\tt start+m*(stride+1), ..., start+m*(stride+lenblk-1), ...}. For example, if {\tt A} is an array of integers, then \begin{itemize} \item {\tt A(1), 5, 3, 2, MPI\_INT} is the buffer component with entries {\tt A(1), A(2), A(4), A(5), A(7)}. \item {\tt A(1), 5, 0, 2, MPI\_INT} is the buffer component with entries {\tt A(1), A(2), A(1), A(2), A(1)}. \item {\tt A(10), 5, -1, 2, MPI\_INT} is the buffer component with entries {\tt A(10), A(11), A(9), A(10), A(8)}. \end{itemize} {\tt Contiguous} is a special case of {\tt vector}. A vector buffer component can be used to extract an arbitrary submatrix of a two-dimensional matrix. \subsubsection{Indexed component} A sequence of elements of the same basic type, specified by \begin{description} \item[start] initial element \item[count] Number of elements ($\tt count \ge 0$) \item[array\_of\_indices] Array of displacements of the elements in the buffer components, relative to the initial element. \item[datatype] Type of elements \end{description} For example, if {\tt A} is an array of integers, and {\tt I} is an array with values {\tt 2, -1, 2, 1} then {\tt A(3), 4, I, MPI\_INT} is the buffer component with entries {\tt A(5), A(2), A(5), A(4)}. {\tt Vector} can be thought of as a special case of {\tt indexed} in which the indices are generated with a constant stride. \subsubsection{Heterogeneous vector (hvector) component} Same as {\tt vector}, except that {\tt stride} is given in multiple of bytes, rather than multiples of elements: \begin{description} \item[start] Initial element \item[count] Number of elements ($\tt count \ge 0$) \item[stride] Number of bytes between the start of each block \item[lenblk] Number of elements in each block ($\tt lenblk > 0$) \item[datatype] Type of elements \end{description} Assume that an element of type {\tt datatype} occupies {\tt m} bytes. Then, in a byte addressable machine, the buffer component described by these five parameters consists of the {\tt count} variables at addresses {\tt start, start+m, ... , start+m*(lenblk-1), start+stride, start+stride+m, ..., start+stride+m*(lenblk-1), ...}. Consider a C array declared as \begin{verbatim} struct { char info[3]; short number; double val; } a[100] \end{verbatim} Assume that all structure components are byte aligned, a {\tt short} occupies two bytes, and a {\tt double} occupies eight bytes. Then the buffer component {\tt \&a, 100, 13, 2, MPI\_CHAR} extracts the first two characters of each structure. The buffer component {\tt \&(a.number), 100, 13, 1, MPI\_SHORT} extracts the {\tt short} field of each structure; and the buffer component {\tt \&(a.val), 100, 13, 1} extracts the {\tt double} field of each structure. Thus, heterogeneous vectors can be used to extract vectors of homogeneous values from heterogeneous data. {\tt Vector} is a special case of {\tt heterogeneous vector}. \subsubsection{Heterogeneous indexed (hindexed) component} Same as indexed, except that displacements are in bytes, rather than multiples of element size. \begin{description} \item[start] initial element \item[count] Number of elements ($\tt count \ge 0$) \item[array\_of\_indices] Array of byte displacements of the elements in the buffer components, relative to the initial element. \item[datatype] Type of elements \end{description} For example, if {\tt a} is the C array of the previous example, and {\tt I} is an array with values {\tt 0, 13, 13, 27, 26} then {\tt \&a, 5, I, MPI\_CHAR} is the buffer component with entries {\tt a[0].info[0], a[1].info[0], a[1].info[0], a[1].info[1], a[1].info[0]}. Like {\tt heterogeneous vector}, {\tt heterogeneous indexed} extends the {\tt indexed} facility so as to allow the extraction of homogeneous data from a heterogeneous layout. \discuss{ The use of displacements in indexed buffer components has the same effect as indirect addressing into an array with initial index zero. I.e., if {\tt a} is an array with initial index zero, then a buffer component defined by parameters {\tt a[0], N, i, type} extract the elements {\tt a[i[0]], a[i[1]], ..., a[i[n]]}. This fits well C usage but ill-fits Fortran usage (initial index one). We took a decision of principle to force ``start from zero'' index arithmetic on Fortran users, rather than have different index arithmetic in the C and Fortran binding of MPI. The use of both buffer components with byte and element size displacements is motivated by the observation that some applications require the more general byte displacement, but that most applications would use element size displacement, which is much more natural and convenient (and machine independent). Thus, rather than forcing all users to do byte arithmetics, we decided to provide separately the more advanced function to ``expert programmers''. Leslie Hart proposes to replace ``heterogeneous'' with ``general''. } \subsection{Buffer operations} A buffer is described by an opaque {\bf buffer descriptor} object accessed via a {\bf buffer handle}. Such an object is created by {\tt MPI\_CREATE\_BUFFER}. It is associated with successive buffer components by calling in succession one of the functions {\tt MPI\_APPEND\_xxx}, where {\tt xxx} identifies the type of buffer component appended to the buffer. It becomes available for use in communication operations once committed by the function {\tt MPI\_COMMIT\_BUFFER}. Once committed, the descriptor cannot be modified any more. Note that the commit operation does not create yet a message, and does not bind the data to be sent. It is a commit for the buffer descriptor, not a commit for the buffer content. A buffer descriptor is destroyed after the completion of the first communication operation that uses it, if it is ephemeral, or after it is freed by a call to {\tt MPI\_FREE} and any pending communication operation that uses it has completed, if it is persistent. After the buffer descriptor object has been destroyed, the handle is undefined. \mpifunc{MPI\_CREATE\_BUFFER( buffer, persistence )} Create a new buffer descriptor object. The parameters are: \begin{description} \item[OUT buffer] buffer handle \item[IN persistence] buffer persistence (state type with possible values {\tt MPI\_PERSISTENT} or {\tt MPI\_EPHEMERAL}). \end{description} \discuss{ May want to provide information on maximum object size, and perhaps user space for the object. Leslie Hart proposes {\tt MPI\_CREATE\_BUFFER\_DESC}, rather than {\tt MPI\_CREATE\_BUFFER}. More accurate, but longer. Perhaps {\tt MPI\_CREATE\_BUFDESC}? } \mpifunc{MPI\_APPEND\_CONTIGUOUS( buffer, start, count, datatype)} Append a block component to buffer. The parameters are: \begin{description} \item[INOUT buffer] buffer handle \item[IN start] buffer component initial element (choice) \item[IN count] Number of elements (integer) \item[IN datatype] datatype identifier (status) \end{description} \mpifunc{MPI\_APPEND\_VEC( buffer, count, stride, lenblk, datatype )} Append a vector buffer component to buffer. The parameters are: \begin{description} \item[INOUT buffer] buffer handle \item[IN start] buffer component initial element (choice) \item[IN count] Number of elements (integer) \item[IN stride] Number of elements between the start of each block (integer) \item[IN lenblk] Number of elements in each block (integer) \item[IN datatype] datatype identifier (status) \end{description} \mpifunc{MPI\_APPEND\_INDEXED( buffer, start, count, array\_of\_indices, datatype)} Append an indexed buffer component to buffer. The parameters are: \begin{description} \item[INOUT buffer] buffer handle \item[IN start] initial position for indexing (choice) \item[IN count] Number of elements (integer) \item[IN array\_of\_indices] array of displacements of entries relative to start (array of integers) \item[IN datatype] datatype identifier (status) \end{description} The index values used to create this component are the values provided at the call; subsequent updates of the array of indices will not affect the buffer descriptor component. \mpifunc{MPI\_APPEND\_HVEC( buffer, start, count, stride, lenblk, datatype )} Append a heterogeneous vector buffer component to buffer. The parameters are: \begin{description} \item[INOUT buffer] buffer handle \item[IN start] buffer component initial element (choice) \item[IN count] Number of elements (integer) \item[IN stride] Number of bytes between the start of each block (integer) \item[IN lenblk] Number of elements in each block (integer) \item[IN datatype] datatype identifier (status) \end{description} \mpifunc{MPI\_APPEND\_HINDEXED( buffer, start, count, array\_of\_indices, datatype)} Append an heterogeneous indexed buffer component to buffer. The parameters are: \begin{description} \item[INOUT buffer] buffer handle \item[IN start] initial position for indexing (choice) \item[IN count] Number of elements (integer) \item[IN array\_of\_indices] array of byte displacements of entries relative to start (array of integers) \item[IN datatype] datatype identifier (status) \end{description} \mpifunc{MPI\_COMMIT\_BUFFER( bufferin )} Commit a buffer descriptor object. The object cannot be changed anymore after it is committed. A buffer descriptor object can be used for communication only after it is committed. The parameters are: \begin{description} \item[INOUT buffer] buffer descriptor handle \end{description} \discuss{ The commit function may do preprocessing of the buffer descriptor to simplify its use and commit resources in the communication subsystem for its use. The commit function may generate a new object and, hence, modify the object handle. The current proposal does not allow to create a buffer descriptor, use it in a communication after committed, then modify it partially and use it for another communication -- a new buffer descriptor has to be created. Of course, the same buffer descriptor can be reused many times in successive communications that send data from the same buffers; e.g., a new send with a buffer descriptor handle can be initiated once the previous send with this handle has completed. } Example 1: \\ Consider the following fragment of Fortran code \begin{verbatim} DOUBLE PRECISION A(10,20) INTEGER B, C(5,10) INTEGER BH ... CALL MPI_CREATE_BUFFER(BH, MPI_PERSISTENT) CALL MPI_APPEND_BLOCK (BH, B, 1, MPI_INT) CALL MPI_APPEND_VEC (BH, A(1,3), 11, 4, 2, MPI_DOUBLE) CALL MPI_APPEND_INDEX(BH, C(3,7), (4,2,1), MPI_INT) \end{verbatim} Then the buffer associated with the handle {\tt BH} consists of the sequence of variables {\tt B, A(1,3), A(2,3), A(5,3), A(6,3), A(9,3), A(10,3), A(3,4), A(4,4), A(7,4), A(8,4), A(1,5), C(2,8), C(5,7), C(4,7)}. A message created from this buffer will consist of a sequence of one integer, followed by eleven double precision reals, followed by three integers. Example 2: \\ Suppose one has an array of data declared in C as \begin{verbatim} struct { int type; double position[3]; double velocity[3]; } particle[1000] \end{verbatim} and one wants to transfer in one message the number of particles of type 1, the number of particles of type 2, and the positions of all the particles of these two types. The following code fragment creates a suitable buffer descriptor. \begin{verbatim} int count1 = 0; int count2 = 0; ... mpi_create_buffer( &handle, MPI_PERSISTENT); mpi_append_contiguous( &handle, &count1, MPI_INT); mpi_append_contiguous( &handle, &count2, MPI_INT); for (i=0; i<1000; i++) { if (particle[i].type == 1) { count1++; mpi_append_contiguous( &handle, &particle[i].position, 3, MPI_DOUBLE); } elseif (particle[i].type == 2) { count2++; mpi_append_contiguous( &handle, &particle[i].position, 3, MPI_DOUBLE); } } mpi_commit_buffer(handle) ... \end{verbatim} The buffer descriptor handle can now be used to send (repeatedly) the particle counts for type 1 and 2, and the positions of the particles with these two types, as long as the types of the particles is not modified. Suppose the incoming data is to be stored at the receiver into the two count variables {\tt count1} and {\tt count2} and into the {\tt position} components of the initial entries of the array of particles. The following code generates a suitable buffer descriptor for the receive operation (we assume 4 bytes for integer and 8 bytes for double). \begin{verbatim} ... mpi_create_buffer( &handle, MPI_PERSISTENT); mpi_append_contiguous( &handle, &count1, MPI_INT); mpi_append_contiguous( &handle, &count2, MPI_INT); mpi_append_hvec( &handle, &particle[0].position, 1000, 52, 3, MPI_DOUBLE); mpi_commit_buffer( &handle) ... \end{verbatim} A buffer handle can be used for communication, even if it is not associated with any variables (i.e., even if it was not set by any {\tt MPI\_APPEND\_xxx} call). Such a handle is associated with an empty buffer, and a message created from it contains no data. The handle still need be committed before it is used. The same entry may appear more than once in a buffer component, and distinct buffer components may overlap. If a buffer descriptor with replicated entries is used to receive a message, then the final value of an entry that occurs more than once in the buffer descriptor is undefined. \discuss{ It is not clear to me that overlaps have any useful purpose, and they may complicate implementation. But we had an overwhelming vote in favor of this feature. } \subsection{Type Matching and Data Conversion} \subsubsection{Type matching} One can think of message transmission as consisting of three phases: \begin{enumerate} \item Data is pulled out of the send buffer and a message is assembled \item A message is transferred from sender to receiver \item Data is pulled from the incoming message and disassembled into the receive buffer \end{enumerate} Type matching has to be observed at each of these three phases: The type of each variable in the sender buffer has to match the type of the corresponding entry in the buffer descriptor; the type of an entry in the buffer descriptor used to send a message has to match the type of the corresponding entry in the buffer descriptor used to receive the message; and the type of each variable in the receive buffer has to match the corresponding entry in the buffer descriptor for this buffer. A program that fails to observe these three rules is erroneous. To define type matching more precisely, we need to deal with two issues: matching of types of the host language with buffer descriptor types; and matching of buffer descriptor types at sender and receiver. We restrict ourselves to the case where the sending and receiving programs are written in the same language. The type of an entry in the sender buffer descriptor matches the type of an entry in the receiver buffer descriptor if they have identical names: {\tt MPI\_INT} matches {\tt MPI\_INT}, {\tt MPI\_REAL} matches {\tt MPI\_REAL}, and so on. The type of a variable in a host program matches the type of the entry in a buffer descriptor if the name of the buffer descriptor entry type corresponds to the basic type of the host program variable: an entry with type name {\tt MPI\_INT} matches a Fortran variable of type {\tt INTEGER}, an entry with type name {\tt MPI\_REAL} matches a Fortran variable of type {\tt REAL}, and so on. There is one exception to this last rule: An entry with type name {\tt MPI\_BYTE} can be used to match any byte of storage (on a byte-addressable machine), irrespective of the datatype of the variable that contains this byte. The value of the message entry will be the binary value of the corresponding byte in memory. We thus have two cases: \begin{itemize} \item Communication of typed values (e.g., with datatype different from {\tt MPI\_BYTE}, where the datatypes of the corresponding entries in the sender program, in the sender buffer descriptor, in the receiver buffer descriptor and in the receiver program should all match. \item Communication of untyped values (e.g., of datatype {\tt MPI\_BYTE}, where the the corresponding entries in the sender and the receiver buffer descriptors have both datatype {\tt MPI\_BYTE}. In this case, there are no requirements on the datatypes of the corresponding entries in the sender and the receiver programs, nor is it required that they be the same. \end{itemize} \subsection{Data conversion} We use the following terminology: \begin{description} \item[type conversion] changes the datatype of a value, e.g. by rounding a {\tt REAL} to an {\tt INTEGER}. \item[representation conversion] changes the binary representation of a value, e.g. from Hex floating point to IEEE floating point. \end{description} Representation conversion is needed when data is moved across machines that use different binary encodings for the same datatype. Representation conversion preserves, to the greatest possible extend, the {\bf value} represented. However, rounding errors and overflow and underflow exceptions may occur during floating point conversions; conversion of integers may also lead to exceptions when a value that can represented in one system cannot be represented in the other system. MPI does not specify rules for representation conversion. The type matching rules imply that MPI communication never entails type conversion. On the other hand, MPI requires that a representation conversion be performed when a typed value is transferred across environments that use different representations for the datatype of this value. An exception occurring during representation conversion results in a failure of the communication; an error occurs either in the send operation, or the receive operation, or both. If a value sent in a message is untyped (i.e., of type {\tt MPI\_BYTE}), then the binary representation of the byte stored at the receiver is identical to the binary representation of the byte loaded at the sender. This holds true, whether sender and receiver run in the same or in distinct environments; no representation conversion is required. Note that no conversions ever occur when an MPI program executes in a homogeneous system, where all processes run in the same environment. Also note the different behavior of {\tt MPI\_BYTE} and of {\tt MPI\_CHAR}. A buffer descriptor entry with datatype of {\tt MPI\_CHAR} can only match a Fortran variable of type {\tt CHAR}; and representation conversion may occur when values of type {\tt MPI\_CHAR} are transferred., e.g., from an EBCDIC encoding to an ASCII encoding. Consider the following examples. \begin{verbatim} ! FIRST EXAMPLE INTEGER A(100) ... CALL MPI_CREATE_BUFFER(BH, MPI_PERSISTENT) CALL MPI_APPEND_CONTIGUOUS (BH, A(1), 1, MPI_CHAR) \end{verbatim} This program is erroneous, since the type of the buffer descriptor does not match the type of the variable in the buffer. The behavior or this program is undefined. A desirable behavior is for that error to be detected (at compile time or run time) and result in an error condition. \begin{verbatim} ! SECOND EXAMPLE INTEGER A(100) ... CALL MPI_CREATE_BUFFER(BH, MPI_PERSISTENT) CALL MPI_APPEND_CONTIGUOUS (BH, A(1), 1, MPI_BYTE) \end{verbatim} This program is correct. The byte that will be transmitted in a message created with this buffer descriptor is the first byte in the storage occupied by {\tt A(1)}. The value of this byte is implementation dependent, and will vary according to the number of bytes used to store Fortran variables of type {\tt INTEGER} and the byte ordering on the host machine. \begin{verbatim} ! THIRD EXAMPLE INTEGER A(100) ... CALL MPI_CREATE_BUFFER(BH1, MPI_PERSISTENT) CALL MPI_APPEND_CONTIGUOUS (BH1, A(1), 1, MPI_INTEGER) ... CALL MPI_CREATE_BUFFER(BH2, MPI_PERSISTENT) CALL MPI_APPEND_CONTIGUOUS (BH2, A(1), 4, MPI_BYTE) \end{verbatim} Consider the following three cases: \begin{enumerate} \item A message is sent using buffer descriptor handle {\tt BH1} and received using buffer descriptor handle {\tt BH2}. The program is erroneous, since the sender and receiver buffer descriptors don't match. The behavior of the program is undefined, and a possible (desirable but unlikely) behavior is for an error condition to occur. \item Both sender and receiver use buffer descriptor handle {\tt BH1}. The program is correct, and an integer value is transmitted from sender to receiver. The value assigned to {\tt A(1)} at the receiver is identical to the value of {\tt A(1)} at the sender, but the binary representation of this value may be different, if sender and receiver run in different environments. \item Both sender and receiver use buffer descriptor handle {\tt BH2}. The program is correct. Assume that both sender and receiver use four bytes to store integers. Then the value stored in {\tt A(1)} at the receiver has the same binary representation as the value of {\tt A(1} at the sender. If the same byte ordering and the same encoding of integer values is used at both ends then {\tt A(1)} at the receiver is set to the value of {\tt A(1)} at the sender; otherwise the same 32 bits may encode different values. \end{enumerate} \implement{ The current definition does not require messages to carry type information. A message can be composed and sent using only the information in the sender buffer descriptor, and can be received and stored using only the information in the receiver buffer descriptor. If messages are sent between different machines then one can either use a ``universal'' data encoding for messages, use knowledge of the receiver environment in order to convert data at the sender, or use knowledge of the sender environment in order to convert data at the receiver. In either case the local buffer descriptor can be used to derive the types of the values transferred. However, additional type information carried by messages can provide better error detection. It is possible to add always such additional type information, for better error checking, or add it when running in ``debug'' mode, where some performance is sacrificed for added error checking, etc. } \discuss{ There is no agreement yet on the need for a definition and, if needed, on the right definition of type matching and conversion between programs written in different languages. The natural interpretation of ``send from C and receive in Fortran'' is either ``send from C, receive in C and convert to Fortran'', or ``convert to Fortran, send from Fortran and receive in Fortran''. However, the correspondence between Fortran types and C types is implementation dependent. Thus, the diagram shown below is not commutative, and the two interpretations of C to Fortran communication may lead to different results. \begin{tabular}{c c c} machine 1 & & machine 2 \\ & send & \\ C program & $\rightarrow$ & C program \\ & & \\ $\downarrow$ & & $\downarrow$ \\ & send & \\ Fortran program & $\rightarrow$ & Fortran program \end{tabular} A good solution would make this diagram commutative. Otherwise, one needs to pick one of the two paths as the canonical definition of interlanguage communication. If the upper path is chosen (message carries sender datatypes), then the receiver of a message needs to know the language the message was sent from; if the lower path is chosen, (message carries receiver datatypes), then the sender needs to know the language the message is received into. A possible solution (proposed by Danny Nessett) is to leave to the user the choice of specifying whether to send ``Fortran datatypes'' or ``C datatypes'', in the above situation. One would still require exact matching of send buffer descriptors and receive buffer descriptors. However, a Fortran program could use a buffer descriptor with ``C datatypes'', and vice versa. The standard would specify which matching of Fortran and C datatypes are legal. For example, it would be correct to send a Fortran variable of type {\tt REAL} using a buffer descriptor of type {\tt MPI\_FLOAT}. Data conversion might be required if C float numbers and Fortran real numbers do not use the same presentation. Another (non)solution is to ignore the issue and let the behavior of C to Fortran communication be undefined. The user can always match language at sending and receiving end if he or she is willing to struggle with the issue of calling MPI C functions from Fortran or vice-versa. } \section{Receive Criteria} The selection of a message by a receive operation is done uniquely according to the value of the message envelope. The receive operation specifies an {\bf envelope pattern}; a message can be received by that receive operation only if its envelope matches that pattern. A pattern specifies values for the {\tt source}, {\tt tag} and {\tt context} fields of the message envelope. In addition, the value for the {\tt dest} field is set, implicitly, to be equal to the receiving process id. The receiver may specify a {\tt DONTCARE} value for {\tt source}, or {\tt tag}, indicating that any source and/or tag are acceptable. It cannot specify a DONTCARE value for {\tt context} or {\tt dest}. Thus, a message can be received by a receive operation only if it is addressed to the receiving task, has a matching context, has matching source unless source=DONTCARE in the pattern, and has a matching tag unless tag=DONTCARE in the pattern. DONTCARE values for source or tag are specified via a named constant for source dontcare and a named constant for tag dontcare. The length of the received message must be less then or equal the length of the receive buffer. I.e., all incoming data must fit, without truncation, into the receive buffer. It is erroneous to receive a message which length exceed the receive buffer, and the outcome of program where this occurs is undetermined. \section{Communication Mode} A send operation can occur in one of three modes: \begin{description} \item[STANDARD] The send may start whether or not a matching receive has been posted. \item[READY] The send may start only if a matching receive has been posted. \item[SECURE] The send operation will not complete successfully until it is guaranteed that the message sent will be received. \end{description} The start and completion of a {\bf standard} send does not depend on the status of a matching receive. A {\bf ready send} can start only if a matching receive is already posted; otherwise the operation is erroneous and its outcome is undefined. In some systems, this allows the removal of a hand-shake operation that is otherwise required, and results in improved performance. Its completion does not depend on the status of a matching receive. A {\bf secure send} will complete if a matching receive is posted, and the receive operation is committed to receive the message sent by the secure send. (I.e., the send and receive operations have already been matched, the receive operation has already started and cannot be cancelled anymore.) The start of a secure send does not depend on the status of a matching receive. A receive operation can occur in one of two modes: {\bf STANDARD} and {\bf SECURE}. A secure send can be matched only by a {\bf secure receive}; a program where a secure send can be matched by a standard receive is erroneous and its behavior is undefined. The secure receive can be posted either before or after the matching send. A standard send and a ready send can both be matched only by a {\bf standard receive}; a program where a secure receive can match a nonsecure send is erroneous and its behavior is undefined. The receive has to be posted ahead of the send for a ready send and can be posted either before or after the send is posted for a standard send. \discuss{ I got criticisms about the previous notation (REGULAR) and the current one (STANDARD). I can also try NORMAL, DEFAULT, ... We should probably vote, at some point. Also, Lyndon proposes to use SYNCHRONOUS, rather than secure, since the other modes can also offer secure communication with the right use. } \implement{ The current definition assumes it is the user responsibility to prevent matching of a regular send with a secure receive and vice-versa. Otherwise, messages need be tagged as standard or secure. Assume an implementation of MPI communication that avoids retries. What protocols can be used? Since a receive may have a wildcard source field, the first step in the protocol must be executed by the sender. For a ready send, this first step may involve sending the data, since the receiver is guaranteed to have space to store it. The protocol is send, possibly followed by an ack-send. The protocol for secure send is likely to be the following: The sender sends a request-to-send message. The receiver stores this request (which contains a copy of the message envelope). When a matching receive is posted, the receiver sends back an acknowledge, and the sender now sends the message. The protocol is (req, ack-req, send), possibly followed by an ack-send. What protocol is used for standard sends? Either the second protocol (req, ack-req, send) or a mix of the first and the second (e.g. first used for short messages, second for long messages). When the receiver does not wildcard the source field then the protocol can be reversed: The receiver sends a clear-to-send message, and the sender can proceed with the short protocol if it has a clear-to-send when the send is executed. What do we gain from the ``secure receive''? Some optimizations for regular receives (e.g., in a regular receive of a short message the receiver knows the short protocol will be used, and need not check for a request-to-send). Q: is that enough justification for the secure receive? A counter argument (Leslie Hart) is that one could conceive of an implementation where a regular receive reserves buffer space, so that a program that use regular receives is not safe, even if only secure sends are used. I doubt this is a strong argument. On the other hand, it indicates that, if the expectation is that programs written using only secure sends are safe, then this is an additional correctness condition on the implementation that need be made explicit. } \section{Communication Objects} An opaque communication object identifies various properties of a communication operation, such as the buffer descriptor that is associated with it, its context, the tag and destination parameters to be used for a send, or the tag and source parameters to be used for a receive. In addition, this object stores information about the status of the last communication operation that was performed with this object. This object is accessed using a communication handle. One can consider communication operations to consist of the following suboperations: \begin{description} \item[INIT(operation, params, handle)] Process provides all relevant parameters for its participation in the communication operation (type of operation, data buffer, tag, participants, etc.). An object is created that identifies the operation. \item[START(handle)] The communication operation is started \item[COMPLETE(handle)] The communication operation is completed. \item[FREE(handle)] The communication object, and associated resources are freed. \end{description} Correct invocation of these suboperations is a sequence of the form \[ \bf INIT \ (START \ COMPLETE )^* \ FREE .\] I.e., an object needs be created before communication occurs; it can be reused only after the previous use has completed; and it needs to be freed eventually (of course, one can assume that all objects are freed at program termination, by default). The above scenario pertains to {\em persistent} objects. One can also create {\em ephemeral} objects. Such an object persists only until the communication operation is completed, at which point it is destroyed. Thus the correct invocation of suboperations with an ephemeral object is {\bf INIT START COMPLETE}. A user may directly invoke these suboperations. This would allow the amortization of the overhead of setting up a communication over many successive uses of the same handle, and allows the overlap of communication and computation. Simpler communication operations combine several of these suboperations into one operation, thus simplifying the use of communication primitives. Thus, one only needs to specify precisely the semantics of these suboperations in order to specify the semantics of MPI point to point communication operations. We say that a communication operation (send or receive) is {\bf posted} once a {\bf start} suboperation has been invoked; the operation is {\bf completed} once the {\bf complete} suboperation has completed. A send and a receive operation {\bf match} if the receive pattern specified by the receive matches the message envelope created by the send. \subsection{Communication Object Creation} \discuss{ This section has not yet been approved} An object for a send operation is created by a call to {\bf MPI\_INIT\_SEND}. A call to {\bf MPI\_INIT\_RECV} is similarly used for creating an object for a receive operation. The creation of a communication object is a local operation that need not involve communication with a remote process. \mpifunc{MPI\_INIT\_SEND (handle, buffer\_handle, dest, tag, context, mode, persistence)} Creates a send communication object. Parameters are \begin{description} \item[OUT handle] message handle. The handle should not be associated with any object before the call. \item[IN buffer\_handle] handle to send buffer descriptor \item[IN dest] rank in context of destination (integer) \item[IN tag] user tag for messages sent with this handle (integer) \item[IN context] context of messages sent with this handle \item[IN mode] send mode (state type, with three values: {\tt MPI\_STANDARD}, {\tt MPI\_READY} and {\tt MPI\_SECURE}) \item[IN persistence] handle persistence (state type, with two values: {\tt MPI\_PERSISTENT} and {\tt MPI\_EPHEMERAL}) \end{description} \mpifunc{MPI\_INIT\_RECV (handle, buffer\_handle, source, tag, context, mode, persistence)} Create a receive handle. Parameters are \begin{description} \item[OUT handle] message handle. The handle should not be associated with any object before the call. \item[IN buffer\_handle] handle to receive buffer descriptor. \item[IN source] rank in context of source, or {\tt MPI\_ANY\_SOURCE} (integer). \item[IN tag] user tag for messages received with this handle, or {\tt MPI\_ANY\_TAG} (integer). \item[IN context] context of messages received with this handle. \item[IN mode] receive mode (state type, with two values: {\tt MPI\_STANDARD}, and {\tt MPI\_SECURE}) \item[IN persistence] handle persistence (state type, with two values: {\tt MPI\_PERSISTENT} and {\tt MPI\_EPHEMERAL}) \end{description} See Section~\ref{subsec:envelope} for a discussion of source, tag and context. \discuss{ I have not included proposals for partially specified message handles, that some peoples seem to desire. I have merged all handle setup into one call. The functions {\tt MPI\_INIT\_SEND} and {\tt MPI\_INIT\_RECV} have the same list of parameters. Jim Cownie suggests to have only one function with an extra argument. Ephemeral ``persistent handles'' are not terribly useful as such. But their existence allow us to describe all the higher level send/receive operations in terms of basic operations on handles. } \subsection{Communication Start} \mpifunc{MPI\_START(handle)} \begin{description} \item[IN handle] communication handle \end{description} The {\tt MPI\_START} function starts the execution of a communication operation (send or receive). A sender should not update the send buffer after a send operation has started until after it has completed. A receiver should not access the receive buffer after a receive operation was started until after it has completed. A program that does not satisfy these conditions is erroneous and its outcome is undetermined. \subsection{Communication Completion} \label{subsec:complete_ops} \mpifunc{MPI\_WAIT ( handle, return\_status\_handle)} \begin{description} \item[IN handle] communication handle \item[IN return\_handle] handle that is associated with return status object. the return status object is updated by call. \end{description} A call to {\bf MPI\_WAIT} returns when the send operation identified by {\tt handle} is complete. The completion of a send operation indicates that the sender is now free to update the locations in the send buffer, or any other location that can be referenced by the send operation. However, it does not indicate that the message has been received; rather it may have been buffered by the communication subsystem. The completion of a receive operation indicates that the receiver is now free to access the locations in the receive buffer, which contain the received message, or any other location that can be referenced by the receive operation. It does not indicate that the matching send operation has completed. The call returns a handle to an opaque object that contains information on the completed operation -- the {\bf return status} object. \mpifunc{MPI\_STATUS (handle, flag, return\_handle)} \begin{description} \item[IN handle] communication handle \item[OUT flag] logical \item[OUT return\_handle] handle that is associated with return status object. \end{description} A call to {\bf MPI\_STATUS} returns {\tt flag=true} if the operation identified by {\tt handle} is complete. In such case, the return handle points to an opaque object that contains information on the completed operation. It returns {\tt flag=false}, otherwise. In such case, the value of the return handle is undefined. A call to {\tt MPI\_WAIT} blocks only the executing thread. If the executing process is multithreaded, then other threads within the process can be scheduled for execution. The use of a blocking receive operation ({\tt MPI\_WAIT}) allows the operating system to deschedule the blocked thread and schedule another thread for execution, if such is available. The use of a nonblocking receive operation ({\tt MPI\_STATUS}) allows the user to schedule alternative activities within a single thread of execution. The intended implementation of {\tt MPI\_STATUS} is for that operation to return as soon as possible. However, if repeatedly called for an operation that is enabled, it must eventually succeed. The return status object for a send operation carries no information. The return status object for a receive operation carries information on the source, tag and length of the received message. These fields are required because the receive operation may have specified {\tt DONTCARE} in either source or tag field, and the message may have been shorter than the receive buffer. \mpifunc{MPI\_QUERY( handle, count, source, tag)} \begin{description} \item[IN handle] handle to return status object \item[OUT count] number of elements received. \item[OUT source] rank of message sender in message context (integer). \item[OUT tag] tag of received message (integer). \end{description} \mpifunc{MPI\_CREATE\_STATUS( handle)} Creates a new return status object. \begin{description} \item[OUT handle] handle to return status object. \end{description} \discuss{ The use of a return status object, rather than a list of parameters may simplify the use of MPI routines, if the values stored in the object are seldom checked. A predefined return status object should be provided, to ease programming. The main reason for the use of a return-status object is that one wants to be able to use the same {\tt MPI\_WAIT} and {\tt MPI\_STATUS} calls for checking on different types of operations, e.g. both sends and receives, and perhaps new operations in the future. These different operations return different information (albeit we could decree that a send returns the same information as a receive). The issue of memory allocation for return\_status handles is not yet solved. There is a desire that the user will be able to allocate space for them (e.g., on the stack). We return number of elements received, rather than bytes received because ``number of elements'' is closer to the application semantic level (user does not need to be aware of the size of elements), and is more invariant. The return status object returns the number of {\bf elements} received. If there was no truncation, then this is equal to the number of elements sent. On the other hand, the number of bytes collected from the sender memory, the number of bytes sent over the wire, and the number of bytes stored in the receiver memory may all be different, in a heterogeneous environment. The number of elements sent can be computed form the send buffer descriptor; the number of elements received can be computed from the receive buffer descriptor and the length (in bytes) of the received message. {\tt MPI\_STATUS} and {\tt MPI\_QUERY} are not good names -- they can be easily confused. Leslie Hart suggests {\tt MPI\_TEST} or {\tt MPI\_CHECK} for the first. Perhaps we should replace {\tt MPI\_STATUS} with {\tt MPI\_TEST} and {\tt MPI\_QUERY} with {\tt MPI\_STATUS}. } \subsection{Multiple Completions} It is convenient to be able to wait for the completion of any or all the operations in a set, rather than having to wait for a specific message. A call to {\tt MPI\_WAITANY} or {\tt MPI\_STATUSANY} can be used to wait for the completion of one out of several operations; a call to {\tt MPI\_WAITALL} can be used to wait for all pending operations in a list. \mpifunc{MPI\_WAITANY ( list\_of\_handles, count, index, return\_handle)} Blocks until one of the operations associated with the communication handles in the array has completed. Returns the index of that handle in the array, and returns the status of that operation in the object associated with the return\_handle. The parameters are: \begin{description} \item[IN list\_of\_handles] list of handles to communication objects. \item[IN count] list length (integer) \item[OUT index] index of handle for operation that completed (integer). \item[OUT return\_handle] handle that is associated with return status object. Set to return status of operation that completed. \end{description} The successful execution of {\tt MPI\_WAITANY(list\_of\_handles, index, return\_handle)} has the same effect as the successful execution of {\tt MPI\_WAIT(handle[i], return\_handle)}, where {\tt i} is the value returned by {\tt index} and {\tt handle[i]} is the {\tt i}-th handle in the list, and the cancellation of all remaining wait operations. If more then one operation is enabled and can terminate, one is arbitrarily chosen (subject to the restrictions on operation termination order, and fairness, see Section~\ref{subsec:correct}). {\tt MPI\_WAITANY ( list\_of\_handles, count, index, return\_status\_handle)} is \begin{verbatim} {MPI_WAIT (handle[0], return_handle); index = 0} || ... || {MPI_WAIT (handle[count-1], return_handle); index = count-1} \end{verbatim} (``$||$'' indicates choice; one of the alternatives is chosen, nondeterministically.) \mpifunc{MPI\_STATUSANY ( list\_of\_handles, count, index, return\_handle)} Causes either one or none of the operations associated with the communication handles to return. In the former case, it has the same return semantics as a call to {\tt MPI\_WAIT\_ANY}. In the latter case, it returns a value of --1 in {\tt index} and {\tt return\_handle} is undefined. The parameters are: \begin{description} \item[IN list\_of\_handles] list of handles to communication objects. \item[IN count] list length (integer) \item[OUT index] index of handle for operation that completed, or -1 if none completed (integer). \item[OUT return\_handle] handle that is associated with return status object. Set to return status of operation that completed, if any; undefined when {\tt index = -1}. \end{description} \mpifunc{MPI\_WAITALL(list\_of\_handles, count, list\_of\_return\_handles)} Blocks until all communication operations associated with handles in the list complete, and return the status of all these operations. The parameters are: \begin{description} \item[IN list\_of\_handles] list of handles to communication objects. \item[IN count] lists length (integer) \item[OUT list\_of\_return\_handles] Must have the same length as the first list. Each return status object is set to the return status of the corresponding operation in the first list. \end{description} \discuss{ The fairness requirement given in Section~\ref{subsec:correct} implies that the use of {\tt WAIT\_ANY} cannot lead to starvation: If the sending process has issued a send complete operation, and the receiving process repeatedly post a receive for the message sent, then that message must be eventually received. Section~\ref{subsec:correct} has not yet been discussed. The fairness requirement can be attacked either for being too weak (eventually is not good enough), or too strong (hard to implement). In the later case, if the requirement of fairness is dropped from MPI implementations, then some mechanism need be provided to the user to achieve fairness by his or her own devices. One such proposal is to be able to specify a rotating priority order on the operations posted by a {\tt MPI\_WAITANY}: search the list sequentially, starting from a user specified position. } \section{Blocking Communication} Blocking send and receive operations combine all communication suboperations into one call. The operation returns only when the communication completes and no communication object persists after the call completed. However, the buffer descriptor object needs be created ahead of the call. We use the following naming convention for such operations: \[ \left[ \begin{array}{c} - \\ \bf R \\ \bf S \end{array} \right] \left[ \begin{array}{c} \bf SEND \\ \bf RECV \end{array} \right] \] The first letter (void, {\bf R} or {\bf S}) indicates the start mode (standard, ready, or secure). Only two of the combinations (standard and secure) are meaningful for receives. \mpifunc{MPI\_SEND (buffer\_handle, dest, tag, context)} is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, MPI_STANDARD, MPI_EPHEMERAL) MPI_START(handle) MPI_WAIT(handle, null) \end{verbatim} \mpifunc{MPI\_RSEND (buffer\_handle, dest, tag, context)} is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, MPI_READY, MPI_EPHEMERAL) MPI_START(handle) MPI_WAIT(handle, null) \end{verbatim} \mpifunc{MPI\_SSEND (buffer\_handle, dest, tag, context)} is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, MPI_SECURE, MPI_EPHEMERAL) MPI_START(handle) MPI_WAIT(handle, null) \end{verbatim} \mpifunc{MPI\_RECV(buffer\_handle, source, tag, context, return\_handle)} is \begin{verbatim} MPI_INIT_RECV(handle, buffer_handle, source, tag, context, MPI_STANDARD, MPI_EPHEMERAL) MPI_START(handle) MPI_WAIT(handle,return_handle) \end{verbatim} \mpifunc{MPI\_SRECV(buffer\_handle, source, tag, context, return\_handle)} is \begin{verbatim} MPI_INIT_RECV(handle, buffer_handle, source, tag, context, MPI_SECURE, MPI_EPHEMERAL) MPI_START(handle) MPI_WAIT(handle,return_handle) \end{verbatim} \implement{ While these functions can be implemented via calls to functions that implement suboperations, as described in this subsection, an efficient implementation may optimize away these multiple calls, provided it does not change the behavior of correct programs. } \discuss{ We use a different function name, rather than an additional mode parameter, in order to save on an additional parameter (performance and user convenience). Lyndon and perhaps other prefer fewer functions and an additional parameter. } \section{Nonblocking Communication} Nonblocking send and receive operations combine the first two suboperations ({\tt INIT} and {\tt START}) into one call. They use ephemeral communication objects, so that the operation is completed, and the associated resources are freed, by using one of the functions {\tt MPI\_WAIT, MPI\_STATUS, MPI\_WAITANY, MPI\_STATUSANY}, or {\tt MPI\_WAITALL}. Here, too, a buffer object has to be created ahead of the communication initiation operation. We use the same naming convention as for blocking operations: a prefix of {\bf R} ({\bf S}) indicates the {\tt READY} ({\bf SECURE}) mode. In addition, a prefix of {\bf I} is used to indicate {\em immediate} (i.e., nonblocking) execution. \mpifunc{MPI\_ISEND (handle, buffer\_handle, dest, tag, context)} is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, MPI_STANDARD, MPI_EPHEMERAL) MPI_START(handle) \end{verbatim} \mpifunc{MPI\_IRSEND (handle, buffer\_handle, dest, tag, context)} is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, MPI_READY, MPI_EPHEMERAL) MPI_START(handle) \end{verbatim} \mpifunc{MPI\_ISSEND (handle, buffer\_handle, dest, tag, context)} is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, MPI_SECURE, MPI_EPHEMERAL) MPI_START(handle) \end{verbatim} \mpifunc{MPI\_IRECV(handle, buffer\_handle, source, tag, context, return\_status\_handle)} is \begin{verbatim} MPI_INIT_RECV(handle, buffer_handle, source, tag, context, MPI_STANDARD, MPI_EPHEMERAL) MPI_START(handle) \end{verbatim} \mpifunc{MPI\_ISRECV(handle, buffer\_handle, source, tag, context, return\_status\_handle)} is \begin{verbatim} MPI_INIT_RECV(handle, buffer_handle, source, tag, context, MPI_SECURE, MPI_EPHEMERAL) MPI_START(handle) \end{verbatim} \section{Contiguous Block Communication Operations} The most frequent type of buffer used is a buffer with one contiguous component. We specialize the functions in the two previous subsections to this case, thus avoiding the need for the creation of a buffer descriptor object. We use the same naming scheme used in the previous subsections, and append a {\bf C} in the function name, for {\tt CONTIGUOUS}. \mpifunc{MPI\_SENDC (start, count, datatype, dest, tag, context)} is \begin{verbatim} MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL) MPI_APPEND( buffer_handle, start, count, datatype) MPI_SEND (buffer_handle, count, dest, tag, context) \end{verbatim} \mpifunc{MPI\_RSENDC (handle, start, count, datatype, dest, tag, context)} is \begin{verbatim} MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL) MPI_APPEND( buffer_handle, start, count, datatype) MPI_RSEND( buffer_handle, count, dest, tag, context) \end{verbatim} \mpifunc{MPI\_SSENDC (handle, start, count, datatype, dest, tag, context)} is \begin{verbatim} MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL) MPI_APPEND( buffer_handle, start, count, datatype) MPI_SSEND( buffer_handle, count, dest, tag, context) \end{verbatim} \mpifunc{MPI\_RECVC (start, count, datatype, source, tag, context, return\_status\_handle)} is \begin{verbatim} MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL) MPI_APPEND( buffer_handle, start, count, datatype) MPI_RECV( buffer_handle, source, tag, context, return_status_handle) \end{verbatim} \mpifunc{MPI\_SRECVC (start, count, datatype, source, tag, context, return\_status\_handle)} is \begin{verbatim} MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL) MPI_APPEND( buffer_handle, start, count, datatype) MPI_SRECV( buffer_handle, source, tag, context, return_status_handle) \end{verbatim} \mpifunc{MPI\_ISENDC (handle, start, count, datatype, dest, tag, context)} is \begin{verbatim} MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL) MPI_APPEND( buffer_handle, start, count, datatype) MPI_ISEND( handle, buffer_handle, dest, tag, context) \end{verbatim} \mpifunc{MPI\_IRSENDC (handle, start, count, datatype, dest, tag, context)} is \begin{verbatim} MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL) MPI_APPEND( buffer_handle, start, count, datatype) MPI_IRSEND( handle, buffer_handle, dest, tag, context) \end{verbatim} \mpifunc{MPI\_ISSENDC (handle, start, count, datatype, dest, tag, context)} is \begin{verbatim} MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL) MPI_APPEND( buffer_handle, start, count, datatype) MPI_ISSEND( handle, buffer_handle, dest, tag, context) \end{verbatim} \mpifunc{MPI\_IRECVC(handle, start, count, datatype, source, tag, context, return\_status\_handle)} is \begin{verbatim} MPI_CREATE_BUFFER (buffer_handle, MPI_EPHEMERAL) MPI_APPEND( buffer_handle, start, count, datatype) MPI_IRECV( handle, buffer_handle, source, tag, context) \end{verbatim} \mpifunc{MPI\_ISRECVC(handle, start, count, datatype, source, tag, context, return\_status\_handle)} is \begin{verbatim} MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL) MPI_APPEND( buffer_handle, start, count, datatype) MPI_ISRECV( handle, buffer_handle, source, tag, context) \end{verbatim} \section{Probe and Cancel} The {\tt MPI\_PROBE} operation allows incoming messages to be checked for, without actually receiving them. The user can then decide where to receive them, based on the information returned by the probe (basically, the information on the message envelope). An additional function, {\tt MPI\_GET\_LEN} allows the amount of storage needed to receive the message to be computed, when this length is not readily computed from the information returned by {\tt MPI\_PROBE}. The {\tt MPI\_CANCEL} operation allows pending communications to be cancelled. This is required for cleanup. Posting a send or a receive ties user resources (send or receive buffers), and a cancel may be needed to free these resources gracefully. \mpifunc{MPI\_PROBE( source, tag, context, flag, datatype, return\_status)} \begin{description} \item[IN source] rank in context of source, or {\tt MPI\_ANY\_SOURCE} (integer). \item[IN tag] user tag for messages received with this handle, or {\tt MPI\_ANY\_TAG} (integer). \item[IN context] context of messages received with this handle. \item[OUT flag] (logical) \item[IN type] assumed type of data in message (status variable). \item[OUT return\_handle] handle that is associated with return status object. \end{description} {\tt MPI\_PROBE} returns {\tt flag = true} if there is a message that can be received and that matches the pattern specified by the parameters {\tt source}, {\tt tag}, and {\tt context}. It returns {\tt flag = false}, otherwise. If {\tt MPI\_PROBE} returns {\tt flag = true}, then the length, source and tag of the message matched are returned in the return status object. These are the same values that would have been returned by a call to {\tt MPI\_RECV} or to {\tt MPI\_SRECV} executed at the same point in the program (with a caveat concerning length; see below). These values can be decoded from the return status object using the {\tt MPI\_RETURN\_STAT} function. The value returned in the return status object is undefined if {\tt flag=false}. The length value returned by the return status object is (correctly) the number of elements in the message, provided that all elements in the message are of the type specified by the {\tt type} parameter; otherwise the length value returned is undefined. A subsequent receive executed with the same context, and the source and tag returned by the call to {\tt MPI\_PROBE} will receive the message that was matched by the probe, if no other intervening receive occurred after the probe. If the receiving process is multithreaded, it is the user responsibility to ensure that the last condition holds. \discuss{ Do we want {\tt MPI\_PROBE} to return mode (standard or secure)? If yes, we need to carry mode information with messages (it's deja vu all over again). If not, then it is up to the user to encode this information (in the tag) so that receiver can decide whether to use a secure or standard send. MPI guarantees that successive messages sent from a source to a destination within the same context are received in the order they are sent. Thus, MPI must support, either explicitly or implicitly, a FIFO structure to manage messages between each pair of messages, for each context. {\tt MPI\_PROBE} returns information on the first matching message in this FIFO; this will also be the message received by the first subsequent receive with the same source, tag and context as the message matched by {\tt MPI\_PROBE}. Message passing in MPI can be implemented without appending type information to messages. A message is merely a string of bytes and the interpretation of these bytes into a sequence of typed elements is done using the information in the buffer descriptors at each end. The ability to use such implementation strategy is deemed to be an important goal. In such implementation, when a message arrives, it is not be known how many elements it contains, or even how much storage is needed to receive that message (because of possible representation conversion in a heterogeneous environment). The probe function cannot use a buffer descriptor; this defeats the purpose of probing in order to decide where to receive a message. Therefore, probe cannot, in general, return correct length information. Still, it is often the case that probe is used to decide how much storage to allocate in order to receive a message. Encoding such information in the message tag is deemed to be too awkward, and it is deemed important for {\tt MPI\_PROBE} to return some useful size information. The current definition of {\tt MPI\_PROBE} is a compromise between these two goals. For the most common case of messages where all entries have the same type {\tt MPI\_PROBE} returns the correct length information; the more esoteric case is handled by the {\tt MPI\_GET\_LEN} that is described below. The current solution saves us the need for one additional word per message that would otherwise be needed to transfer the message length (in elements) with the message. There was a request for a blocking probe function. } \mpifunc{MPI\_GET\_LEN( count, return\_status, buffer\_descriptor)} Computes the number of elements that were to be received, if the message that is associated with the return status handle would be received in the buffer associated with the buffer descriptor handle. \begin{description} \item[OUT count] number of elements that were to be received (integer) \item[IN return\_status] handle to return status descriptor \item[IN buffer\_descriptor] handle to buffer descriptor \end{description} \discuss { In order to support this function, an additional field is needed in the return status object, i.e., number of bytes in the incoming message matched by probe, or value of datatype parameter provided by probe (unless we add this parameter to the {\tt MPI\_GET\_LEN} function, which is too ugly to bear). } \mpifunc{MPI\_CANCEL( handle, flag)} \begin{description} \item [IN handle] handle to communication object \item[OUT flag] (logical) \end{description} A call to {\tt MPI\_CANCEL} cancels a pending communication operation (send or receive). The call returns {\tt flag = true} if the cancel operation succeeded, {\tt flag = false} otherwise. It must be the case that either the cancel operation succeeds or that the pending communication operation completes (but not both). If a communication operation was cancelled successfully, the effect is as if this operation was never executed. If a send was cancelled successfully then the message sent will not be received, the receive buffer of any posted receive for that message will not be altered, and any such receive has to be satisfied by another send; if a receive was cancelled successfully, then the receive buffer posted will not be altered, and any send matching this receive has to be satisfied by another receive. \implement{ There is not expectation that a cancel operation will be fast. } \section{Correctness} \label{subsec:correct} \discuss{The material in this section has not yet been discussed by MPIF. Some or all of it is likely to move to Section~\ref{sec:formal}. It is incorporated here for completeness.} \subsection{Order} MPI preserves the order of messages within the same context, between any fixed pair of processes. In other words, if process A executes two successive send {\tt start} suboperations within the same context, process B executes two successive receive {\tt start} operations, and both receives match either sends, then the first receive will receive the message sent by the first send, and the second receive will receive the message sent by the second send. Thus, if a two messages from the same source can satisfy a pending receive, the first message sent is accepted; if a message can satisfy two pending receives, the first receive posted is satisfied. The last paragraph assumes that the send {\tt start} operations are ordered by the program order at process A, and the receive {\tt start} operations are ordered by the program order at process B. If a process is multithreaded and the operations are executed by distinct threads, then the semantics of the threaded system may not define an order between the two operations, in which case the condition is void. \subsection{Progress and Fairness} We can model the execution of MPI programs as an interaction between executing processes that execute each their own program, and the {\bf communication subsystem}. The communication subsystem may have various constraints on the amount of resources it can use. E.g.: Bounds on the number and total sizes of active communication objects. Such bound can be global, per node, or per pair of communicating nodes. Bounds on the number and total sizes of messages buffered in the system. Such bound can, again, be global, per node, or per pair of communicating node. In addition, a message may be buffered at the sender, at the receiver, at both, or perhaps at another place altogether. Thus, it will be difficult to set rules on resource management of the communication subsystem. However, it is generally expected that implementers will provide information on the mechanism used for resource allocation, and that query and set functions will allow to query and possibly control the amount of available resources. We provide in this section a set of minimal requirements on the communication subsystem. Programs that execute on any subsystem that fulfils these minimal requirements are {\bf safe} and will port to any MPI implementation. {\bf Unsafe} programs may execute on some MPI implementations, depending on the amount of available resources and the implementation used for the MPI communication subsystem. Finally {\bf erroneous} programs never execute correctly. (While it is desirable to detect erroneous programs, it is not possible to do so at compile time, and often prohibitive to do so a run time. Thus, the document does not specify a behavior for erroneous programs, although the desired behavior is to return a useful error message.) \begin{enumerate} \item If a process executes an {\tt INIT} operation, then the operation eventually succeeds, or a {\em resource exception} occurs. The standard does not specify when a resource exception is allowed to occur. It is expected that an operational definition will be made available, in the form of test programs that have to execute with no resource exceptions. It is highly desirable to have generous bounds on the number of concurrently active communication objects each process may have, so that, in practice, {\tt INIT} operations will always be guaranteed to succeed. \item Each process can initiate a communication operation for each active communication object. I.e. correct {\tt START} operations always succeed (eventually). \item A send operation is {\bf enabled} if the sending process has issued a {\tt COMPLETE} operation and the receiving process has issued a {\tt START} operation for a matching receive. Symmetrically, a receive operation is {\bf enabled} if the receiving process has issued a {\tt COMPLETE} operation and the sending process has issued a {\tt START} operation for a matching send. An enabled operation may become {\bf disabled} either because it completes successfully or, in the case of a receive, because the matching message is successfully received by another receive operation or, in the case of a send, because the matching receive successfully receives another message. {\bf An enabled operation either completes successfully or becomes permanently disabled.} I.e., an enabled operation either eventually succeeds, or becomes disabled (progress); and an operation cannot be enabled infinitely many times without ever succeeding (fairness). \item A {\tt FREE} operation always succeeds (eventually). \end{enumerate} The four conditions guarantee progress in the communication subsystem. The third condition guarantee (weak) fairness among competing communication requests. Examples (involving two processors with ranks 0 and 1) The following program is safe, and should always succeed. \begin{verbatim} CALL MPI_RANK(rank, context) IF (rank.EQ.0) THEN CALL MPI_SENDC(sendbuf, count, 1, tag, context) CALL MPI_RECVC(recvbuf, count, 1, tag, context) ELSE ! rank.EQ.1 CALL MPI_RECVC(recvbuf, count, 0, tag, context) CALL MPI_SENDC(sendbuf, count, 0, tag, context) END IF \end{verbatim} The following program is erroneous, and should always fail. \begin{verbatim} CALL MPI_RANK(rank, context) IF (rank.EQ.0) THEN CALL MPI_RECVC(recvbuf, count, 1, tag, context) CALL MPI_SENDC(sendbuf, count, 1, tag, context) ELSE ! rank.EQ.1 CALL MPI_RECVC(recvbuf, count, 0, tag, context) CALL MPI_SENDC(sendbuf, count, 0, tag, context) END IF \end{verbatim} The receive operation of the first process must complete before its send, and can complete only if the matching send of the second processor is executed; the receive operation of the second process must complete before its send and can complete only if the matching send of the first process is executed. This program will deadlock. The following program is unsafe, and may succeed or fail, depending on implementation. \begin{verbatim} CALL MPI_RANK(rank, context) IF (rank.EQ.0) THEN CALL MPI_SENDC(sendbuf, count, 1, tag, context) CALL MPI_RECVC(recvbuf, count, 1, tag, context) ELSE ! rank.EQ.1 CALL MPI_SENDC(sendbuf, count, 0, tag, context) CALL MPI_RECVC(recvbuf, count, 0, tag, context) END IF \end{verbatim} The message sent by each process has to be copied out before the send operation returns and the receive operation starts. For the program to complete, it is necessary that at least one of the two messages sent is buffered out of either processes' address space. Thus, this program can succeed only if the communication system has sufficient buffer space to buffer {\tt count} words of data. If additional requirements will become part of the standard (e.g., bounds on the minimal number of concurrently active handles that need be supported, then further programs become safe. \section{Missing} \paragraph*{send-receive} a function of the form (for blocking communication) {\tt MPI\_SEND\_RECV( send\_buffer, destination, recv\_buffer, source, tag, context)} It may be superfluous because the collective communication shift has similar functionality. Yet it allows to generate more easily communication patterns that would lead to deadlock if implemented naively, with blocking communication. An important variant is the case where {\tt send\_buffer = recv\_buffer} (this can either be a subcase of the general send\_recv function, or a separate function, if send buffer and receive buffer are required to be disjoint in the general function). This introduces new functionality that is not available, otherwise, and which seems to be often required. An {\tt exchange} is the special case where source = destination. \end{document} >From weeks@mozart.convex.com Wed Jun 9 15:10:24 1993 Return-Path: Received: from ens.ens-lyon.fr by lip.ens-lyon.fr (4.1/SMI-4.1) id AA15090; Wed, 9 Jun 93 15:10:24 +0200 Received: from convex.convex.com by ens.ens-lyon.fr (4.1/SMI-4.1) id AA12755; Wed, 9 Jun 93 15:10:03 +0200 Received: from mozart.convex.com by convex.convex.com (5.64/1.35) id AA01731; Wed, 9 Jun 93 08:09:24 -0500 Received: by mozart.convex.com (5.64/1.28) id AA29119; Wed, 9 Jun 93 08:11:37 -0500 Date: Wed, 9 Jun 93 08:11:37 -0500 From: weeks@mozart.convex.com (Dennis Weeks) Message-Id: <9306091311.AA29119@mozart.convex.com> To: Jack.Dongarra@lip.ens-lyon.fr Subject: May 14 mpi mail #1 of 1 Status: R >From snir@watson.ibm.com Fri May 14 00:15:16 1993 Received: from convex.convex.com by mozart.convex.com (5.64/1.28) id AA28640; Fri, 14 May 93 00:14:55 -0500 Received: from watson.ibm.com by convex.convex.com (5.64/1.35) id AA02615; Fri, 14 May 93 00:15:01 -0500 Message-Id: <9305140515.AA02615@convex.convex.com> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 1089; Fri, 14 May 93 01:14:38 EDT Date: Fri, 14 May 93 01:14:38 EDT From: "Marc Snir" To: weeks@convex.com, tony@cs.msstate.edu Status: R % MPI Authors: % This is MY version of YOUR chapter. It has a wrapper so that you % should be able to simply LaTeX this. % % Please work from this text so that we are in synch. % % --Steve Otto \documentstyle[twoside]{article} \pagestyle{headings} %\markright{ {\em Draft Document of the MPI Standard,\/ \today} } \marginparwidth 0pt \oddsidemargin=.25in \evensidemargin .25in \marginparsep 0pt \topmargin=-.5in \textwidth=6.0in \textheight=9.0in \parindent=2em % ---------------------------------------------------------------------- % mpi-macs.tex --- man page macros, % discuss, missing, mpifunc macros % % ---------------------------------------------------------------------- % a couple of commands from Marc Snir, modified S. Otto \newlength{\discussSpace} \setlength{\discussSpace}{.7cm} \newcommand{\discuss}[1]{\vspace{\discussSpace} {\small {\bf Discussion:} #1} \vspace{\discussSpace} } \newcommand{\missing}[1]{\vspace{\discussSpace} {\small {\bf Missing:} #1} \vspace{\discussSpace} } \newcommand{\implement}[1]{\vspace{\discussSpace} {\small {\bf Implementation note:} #1} \vspace{\discussSpace} } \newlength{\codeSpace} \setlength{\codeSpace}{.3cm} \newcommand{\mpifunc}[1]{\vspace{\codeSpace} {\bf #1} \vspace{\codeSpace} } % ----------------------------------------------------------------------- % A few commands to help in writing MPI man pages % \def\twoc#1#2{ \begin{list} {\hbox to95pt{#1\hfil}} {\setlength{\leftmargin}{120pt} \setlength{\labelwidth}{95pt} \setlength{\labelsep}{0pt} \setlength{\partopsep}{0pt} \setlength{\parskip}{0pt} \setlength{\topsep}{0pt} } \item {#2} \end{list} } \outer\long\def\onec#1{ \begin{list} {} {\setlength{\leftmargin}{25pt} \setlength{\labelwidth}{0pt} \setlength{\labelsep}{0pt} \setlength{\partopsep}{0pt} \setlength{\parskip}{0pt} \setlength{\topsep}{0pt} } \item {#1} \end{list} } \def\manhead#1{\noindent{\bf{#1}}} \hyphenation{RE-DIS-TRIB-UT-ABLE sub-script mul-ti-ple} \begin{document} \setcounter{page}{1} \pagenumbering{roman} \title{ {\em D R A F T} \\ Maps, Groups and Contexts } \author{ Lyndon Clarke, Mark Sears, Tony Skjellum, Marc Snir} \date{May 13, 1993} \maketitle \hfuzz=5pt %\tableofcontents %\begin{abstract} %We don't have an abstract yet. %\end{abstract} \setcounter{page}{1} \pagenumbering{arabic} \section{Introduction} Goals Assumptions (closed universe -- to be extended later) Main objects: Maps and communicators \section{Maps} A {\bf map} is a one-to one-mapping $f: 0...m-1 \rightarrow 0...n-1$. A possible representation for such map is a list of $m$ integers in the range $0...n-1$; it is often convenient to think of a map as being such list. Another possible representation is as an algorithmic specification of the map (e.g., hash function). We represent a map by an opaque map object. A {\bf group} is an ordered set of processes (where a process is a primitive of MPI discussed elsewhere). Each process in a group is associated with a {\bf rank}, starting from zero. We shall use maps in order to represent groups. If processes are numbered from $0$ to $n-1$, then a map $f: 0...m-1 \rightarrow 0...n-1$ represents a subset group of $m$ processes. Note that a map, by itself, does not represent a group; it does so only relative to a pre-defined containing group. \section{Context} A {\bf context} is the MPI mechanism for partitioning communication space. A send made in a context cannot be received in another context. Contexts are identified in MPI using {\bf context\_id}'s. \section{Context ID's} A {\bf context\_id} is a integer-like type that can be shared between processes in MPI. \section{Communicators} All MPI communication (point to point and collective) functions use {\bf communicators} {this replaces the word {\bf context} everywhere in current pt2pt and collcomm drafts} to provide a context for the communication. Communicators bring together the concepts of group and context. The source and destination of a message is identified by the rank of that process within the group, and communication using this communicator is restricted to processes within this group. Thus, the communicator restricts the ``spatial'' scope of communication, and provides local process addressing. \discuss{ The current communicators correspond to C1 objects in the previous draft. Future extensions are expected to include C2 objects, that encapsulate two groups: the sender group and the receiver group. } Communicators are represented by opaque {\bf communicator objects}. Such objects are opaque, and cannot be transferred from one node to another. However, communicator objects can be identified using {\bf context\_id}'s; these can be communicated across processes. \discuss{ Is the datatype of context\_id's {\tt INTEGER} or MPI-defined?} Let {\tt comm} be a communicator that is associated with a group $G$ of size $n$, and let $f:0..m-1 \rightarrow 0..n-1$ be a map. Then the pair {\tt (comm, f)} defines a subgroup {\tt G(comm,f)} of $G$, namely the subgroup of processes with ranks $f(0),...,f(m-1)$. We say that {\tt map} is the map of the subgroup {\tt G(comm,f)} within {\tt comm}. An initial communicator {\tt MPI\_COMM\_INIT} is defined when the program starts. its associated group contains all processes that start the computation. \section{Operations on maps} \mpifunc{domain = MPI\_MAP\_DOMAIN(map)} \begin{description} \item[IN map] handle to map object \item[OUT domain] domain of map (integer) -- the domain is the set $0...{\tt domain} -1$. \end{description} \mpifunc{element = MPI\_MAP\_ELEMENT( map, rank)} \begin{description} \item[IN map] handle to map object \item[IN rank] map argument (integer) \item[OUT element] value of map on argument {\tt rank} (integer) \end{description} \mpifunc{rank = MPI\_MAP\_RANK( map, element)} \begin{description} \item[IN map] handle to map object \item[IN element] element in range of map \item[OUT rank] inverse image of element, MPI\_INVALID\_ELEMENT results if element is not in range \end{description} \mpifunc{MPI\_MAP\_FLATTEN(map, array\_of\_integer, domain)} \begin{description} \item[IN map] handle to map object \item[OUT array\_of\_integer] list of values in the range of {\tt map} \item[OUT domain] length of returned list -1 (integer). \end{description} \discuss{Definition of ``mpi\_map\_flatten'' is not definitive.} \section{Map constructors} The map constructors may be either local or may require communication. \subsection{Local map constructors} The execution of the following operation does not require interprocess communication. Examples of local map constructors are as follows: \mpifunc{MPI\_MAP\_BUILD(map, array\_of\_integer)} \begin{description} \item[OUT map] handle to map object \item[IN array\_of\_integer] list of values in the range of {\tt map} \item[IN domain] length of returned list -1 (integer). \end{description} \discuss{Syntax may change, (see also ``flatten'')} \mpifunc{MPI\_MAP\_CONCAT(map1, map2, mapout)} \begin{description} \item[IN map1] map object handle \item[IN map2] map object handle \item[OUT mapout] map object handle \end{description} If map1 and map2 are distinct, then is the concatenation of the maps. $f(i) = f_1(1) \rm{if} i \le {\rm dom}(f_1), {\rm else} = f_2(i-{\rm dom}(f_1))$. \subsection{Collective map constructors} The execution of the following operations require collective communication within a group. Examples are as follows: \mpifunc{MPI\_MAP\_SPLIT(comm, mapin, key, index, mapout)} This is a collective function that is called by all processes in the group associated with {\tt (comm, mapin)}. All calling processes provide the same values for the parameters {\tt comm} and {\tt mapin}. A separate group of processes is formed for each distinct value of {\tt key}, with the processes ordered according to the value of {\tt index}. Each process in such group is returned in {\tt mapout} the map of this group within {\tt comm}. \begin{description} \item[IN comm] communicator object handle \item[IN mapin] map object handle \item[IN key] (integer) \item[IN index] (integer) \item[OUT mapout] map object handle \end{description} \section{Operations on communicators} \mpifunc{MPI\_ALLOC\_CONTEXT(comm, map, array\_of\_contextids, len)} Allocates an array of contextids. This is a collective operation that is executed by all processes in the group defined by {\tt (comm, map)}. The context\_ids that are returned are unique within the group associated with {\tt (comm, map)}. The array returned is the same on all processes that call the function (same order, same number of elements). \begin{description} \item[IN comm] communicator object handle \item[IN map] map object handle \item[OUT array\_of\_contextids] \item[IN len] number of context\_id's to allocate \end{description} \mpifunc{MPI\_MAKE\_COMM(comm, map, context\_id, newcomm)} Creates a new communicator object, which is associated with the group defined by {\tt (comm, map)}, and the contextid {\tt contextid}. The operation does not require communication. However, the new communicator object should not be used for communication between two processes unless they both have called the function. It is erroneous to invoke this function twice in the same process with the same context\_id (unless the context\_id has been freed). \discuss{Mark Sears disagrees with this!} \begin{description} \item[IN comm] communicator object handle \item[IN map] map object handle \item[IN contextid] contextid \item[OUT newcomm] communicator object handle \end{description} \mpifunc{MPI\_SAFEMAKE\_COMM(comm, map, newcomm)} is \begin{verbatim} MPI_ALLOC_CONTEXT(comm, map, context_id, 1); MPI_MAKE_COMM(comm, map, context_id, newcomm); ``MPI_SYNCH(comm,map)'' /* by which we mean a synchronization operation in the subgroup defined by (comm,map) */ \end{verbatim} \discuss{ {\tt MPI\_SYNCH(comm,map)} is important to bootstrap -- but can be coded using point-to-point.} {\tt MPI\_SAFEMAKE\_COMM} creates a new communicator object and synchronizes all processes in the group associated with this object. Thus, when this call returns, the new communicator can be safely used. \mpifunc{MPI\_MAP(comm, subcomm, map)} Returns a map such that {\tt (comm, map)} is the group associated with {\tt subcomm} ({\em i.e.}, returns the indices in the group associated with {\tt comm} for all the processes in the group associated with {\tt subcomm}). \begin{description} \item[IN comm] communicator object handle \item[IN subcomm] communicator object handle \item[OUT map] map object handle \end{description} \mpifunc{MPI\_CONTEXTID(comm, contextid)} Returns the context\_id associated with the communicator {\tt comm}. \begin{description} \item[IN comm] communicator object handle \item[OUT contextid] contextid \end{description} \discuss{Possible addition of functions are as follows: {\tt MPI\_PARENT(comm)} returns the communicator used to create {\tt comm}; } \end{document} >From weeks@mozart.convex.com Wed Jun 9 15:10:57 1993 Return-Path: Received: from ens.ens-lyon.fr by lip.ens-lyon.fr (4.1/SMI-4.1) id AA15095; Wed, 9 Jun 93 15:10:53 +0200 Received: from convex.convex.com by ens.ens-lyon.fr (4.1/SMI-4.1) id AA12679; Wed, 9 Jun 93 15:06:13 +0200 Received: from mozart.convex.com by convex.convex.com (5.64/1.35) id AA01598; Wed, 9 Jun 93 08:04:35 -0500 Received: by mozart.convex.com (5.64/1.28) id AA28422; Wed, 9 Jun 93 08:06:42 -0500 Date: Wed, 9 Jun 93 08:06:42 -0500 From: weeks@mozart.convex.com (Dennis Weeks) Message-Id: <9306091306.AA28422@mozart.convex.com> To: Jack.Dongarra@lip.ens-lyon.fr Subject: May 12 mpi mail #6 Status: R A postscript file from snir or otto (I think this is different from the LaTeX source I forwarded just before this one...) %!PS-Adobe-2.0 %%Creator: dvips 5.47 Copyright 1986-91 Radical Eye Software %%Title: PT2PT-V3.DVI.* %%Pages: 29 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: texc.pro /TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{ ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]/id ch-image N /rw ch-width 7 add 8 idiv string N /rc 0 N /gp 0 N /cp 0 N{rc 0 ne{rc 1 sub /rc X rw}{G}ifelse}imagemask restore}B /G{{id gp get /gp gp 1 add N dup 18 mod S 18 idiv pl S get exec}loop}B /adv{cp add /cp X}B /chg{rw cp id gp 4 index getinterval putinterval dup gp add /gp X adv}B /nd{/cp 0 N rw exit}B /lsh{rw cp 2 copy get dup 0 eq{pop 1}{dup 255 eq{pop 254}{dup dup add 255 and S 1 and or}ifelse}ifelse put 1 adv}B /rsh{rw cp 2 copy get dup 0 eq{pop 128}{dup 255 eq{pop 127}{dup 2 idiv S 128 and or}ifelse}ifelse put 1 adv}B /clr{rw cp 2 index string putinterval adv}B /set{rw cp fillstr 0 4 index getinterval putinterval adv}B /fillstr 18 string 0 1 17{2 copy 255 put pop}for N /pl[{adv 1 chg}bind{adv 1 chg nd}bind{1 add chg}bind{1 add chg nd}bind{adv lsh}bind{ adv lsh nd}bind{adv rsh}bind{adv rsh nd}bind{1 add adv}bind{/rc X nd}bind{1 add set}bind{1 add clr}bind{adv 2 chg}bind{adv 2 chg nd}bind{pop nd}bind]N /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval (Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail} B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{ /SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 6 54 df20 DI50 DI<12E0B3B3B3B1EAFFFCA3 0E4A73811C>I<131CB3B3B3B1EAFFFCA30E4A80811C>I E /Fb 1 4 df<12041206A2EAC660EA F7E0EA3F80EA1F00EA3F80EAF7E0EAC660EA0600A212040B0D7E8D11>3 D E /Fc 2 36 df<1560A28181A2150EB71280A2C8EA0E001518A25D5DA2210E7E8F26>33 D<1203B3A7EA8304EAE31CEA7B78EA1FE0EA0FC0EA0780EA03007E0E217E9913>35 D E /Fd 2 63 df<127012F8A3127005057D840C>58 D<12E01278121EEA0780EA01E0EA007813 1EEB0780EB01E0EB0078141EEC0780A2EC1E001478EB01E0EB0780011EC7FC1378EA01E0EA0780 001EC8FC127812E019187D9520>62 D E /Fe 4 107 df0 D15 D<12C012F0123C120FEA03C0EA00F0133C130FEB03C0EB00F0143C140FEC0380EC0F00143C14F0 EB03C0010FC7FC133C13F0EA03C0000FC8FC123C12F012C0C9FCA7B61280A219227D9920>21 D<12C0B3B3A502297B9E0C>106 D E /Ff 36 122 df<1238127C127EA2123E121EA2127C12F8 1260070A7A8414>44 D<127012F8A312700505798414>46 D48 D<12035AA25A123FB4FC12F712471207ACEAFFF0A30C177C9614>I65 DIIIII I<38FE3F80A338380E00A7EA3FFEA3EA380EA738FE3F80A311177F9614>II75 DI<38FE0FE0A3383B1B80A413BBA2EA39B3A313F3EA38E3A21303A538FE0FE0 A31317809614>I<38FE3F80A3383B0E00A4138E1239A213CEA31238A213EE136EA4EAFE3EA311 177F9614>IIIIIII<38FF8FF8A338 1C01C0AEEA1E03000E1380EA0F073807FF006C5AEA00F81517819614>I<38FE3F80A338380E00 A26C5AA56C5AA4EA0630EA0770A3EA0360A213E0A26C5A11177F9614>I<38FE0FE0A338380380 A50018130013E3EA1DF7A213B7A5EA0DB6A21316EA0F1EA31317809614>I<38FE3F80A3383C1E 00EA1C1CEA1E3CEA0E38A26C5AA2EA036013E0A26C5AA7EA07F0A311177F9614>89 D91 D 93 D97 D101 D<1203EA0780A2EA0300C7FCA4EAFF80 A31203AAEAFFFEA30F187D9714>105 D110 D112 D<1207A5EAFFFCA3EA0700A6130EA3139E13FC EA03F8EA00E00F157F9414>116 D<38FF3F80A3381C1C00A2120E5BA212071330A2EA0370A26C 5AA35BA3EA7B80127F90C7FC127E123C11187F8F14>121 D E /Fg 29 86 df<1230127812F81278127005057D840C>46 D<13181370EA03F0120FEA1C701200A213E0A6EA 01C0A6EA0380A61207EAFFF8A20D1C7C9B15>49 D<137E3801FF80380387C0EA0603EB01E0120F A2121FA2120E1200EB03C0A2EB0780EB0F00130E5B5B13605BEA038038070180120E381C030012 30EA7FFF485AA2131C7E9B15>I<137C48B4FC38038780EA0603000F13C0A3380E078012001400 130E133CEA01F8485AEA001C7F130FA4127012F8A2EAF01E12E05BEA7078EA3FE0EA0F80121D7D 9B15>II<380601C03807FF 80140013FC0006C7FCA25AA413F8EA0FFCEA1F0EEA1C0F7F121800001380A3EB0F00127012F013 0EEAE01EEAC01CEA6038EA70F0EA3FE0EA0F80121D7D9B15>I<1218383FFFC0A21480EA700138 6003001306485AC65A5BA25B5B12015B1203A248C7FCA25A120EA2121EA35AA31218121D7B9B15 >55 D<137C48B4FC38038780EA0703120EEB01C0A2381E0380A2EB0700EA0F8613DCEA07F81203 487EEA0E7E487EEA380F12707F12E0A31306130EEA701CEA3878EA1FF0EA0FC0121D7D9B15>I< 13F8EA03FCEA070EEA0E07121CA2003C138012381278A4130F1400A2EA381F5BEA1FEFEA0FDEEA 001E131CA2EA603812F05B485AEAC3C0EA7F80003EC7FC111D7C9B15>I<14301470A214F08013 01A2EB0378A21306147CEB0C3CA21318A21330A2497EA2EBFFFEA23801801EA23803001F801206 120F397FC0FFF012FF1C1D7F9C1F>65 D<380FFFFC14FF3900F00780EC03C0A2EC01E0A2D801E0 13C01403A2EC0780EC1F00EBFFFC5AEBC03E140FEC0780A3EA0780A4EC0F00141E380F007CB55A 14E01B1C7E9B1D>I<903807F030EB1FFC90387C0E603901F003E0EA03C038078001EA0F00000E 1300001E14C05AA25A1500A25AA5EC0180EC03001270127814066C5B001C131C000F5B3807C0E0 6CB45AD8007EC7FC1C1E7C9C1E>I<380FFFFCECFF803900F007C0EC01E0EC00F015701578485A 1538A3153C1538485A1578A3157015F0484813E0140115C0EC03801407EC0E00380F007CB512F0 14C01E1C7E9B20>I<000FB512E0A23800F003EC01C01400A33801E060A31500EBE1E013FF485B 13C113C0A215C0ECC18038078001A2EC0300A25C140E380F003EB512FE5C1B1C7E9B1C>I<000F B512C0A23800F007EC03801401A3EA01E01461A2EC600014E013E148B45AA213C113C0A3380781 800180C7FCA5120FEAFFFC5B1A1C7E9B1B>I<903807F030EB1FFC90387C0E603901F003E0EA03 C038078001EA0F00000E1300001E14C05AA25A1500A25AA3ECFFF0A2EC0F80150012701278A27E 001C5B000F133E3807C0F63803FFC2C66CC7FC1C1E7C9C21>I<390FFF9FFE143F3900F003C0A5 3901E00780A590B5FC481400EBC00FA53807801EA6000F133E39FFF3FFE015C01F1C7E9B1F>I< 380FFF80A23800F000A5485AA6485AA6485AA6120FEAFFF8A2111C7F9B0F>I<3801FFF0A23800 0F00A5131EA65BA65BA2127012F8A2485A485AEAE1C0EA7F80001FC7FC141D7E9B15>I<390FFF 87FEA23900F003E0EC0180EC030014065C48485A14705CEBE1C013E313E73803CDE013D9EBF1F0 13E013C01478EA078080A280A280000FEB1F8039FFF87FF013F01F1C7E9B20>I<380FFFC0A238 00F000A5485AA6485AA41406140CEA0780A2141C141814381478380F01F8B512F0A2171C7E9B1A >II<390FF80FFEA23900FC01E0EC00C0A213DEA239018F0180A214811387A2EB83C1 390303C300EB01E3A3EB00F3A20006137EA2143EA3141E001E131C38FFE00C13C01F1C7E9B1F> II<380FFFFC14FF3900F00F80140315C01401A23801E003A315801407EC0F 003803C03EEBFFF814E001C0C7FCA3485AA6120FEAFFF85B1A1C7E9B1C>I<380FFFF814FE3900 F00F80140315C01401A23801E003A3EC0780EC0F00143E3803FFF85CEBC07880A2143E3807803C A51518000F1430EAFFF89038F01FE0C7EA07C01D1D7E9B1F>82 DI<001FB512F05A383C0781 0038EB806012701260A2EB0F0012C0A200001400A3131EA65BA6137C381FFFE0A21C1C7C9B1E> I<39FFF8FFE0A2390F001E00140CA4001E5BA6485BA6485BA300385BA2383C0180D81C03C7FCEA 0E0EEA07FCEA01F01B1D7A9B1F>I E /Fh 53 122 df<90381FE1F890B512FC3903F03F3E3807 C07EEB807C000F141C1500A5B612C0A2390F807C00AE397FE1FFC0A21F1D809C1C>11 D 13 D<13301360EA01C0EA038013005A120E121E121C123CA212381278A312F85AA97E1278A312 38123CA2121C121E120E7E7E1380EA01C0EA006013300C297D9E13>40 D<12C0126012387E120C 120E7E1380120313C0A2120113E0A313F01200A9120113E0A313C01203A2138012071300120E12 0C121C5A12605A0C297D9E13>I<1238127C12FE12FFA2127F123B120312071206A2120C121812 301220080F7D860D>44 DI<1238127C12FEA3127C123807077D860D>I< EA0FF0EA3FFCEA703EEAF03F12F8A3EA203EEA007C13F813F0EA01C0A21380A5C7FCA4EA0380EA 07C0EA0FE0A3EA07C0EA0380101D7D9C17>63 D65 DI<9038 1FE0209038FFF8E03803F80F3807E007380FC001EA1F8048C7FCA2007E1460A212FE1500A7007E 1460A27E15C06C7E390FC001803907E003003803F80E3800FFFCEB1FE01B1C7D9B22>IIII<90380FF00890387FFE383901FC07783907E001F8380FC0004848137848C71238A2481418 127E12FE1500A5EC7FFFA2007EEB01F8127F7EA2EA1F80EA0FC0EA07F03801FC0739007FFE7890 380FF818201C7D9B26>I<39FFFC3FFFA2390FC003F0AA90B5FCA2EBC003AC39FFFC3FFFA2201C 7E9B25>II76 DI<39FFE003FFA2390FF000307FEA0DFCEA0CFE137E7FEB1F8014C0 EB0FE0EB07F01303EB01F814FCEB00FE147F143FEC1FB015F0140F1407140314011400A2D8FFC0 13701530201C7E9B25>IIIII<3807F86038 1FFEE0EA3C07EA7801EA700012F01460A26C130012FEEAFFE0EA7FFE6C7E1480000F13C06C13E0 EA007FEB03F01301130012C0A214E07E38F001C0EAFC0338EFFF00EAC3FC141C7D9B1B>I<007F B512E0A238781F81007013800060146000E0147000C01430A400001400B03807FFFEA21C1C7E9B 21>I<39FFFC03FFA2390FC00030B3120715606C6C13E03901F001C03900FC078090387FFE00EB 0FF8201C7E9B25>I<3AFFFC01FF80A23A0FC00018006C6C5BA26D1370000314606D13E000015C 7F0000495AA2D97E03C7FCA2EB7F07EB3F06148EEB1F8C14CCEB0FD8A2EB07F0A36D5AA26D5AA2 211C7F9B24>I<3BFFFC7FFE0FFCA23B0FC007E000C081D9E003130100071680EC07F8D803F0EC 0300A29039F80CFC0700011506EC1CFE9039FC187E0E0000150CEC387F90397E303F18A290397F 601FB8013F14B002E013F0ECC00F011F5CA26D486C5AA2EC00036D5CA22E1C7F9B31>I<397FFE 1FFEA23907F001803803F8036C6C48C7FC000013066D5AEB7F1C6D5A14B0EB1FE0130FA26D7E6D 7E1307497EEB0CFEEB187EEB387F496C7EEB601F01C07F00016D7E496C7EEA03003AFFF03FFF80 A2211C7F9B24>I<3AFFFC01FF80A23A0FE00038006C6C13306C6C137015606C6C5B3800FE015D D97F03C7FCEB3F871486EB1FCEEB0FFC5C13076D5AAAEB3FFFA2211C7F9B24>I97 DIIII<133F3801FF803803E7C0EA07C7EA0F87EB8380EB8000A5EAFFF8A2EA0F80 AEEA7FF8A2121D809C0F>I<3803F0F0380FFFF8383E1F38383C0F30007C1380A4003C1300EA3E 1FEA1FFCEA33F00030C7FC12707EEA3FFF14C06C13E04813F0387801F838F00078A3007813F038 3E03E0381FFFC03803FE00151B7F9118>II<121E123FA25A7EA2121EC7FCA5B4FCA2121FAEEAFFE0A20B1E7F 9D0E>I107 DI<39FF1FC0FE 90387FE3FF3A1FE1F70F80903980FC07C0A2EB00F8AB3AFFE7FF3FF8A225127F9128>I<38FF1F C0EB7FE0381FE1F0EB80F8A21300AB38FFE7FFA218127F911B>II<38FF1FC0EBFFE0381FC1F8130014FC147C147EA6147C14FCEB80F8EBC1F0EB7FE0EB1F8090 C7FCA6EAFFE0A2171A7F911B>I114 DI<1203A35AA25AA2123FEAFFFCA2EA1F 00A9130CA4EA0F98EA07F0EA03E00E1A7F9913>I<38FF07F8A2EA1F00AC1301EA0F03EBFEFFEA 03F818127F911B>I<38FFC1FCA2381F0060EB80E0000F13C013C03807C180A23803E300A2EA01 F6A213FE6C5AA21378A2133016127F9119>I<38FFC7FCA2381F8180EA0F833807C700EA03EEEA 01FC5B1200137C13FEEA01DFEA039F38070F80380607C0380C03E038FF07FCA216127F9119> 120 D<38FFC1FCA2381F0060EB80E0000F13C013C03807C180A23803E300A2EA01F6A213FE6C5A A21378A21330A25B1270EAF8E0EAC0C0EAE380007FC7FC123E161A7F9119>I E /Fi 74 126 df<126012F0AF12601200A4126012F0A212600419779816>33 D38 D<13E01201EA07C013005A121E5A123812781270A312F05AA77E1270A312781238123C7E7E7E13 C0EA01E012000B217A9C16>40 D<12E07E127C121C121E7EEA0780120313C01201A313E01200A7 120113C0A3120313801207EA0F00121E121C127C12F05A0B217C9C16>III<1238127C127EA2123E120E121E123C127C12F812 60070B798416>II<127012F8A312700505788416>I48 DIII<137C13FC13DC1201EA039CA2EA071C120F120E121E123C1238127812F0 B512E0A338001C00A53801FFC0A313197F9816>II<13F8EA 03FEEA0FFFEA1F0F123E123CEA78060070C7FC12F0EAF7F8EAFFFEA2EAF80F38F00780A2EAE003 12F0A21270EA7807EB0F006C5AEA1FFEEA0FF8EA03E011197E9816>I<12E0B51280A338E00F00 131EC65A13381378137013F05B12015B12035BA3120790C7FCA7111A7E9916>III<1238127CA312381200A81238127CA3123C121C 123C123812F812F012600618799116>59 DII<13E0487EA213B0A2EA03B8A31318EA071C A5EA0E0EA2EA0FFEA2487EEA1C07A3387F1FC000FF13E0007F13C013197F9816>65 DI<3801F180EA07FF5AEA1F0FEA3C071278130312 7000F0C7FC5AA77E387003801278A2EA3C07381F0F00EA0FFE6C5AEA01F011197E9816>II<387FFFC0B5FC7EEA1C01A490C7FCA2131CA2EA1FFCA3EA1C1CA290C7 FC14E0A5EA7FFFB5FC7E13197F9816>II I<387F1FC038FFBFE0387F1FC0381C0700A7EA1FFFA3EA1C07A9387F1FC038FFBFE0387F1FC013 197F9816>II<387F0FE038FF8FF0387F0FE0381C07 80EB0F00131E131C133C5B5BEA1DE07F121F7F1338EA1E3C131CEA1C1E7F7F14801303387F07E0 38FF8FF0387F07E01419809816>75 D I<38FC07E0EAFE0FA2383A0B80EA3B1BA413BBA2EA39B3A413F3EA38E3A21303A538FE0FE0A313 197F9816>I<387E1FC038FF3FE0387F1FC0381D07001387A313C7A2121CA213E7A31367A21377 A21337A31317EA7F1FEAFF9FEA7F0F13197F9816>IIIIII<387FFFE0B5FCA2EAE0E0A400001300AFEA07FC487E6C5A13197F9816> I<387F07F038FF8FF8387F07F0381C01C0B0EA1E03000E1380EA0F8F3807FF006C5AEA00F81519 809816>I<38FE0FE0EAFF1FEAFE0F38380380381C0700A4EA0E0EA4EA060CEA071CA4EA031813 B8A3EA01B013F0A26C5A13197F9816>I<38FC07E0EAFE0FEAFC07387001C0A300301380EA3803 A313E3EA39F3A213B300191300A61313EA1B1BEA0F1EA2EA0E0E13197F9816>I<387F1F80133F 131F380E1E00131CEA073C1338EA03B813F012015B120012017F120313B81207131CA2EA0E0EA2 487E387F1FC000FF13E0007F13C013197F9816>I<38FE0FE0EAFF1FEAFE0F381C0700A2EA0E0E A26C5AA3EA03B8A2EA01F0A26C5AA8EA03F8487E6C5A13197F9816>I91 D93 D95 D97 D<127E12FE127E120EA4 133EEBFF80000F13C0EB83E01301EB00F0120E1470A4000F13F014E01381EB83C013FF000E1300 EA067C1419809816>II<133F5B7F1307A4EA03E7EA0FFF123FEA3C1F487E1270 EAF00712E0A46C5AA2EA781FEA7C3F383FFFE0381FF7F03807C7E014197F9816>II<131FEB7F8013FFEA01E7EBC30013C0A2EA7FFFB5FCA2EA01C0ACEA3FFE487E 6C5A11197F9816>I<3803E3C0380FFFE05A381E3CC0383C1E00EA380EA3EA3C1E6C5AEA1FFC48 5AEA3BE00038C7FC123CEA1FFC48B4FC4813C0EA780338F001E0EAE000A3EAF001387C07C0383F FF80380FFE00EA03F8131C7F9116>I<127E12FE127E120EA4137CEA0FFF148013871303A2120E A9387FC7F038FFE7F8387FC7F01519809816>II<127E12FE127E120EA4EB7FE0A3EB0F00131E5B5B5BEA0FF8A2 13BC131EEA0E0E130FEB0780387F87F0EAFFCFEA7F871419809816>107 DI<38FBC78038FFEFC0EBFFE0EA3E7CEA3C78EA3870 AA38FE7CF8A31512809116>IIII<38FF0F80EB3FE013FFEA07F1EBE0C0EBC0005BA290C7FCA7EAFFFCA313127F9116>114 DI<12035AA4EA7FFFB5FCA20007C7FCA75BEB0380A2130713 873803FF005BEA00F811177F9616>I<387E1F80EAFE3FEA7E1FEA0E03AA1307EA0F0FEBFFF06C 13F83803F3F01512809116>I<387F1FC000FF13E0007F13C0381C0700EA1E0FEA0E0EA36C5AA4 EA03B8A3EA01F0A26C5A13127F9116>I<387F1FC0133F131F380F1C00EA073CEA03B813F01201 6C5A12017FEA03B8EA073C131CEA0E0E387F1FC038FF3FE0387F1FC013127F9116>120 D<387F1FC038FF9FE0387F1FC0381C0700120E130EA212075BA2EA039CA21398EA01B8A2EA00F0 A35BA3485A1279127BEA7F806CC7FC123C131B7F9116>I123 D<12E0B3AE0320779C16>I<12FCB4FC13C012031201A97F6CB4FCEB7F80A2EBFF00EA01E05BA9 120312FF90C7FC12FC11207E9C16>I E /Fj 23 122 df<3801FFE0A238003C00A45BA45BA448 5AA4485AA4485AA4EAFFF0A2131C7E9B10>73 D<48B4EB03FE1507D8003F14C0150F151BA2016F EB3780EB67801567A201C7EBCF00EC818FA2EC830FD80187131E1486A2148C390307983CA214B0 EB03F00006495AA214C0000E138039FFC38FFF140F271C7E9B25>77 D<3801FFFEECFF8039003C 03C0EC01E0A31378A315C0EBF0031580EC0700141E3801FFFC14F001E0C7FCA2485AA4485AA4EA FFF0A21B1C7E9B1C>80 D97 D99 DIIIII<13C0EA01E0 13C01380C7FCA6120E123FEA33801263EAC700A21207120EA35A1340EA38C0A3EA3980EA3F0012 1E0B1C7D9B0D>I108 D<391E1F07C0393F3F8FE0396761D8703863C1F038C781E0A2380701C0A2390E0380 E0A3EC81C2391C0701C6A2EC038CA239380E01F839180600F01F127D9122>IIII114 DI<13C01201A3EA0380A4EAFFE0A2EA0700A2120EA45AA3 1320EA3860A213C0A2EA1F80EA0F000B1A7D990E>I<380F0180EA1F8312331263EB870012C3EA 0707A2EA0E0EA31420380C1C60121C120C380E3CC0380FEF803803C70013127D9116>II120 D<380F0180EA1F8312331263EB870012C3EA0707A2EA 0E0EA4EA0C1C121C120CEA0E3CEA0FF81203EA0038A2EA3070EA786013E0EA71C0EA3F80001EC7 FC111A7D9114>I E /Fk 42 122 df<903807F83F017FB512C03A01FC0FE3E03903F01FC7EA07 E0D80FC01387ED83C0ED8000A6B612FCA2390FC01F80B2397FF8FFF8A223237FA221>11 D<1238127C12FEA3127C123807077C8610>46 D<13181378EA01F812FFA21201B3A7387FFFE0A2 13207C9F1C>49 DII<14E013011303A2 1307130F131FA21337137713E7EA01C71387EA03071207120E120C12181238127012E0B512FEA2 380007E0A7EBFFFEA217207E9F1C>I<00101320381E01E0381FFFC0148014005B13F8EA1BC000 18C7FCA4EA19FCEA1FFF381E0FC0381807E01303000013F0A214F8A21238127C12FCA214F012F8 386007E0003013C0381C1F80380FFF00EA03F815207D9F1C>I<13FE3803FFC0380703E0380E00 F05A1478123C123E123F1380EBE0F0381FF9E0EBFFC06C13806C13C06C13E04813F0381E7FF838 3C1FFCEA7807EB01FEEAF000143E141EA2141C7E007813387E381F01F0380FFFC0000113001720 7E9F1C>56 D<1470A214F8A3497EA2497EA3EB06FF80010E7FEB0C3FA201187F141F01387FEB30 0FA201607F140701E07F90B5FCA239018001FCA200038090C7FCA20006147FA23AFFE00FFFF8A2 25227EA12A>65 DIIIII72 D77 DIII<3801FC043807FF8C381F03FC383C007C007C133C0078131CA200F8130CA27E1400 B4FC13E06CB4FC14C06C13F06C13F86C13FC000313FEEA003FEB03FFEB007F143FA200C0131FA3 6C131EA26C133C12FCB413F838C7FFE00080138018227DA11F>83 D<007FB61280A2397E03F80F 00781407007014030060140100E015C0A200C01400A400001500B3A20003B512F8A222227EA127 >I97 DI< EBFF80000713E0380F83F0EA1F03123E127E387C01E090C7FC12FCA6127C127EA2003E13306C13 60380FC0E03807FF803800FE0014167E9519>II< 13FE3807FF80380F87C0381E01E0003E13F0EA7C0014F812FCA2B5FCA200FCC7FCA3127CA2127E 003E13186C1330380FC0703803FFC0C6130015167E951A>II<3801FE1F0007B51280 380F87E7EA1F03391E01E000003E7FA5001E5BEA1F03380F87C0EBFF80D819FEC7FC0018C8FC12 1CA2381FFFE014F86C13FE80123F397C003F8048131F140FA3007CEB1F00007E5B381F80FC6CB4 5A000113C019217F951C>II<120E121FEA3F80A3EA1F00120EC7FCA7EAFF80A2121FB2EAFFF0A20C 247FA30F>I<131C133E137FA3133E131C1300A7EA03FFA2EA003FB3A5127812FC133E137EEA78 FCEA7FF0EA1FC0102E83A311>I108 D<3AFF87F00FE090399FFC3FF83A1FB87E70FC9039E03EC07C9039C03F807EA201801300AE3BFF F1FFE3FFC0A22A167E952F>I<38FF87E0EB9FF8381FB8FCEBE07CEBC07EA21380AE39FFF1FFC0 A21A167E951F>I<13FE3807FFC0380F83E0381E00F0003E13F848137CA300FC137EA7007C137C A26C13F8381F01F0380F83E03807FFC03800FE0017167E951C>I<38FF8FE0EBBFF8381FF07CEB C03E497E1580A2EC0FC0A8EC1F80A2EC3F00EBC03EEBE0FCEBBFF8EB8FC00180C7FCA8EAFFF0A2 1A207E951F>I114 DI<13C0A41201A212031207120F12 1FB5FCA2EA0FC0ABEBC180A51207EBE300EA03FEC65A11207F9F16>I<38FF83FEA2381F807EAF 14FEA2EA0F833907FF7FC0EA01FC1A167E951F>I<39FFF01FE0A2390FC00600A2EBE00E000713 0CEBF01C0003131813F800015BA26C6C5AA2EB7EC0A2137F6D5AA26DC7FCA2130EA21B167F951E >I<39FFF01FE0A2390FC00600A2EBE00E0007130CEBF01C0003131813F800015BA26C6C5AA2EB 7EC0A2137F6D5AA26DC7FCA2130EA2130CA25B1278EAFC3813305BEA69C0EA7F80001FC8FC1B20 7F951E>121 D E /Fl 71 124 df11 D<137CEA01FEEA0387485A120E130690C7FCA4 B5FCA2EA0E07AC387F0FE0A2131A809915>I<137F485AEA038FEA070FEA0E07A6B5FCA2EA0E07 AC387F9FE0A2131A809915>I<90383E0F803901FF7FC0390383F0E0380707C1000E1381903803 80C01500A4B612E0A2380E0380AC397F8FE3FCA21E1A809920>I34 D<126012F012F812781218A31230127012 E01240050B7D990B>39 D<13C0EA0180EA03001206120E120C5A1238A212301270A21260A212E0 AA1260A21270A212301238A212187E120E12067EEA0180EA00C00A267E9B0F>I<12C012607E7E 121C120C7E1207A27E1380A21201A213C0AA1380A21203A213005AA212065A121C12185A5A5A0A 267E9B0F>I<126012F0A212701230A31260A212C01240040B7D830B>44 DI<126012F0A2126004047D830B>I<130CA2131C1318A213381330A213 701360A213E013C0A212011380A2120313005A1206A2120E120CA2121C1218A212381230A21270 1260A212E05AA20E257E9B13>II<12035AB4FCA21207B3EAFFF0A20C19 7D9813>III<1260EA7FFEA213FCEA600CEAC0181330 A2EA006013C0A21201A2EA0380A41207A8EA03000F1A7E9813>55 D57 D<126012F0A212601200A8126012F0A2126004107D8F0B >I<126012F0A212601200A8126012F0A212701230A31260A212C0124004177D8F0B>I63 D<130C131EA3133FA3497E1367A3EBC3C0A3380181E0A348B47EA213000006 1378A3487F121E39FF81FFC0A21A1B7F9A1D>65 DIIIIII<39FFF3FFC0A2390F003C00A9EBFFFCA2EB003CAB39 FFF3FFC0A21A1A7F991D>III76 D<39FF8001FF6D5A000F14F0A2380DE006A3380CF00CA3 EB7818A3EB3C30A3EB1E60A3EB0FC0A3EB0780121E39FFC78FFFEBC30F201A7F9923>II<137F3801FFC03807C1F0380F00 78001E7F001C131C487F0078130FA200707F00F01480A80078EB0F00A36C131E001C131C001E13 3C6C5B3807C1F03801FFC06C6CC7FC191C7E9A1E>II<137F3801FFC03807C1F0380F0078 001E7F001C131C003C131E487FA200707F00F01480A80070140000785BA2383C1C1E381C3E1C38 1E633C380F61F83807E1F03901FFC080EA007F130014E1ECFF0080143E141C19227E9A1E>III<007FB5FCA238781E0F00601303A200E0148000C01301A3000090C7FC AF3803FFF0A2191A7F991C>I<39FFF0FFC0A2390F001E00140CB17E6D5A12036D5A3801E0E038 007FC0011FC7FC1A1B7F991D>I<3AFFC7FF1FF0A23A1E00F803C091387801806CEC0300A214FC 5DD807801306EB819EA2D803C15B13C3140F01E3131C000114189038E60798A2D800F613B001FE 13F0EBFC03017C5BA2EB7801A201385BEB3000241B7F9927>87 D<39FFE01FF0A2390F800F0000 0713066C6C5A13E000015BEBF038000013306D5A137CEB3CC0133F6D5AA26DC7FCA9EBFFF0A21C 1A80991D>89 D92 D97 D<12FCA2121CA813F8EA1FFE130F381C0380A2EB01C0A6EB0380A2381F0F 00EA1BFEEA18F8121A7F9915>II<137EA2130EA8EA07CEEA1FFEEA3C1EEA700EA212E0A6127013 1EEA3C3E381FFFC0EA07CF121A7F9915>II<13F8EA01FCEA03BCEA073CEA0E181300A5 EAFFC0A2EA0E00ACEA7FE0A20E1A80990C>I<130EEA079FEA1FF7EA3877EA7038A5EA3870EA3F E0EA7780EA7000A2EA3FF013FC13FEEA700FEAE007A3EA700EEA781EEA1FF8EA07E010197F9013 >I<12FCA2121CA813F8EA1FFC131EEA1E0E121CAA38FF9FC0A2121A7F9915>I<1218123CA21218 C7FCA612FCA2121CACEAFF80A2091A80990A>I<13C0EA01E0A2EA00C01300A6EA07E0A21200B0 126012F0EAF1C0EA7F80EA3E000B2183990C>I<12FCA2121CA8EB7F80A2EB3C0013305B5B121D EA1FE0121EEA1C70137813387F131E38FF3FC0A2121A7F9914>I<12FCA2121CB3A4EAFF80A209 1A80990A>I<38FCFC3F39FDFE7F80391F0FC3C0381E0781001C1301AA39FF9FE7F8A21D107F8F 20>IIIIII I<120CA4121CA2123CEAFFC0A2EA1C00A71360A5EA0FC0EA07800B177F960F>II<38FF3F80A2381C0E00130CA26C5AA21338EA07 30A213F06C5AA26C5AA311107F8F14>I<39FF3F9F80A239381E0E00381C3E0C13361337000E5B 13631498000713B013C114F0A2380380E0A319107F8F1C>I<387F1FC0A2380E1E00EA071CEA03 B813B0EA01E012007F1201EA03B8EA071C1206EA0E0E38FF1FE0A21310808F14>I<38FF3F80A2 381C0E00130CA26C5AA21338EA0730A213F06C5AA26C5AA35BA2126100F3C7FC12C712FE127811 177F8F14>III E /Fm 18 118 df<127812FCA412781200A5127812FCA4127806117D900C>58 D 63 D68 D73 D77 D97 D99 D101 D<3807E3C0381FFFE0EA3C3C38381CC038781E00A4EA381CEA3C3CEA3FF8EA37E00070C7FCA2EA 3FFEEBFF806C13C0127F38F003E01301A3387C07C0383FFF803807FC0013197F9016>103 D<121E123FA4121EC7FCA4127FA2121FADEAFFC0A20A1B809A0C>105 D108 D<39FF1F81F890387FE7FE391FE3EE3E903881F81FA2EB01F0AA3AFFE7 FE7FE0A223117F9026>IIII115 D<1203A35AA25A123FEAFFF0A2EA1F00A81318A5EA0FF0EA03E00D187F9711>I<38FF1FE0A2EA 1F03AB1307130F380FFFFCEA03F316117F9019>I E /Fn 42 122 df<903901FF81FF011F01EF 13C0903A7F80FF87E0D9FE01EB0FF03903FC03FE13F8D807F013FCA2EE07E0020190C7FCA6B712 F8A32707F001FCC7FCB3A33A7FFF1FFFE0A32C2A7FA928>11 D<121C127FEAFF80A5EA7F00121C 09097B8813>46 D48 D<13075B137FEA07FFB5FCA212F8C6FCB3AB007F13FEA317277BA622>III<14075C5C5C5C5CA25B5B497E130F130E131C1338137013F013E0EA01C0EA 0380EA07005A120E5A5A5A5AB612F8A3C71300A7017F13F8A31D277EA622>I<000C1303380F80 3FEBFFFEA25C5C14E05C49C7FC000EC8FCA6EB7FC0380FFFF8EB80FC380E007F000C1480C7123F 15C0A215E0A2123E127FEAFF80A315C01300007E137F007814806CEBFF00381F01FE380FFFF800 0313E0C690C7FC1B277DA622>II<1238123E003FB512F0A315E04814C01580A215003870001E5C5C48137014F0495A C6485A13075C130F91C7FC5B5BA3137EA213FEA41201A86C5A13781C297CA822>III66 D<91393FF00180903903FFFE07010FEBFF8F90393FF007FF9038FF80014848C7127FD8 07FC143F49141F4848140F485A003F15075B007F1503A3484891C7FCAB6C7EEE0380A2123F7F00 1F15076C6C15006C6C5C6D141ED801FE5C6C6C6C13F890393FF007F0010FB512C0010391C7FC90 38003FF829297CA832>II73 D77 DIII82 D<007FB712C0A39039807FC03FD87C0014 0700781503A20070150100F016E0A2481500A5C71500B3A490B612E0A32B287EA730>84 D<48B47E000F13F0381F81FC486C7E147FA2EC3F80A2EA0F00C7FCA2EB0FFF90B5FC3807FC3FEA 1FE0EA3F80127F130012FEA3147F7E6CEBFFC0393F83DFFC380FFF0F3801FC031E1B7E9A21>97 DIIII<9038FF81F00003EBE7FC390FC1FE7C391F80FC FC003FEBFE7C9038007E3848EB7F00A66C137EEB80FE001F5B380FC1F8381FFFE0001813800038 C8FC123CA2123E383FFFF814FF6C14C06C14E06C14F0121F397E0007F8007C13015A1400A36C13 01007EEB03F06CEB07E0390FC01F803903FFFE0038007FF01E287E9A22>103 D<1207EA1FC013E0123FA3121F13C0EA0700C7FCA7EAFFE0A3120FB3A3EAFFFEA30F2B7DAA14> 105 DIII<3BFF C07F800FF0903AC1FFE03FFC903AC783F0F07E3B0FCE03F9C07F903ADC01FB803F01F8D9FF0013 8001F05BA301E05BAF3CFFFE1FFFC3FFF8A3351B7D9A3A>I<38FFC07F9038C1FFC09038C787E0 390FCE07F09038DC03F813F813F0A313E0AF3AFFFE3FFF80A3211B7D9A26>II<38FFE1FE9038E7FF8090 38FE07E0390FF803F8496C7E01E07F140081A2ED7F80A9EDFF00A25DEBF0014A5A01F85B9038FE 0FE09038EFFF80D9E1FCC7FC01E0C8FCA9EAFFFEA321277E9A26>I<38FFC3F0EBCFFCEBDC7E38 0FD8FF13F85BA3EBE03C1400AFB5FCA3181B7E9A1C>114 D<3803FE30380FFFF0EA3E03EA7800 127000F01370A27E6C1300EAFFE013FE387FFFC06C13E06C13F0000713F8C613FC1303130000E0 137C143C7EA26C13787E38FF01F038F7FFC000C11300161B7E9A1B>I<1370A413F0A312011203 A21207381FFFF0B5FCA23807F000AD1438A73803F870000113F03800FFE0EB1F8015267FA51B> I<39FFE03FF8A3000F1303B11407A2140F0007131F3A03F03BFF803801FFF338003FC3211B7D9A 26>I<3AFFFE03FF80A33A07F0007000A26D13F000035CEBFC0100015CA26C6C485AA2D97F07C7 FCA2148FEB3F8E14DEEB1FDCA2EB0FF8A36D5AA26D5AA26D5A211B7F9A24>I<3AFFFE03FF80A3 3A07F0007000A26D13F000035CEBFC0100015CA26C6C485AA2D97F07C7FCA2148FEB3F8E14DEEB 1FDCA2EB0FF8A36D5AA26D5AA26D5AA2495AA2EA3807007C90C8FCEAFE0F130E131E5BEA7C78EA 3FE0EA0FC021277F9A24>121 D E /Fo 74 124 df11 D<133FEBFF803803C1C0EA0703A2380E018090C7FCA5B512C0A2EA0E01AE387F87F8A2151D809C 17>II<9038 3F03F09038FFCFF83903C0FC1C390701F03CA2390E00E01892C7FCA5B612FCA2390E00E01CAE3A 7FC7FCFF80A2211D809C23>I34 D<127012F812FCA2127C120CA31218A2123012601240060D7D9C0C> 39 D<13C0EA0180EA03001206120E120C121C121812381230A21270A21260A212E0AC1260A212 70A21230A212381218121C120C120E12067EEA0180EA00C00A2A7D9E10>I<12C012607E7E121C 120C120E120612077EA21380A21201A213C0AC1380A21203A21300A25A1206120E120C121C1218 5A5A5A0A2A7E9E10>I<127012F012F8A212781218A31230A2127012601240050D7D840C>44 DI<127012F8A3127005057D840C>I<1303A213071306A2130E130C131C 1318A213381330A213701360A213E013C0A21201138012031300A25A1206A2120E120CA2121C12 18A21238123012701260A212E05AA210297E9E15>II<12035A123FB4FC12C71207 B3A3EAFFF8A20D1C7C9B15>III<131C A2133C137CA213DC1201139C1203EA071C1206120E120C121812381230126012E0B512C0A23800 1C00A63801FFC0A2121C7F9B15>I I<13F0EA03FCEA070CEA0E0EEA1C1E1238130CEA78001270A2EAF3F0EAF7F8EAFC1CEAF81E130E 12F0130FA51270A2130E1238131CEA1C38EA0FF0EA03E0101D7E9B15>I<1260387FFF80A21400 EA6003EAC0065BA2C65A5BA25BA25BA21201A2485AA41207A76CC7FC111D7E9B15>III<127012F8A312701200A8127012F8A312700512 7D910C>I<127012F8A312701200A8127012F012F8A212781218A31230A2127012601240051A7D 910C>I<007FB512C0B612E0C9FCA8B612E06C14C01B0C7E8F20>61 D<1306130FA3497EA4EB33 C0A3EB61E0A3EBC0F0A338018078A2EBFFF8487FEB003CA200067FA3001F131F39FFC0FFF0A21C 1D7F9C1F>65 DI<90381F8080EBFFE13803F033380780 1B380F000F001E1307001C1303123C5A1401127012F091C7FCA70070EB01801278A27E001CEB03 00121E6C13066C6C5A3803F0383800FFF0EB1F80191E7E9C1E>I III<90381F8080EBFFE13803F0333807801B 380F000F001E1307001C1303123C5A1401127012F091C7FCA5ECFFF0A20070EB07801278A27E12 1C121E7E3807800F3803F0393800FFF090381FC0001C1E7E9C21>I<39FFF3FFC0A2390F003C00 AAEBFFFCA2EB003CAC39FFF3FFC0A21A1C7E9B1F>II76 DIIII82 D<3807E080EA1FF9EA3C1FEA7007130312E01301A36CC7FCA2127CEA7FC0EA3FF8EA1FFEEA07FF C61380130FEB03C0A2130112C0A300E01380130300F01300EAFC0EEACFFCEA83F8121E7E9C17> I<007FB512C0A238780F03007013010060130000E014E000C01460A400001400B03803FFFCA21B 1C7F9B1E>I<39FFF0FFC0A2390F001E00140CB36C5B138000035BEA01C03800E0E0EB7FC0011F C7FC1A1D7E9B1F>I<3AFFE0FFE1FFA23A1F001E007C6C1530143FA20180147000079038678060 A32603C0E713C0ECC3C0A2D801E0EBC1809038E181E1A3D800F3EBF3001400A2017B13F6017E13 7EA3013C133CA3011C133801181318281D7F9B2B>87 D<397FF0FFC0A23907C03E0000031338EB E03000011370EBF06000005B1379EB7D80013FC7FC7FA27F80131FEB3BC0EB33E01371EB61F0EB C0F800011378EB807C3803003C487F380F801F39FFE0FFF0A21C1C7F9B1F>I<39FFF00FFCA239 078003C001C0138000031400EBE006EA01F000005BEBF81CEB7818EB7C38EB3C306D5A131F6D5A A26D5AAAEB7FF8A21E1C809B1F>I92 D97 D<12FCA2121CA9137EEA1DFF 381F8780381E01C0001C13E0130014F0A614E01301001E13C0381F07803819FF00EA187C141D7F 9C17>IIII<1378EA01FCEA039EEA07 1EEA0E0C1300A6EAFFE0A2EA0E00AEEA7FE0A20F1D809C0D>II<12FCA2121CA9137CEA1DFFEA1F 07381E0380A2121CAB38FF9FF0A2141D7F9C17>I<1218123C127C123C1218C7FCA612FCA2121C AEEAFF80A2091D7F9C0C>II<12FCA2121CA9EB7FC0A2EB3E0013185B5B5BEA1DE0121FEA1E70EA 1C781338133C131C7F130F38FF9FE0A2131D7F9C16>I<12FCA2121CB3A7EAFF80A2091D7F9C0C> I<39FC7E07E039FDFF9FF8391F83B838391E01E01CA2001C13C0AB3AFF8FF8FF80A221127F9124 >IIII<3803E180EA0FF9EA1E1FEA3C071278130312F0 A612781307123CEA1E1FEA0FFBEA07E3EA0003A6EB1FF0A2141A7F9116>III<120CA5121CA212 3CEAFFE0A2EA1C00A81330A5EA1E60EA0FC0EA07800C1A7F9910>I<38FC1F80A2EA1C03AC1307 EA0C0F380FFBF0EA03E314127F9117>I<38FF0FE0A2381C0780EB0300EA0E06A36C5AA2131CEA 0398A213F86C5AA26C5AA313127F9116>I<39FF3FCFE0A2391C0F0780EC0300131F380E1B0614 86A2EB318E000713CCA213603803E0F8A33801C070A31B127F911E>I<387F8FF0A2380F078038 070600EA038EEA01DC13D8EA00F01370137813F8EA01DCEA038E130EEA0607380F038038FF8FF8 A21512809116>I<38FF0FE0A2381C0780EB0300EA0E06A36C5AA2131CEA0398A213F86C5AA26C 5AA35BA3EAF180A200C7C7FC127E123C131A7F9116>III E /Fp 10 118 df67 D80 D<903807FFFE017FEBFFE048B612F84815FE4890 39001FFF8003077F48D980017F6F7FA2707EA2707E6C90C7FC6C5A6C5AC9FCA40207B5FC91B6FC 130F013FEBF03F3901FFFE004813F04813C04890C7FC485A485A485AA212FF5BA3167FA26D14FF 6C6CEB01EF003F140301FF90380FCFFF6C9026C07F8F13F8000790B5000713FC6CECFC03C66CEB F0010107903980007FF8362E7DAD3A>97 D<91381FFFC049B512FC010714FF011F158090267FFC 0113C0EBFFF048D9C00313E04813804813005A485AA248486D13C06F1380007FED7F0093C7FC5B A212FFAA127F7FA3123F6D15F8121F16016C6C15F06C6D13036C6DEB07E06C6DEB0FC06C01F8EB 3F80903A7FFE01FF00011FB55A010714F8010114E0D9001F90C7FC2D2E7CAD35>99 D<13FCEA03FF4813804813C014E05AA67E14C06C13806C1300EA00FC90C7FCABEB7FC0B5FCA512 037EB3B0B6FCA518497CC820>105 D<90287FC001FFC0ECFFE0B5010F01F8010713FC033F01FE 011F13FF92B6017F809126C1FE07902680FF037F9126C3F0039026C1F8017F00039026C7C00190 26E3E0007F6CD9CF00ECE78002DE03EFC7FC02DC6D01FE6E7E14FC4A5DA24A5DA24A5DB3A8B6D8 C07F9026FFE03FB512F0A55C2E7CAD63>109 D<903A7FC001FFC0B5010F13F8033F13FE92B6FC 9126C1FE077F9126C3F0037F00039026C7C0017F6CEBCF0014DE02DC6D7F14FC5CA25CA25CB3A8 B6D8C07FEBFFE0A53B2E7CAD42>II116 DI E /Fq 8 117 df<140F5C147FEB03FF131FB6FCA313E7EA0007B3B3A7007FB612E0A4233879B7 32>49 D67 D97 D<903801FFC0011F13F8017F 13FE9038FFC1FF00039038007F80D807FCEB1FC0484814E0ED0FF0485A003FEC07F8A2485AED03 FCA212FFA290B6FCA301E0C8FCA5127FA27F003F153CA26C6C147C000F15786C6C14F86C6CEB01 F06C6CEB07E06C9038E03FC0013FB51200010F13FC010013E026267DA52D>101 D<13FFB5FCA412077EB0ED7FC0913803FFF8020F13FE91381F03FFEC3C01027814804A7E4A14C0 5CA25CA291C7FCB3A4B5D8FC3F13FFA4303C7CBB37>104 D<9039FF01FF80B5000F13F0023F13 FC9138FE03FFDAF00113C000039039E0007FE0028014F0EE3FF891C7121F17FC160F17FEA3EE07 FFAAEE0FFEA3EE1FFCA217F86EEB3FF06E137F6EEBFFE06E481380DAFC07130091383FFFFC020F 13F0020190C7FC91C9FCADB512FCA430377DA537>112 D<9038FE03F000FFEB0FFE91383FFF80 91387C7FC014F00007ECFFE06C6C5A5CA25CED7FC0ED3F80ED0E0091C8FCB3A3B512FEA423267D A529>114 D116 D E /Fr 36 119 df38 D<127012F8A212F012E005057A840F>46 D<14035CA25C1580141FA214371477146714C7A2EB01 87A2EB0307A21306130E130C1318A2133090383FFFC05BEB600313C012011380EA0300A21206A2 121E39FFC03FFC13801E237DA224>65 D<90B512E015F890380F003C151E131E150E150FA24913 0E151EA2153C49137815F0EC01E090387FFFC090B5FC9038F003E0EC00F01578485A1538153CA2 48481378A315F039078001E01403EC07C0EC1F00B512FE14F020227DA122>I<90B512F015FC90 380F003E150F011EEB0780150316C015015B16E0A35BA449EB03C0A44848EB0780A216005D4848 130E5D153C5D48485B4A5AEC0780021FC7FCB512FC14F023227DA125>68 D<91387E0180903903FF810090380F80C390383E00670178133F49133ED801C0131E485A120748 C7121C120E121E5A15185A92C7FCA25AA4EC3FFC5AEC01E0A26C495AA312700078495A1238003C 130F6C131B260FC0F3C7FC3803FFC1C690C8FC212479A226>71 D73 D<903807FFC04913809038003C00 A25CA45CA4495AA4495AA4495AA449C7FCA212381278EAF81EA2485AEA6078EA70F0EA3FE0EA1F 801A237CA11A>I<9039FFF80FFCA290390F0007C01600011E130E5D5D1560495B4A5A4AC7FC14 0E495A5C147814FCEBF1BCEBF33CEBFE1E13FC3801F01F497EA2813803C007A26E7EA2EA07806E 7EA28139FFF80FFEA226227DA125>III<01FFEB1FFC1480010FEB03C01680D91BC01300A3EB19E001 311306A2EB30F0A201605B1478A3496C5AA3141ED801805BA2140FA2D803005BEC07E0A300065C 1403A2121F39FFE00180A226227DA124>I<14FE903807FF8090380F03E090383C00F001701378 4913384848133C4848131C48C7FC48141E121EA25AA25AA348143CA31578A34814F0A26CEB01E0 15C01403EC07800078EB0F00141E6C5B6C13F8380F83E03807FF80D801FCC7FC1F2479A225>I< 90B512C015F090380F0078153C011E131E150EA349131EA3153C491338157815F0EC03C090B512 005CEBF00FEC07803901E003C0A43903C00780A43907800F001503A2EC0706D8FFF8138EEC03FC C7EA01F020237DA124>82 D<903801F06090380FFC4090381E0EC0EB3807EB700301E0138013C0 1201A2D803801300A26DC7FCA27FEA01F813FF6C13E06D7EEB1FF8EB03FCEB007C143C80A30030 131CA31418007013385C00781360387C01C038EF0380D8C7FFC7FCEA81FC1B247DA21B>I<001F B512F8A2391E03C07800381438EB0780123000601430A2EB0F0012C0A3D8001E1300A45BA45BA4 5BA4485AA31203B57E91C7FC1D2277A123>I<393FFE07FFA23903C000F015E0484813C0A4390F 000180A4001EEB0300A4481306A4485BA4485BA25C12705C5C6C485A49C7FCEA1E0EEA0FFCEA03 F0202377A124>I<3BFFF03FF81FF8D9E07FEB3FF03B1F0007800780001E010FEB03001606141F 5E14375E14675E14C75E381F0187000F5DEB0307ED818013060383C7FC130C158613180138138C 0130139C0160139815B801C013B015E013805D13005D120E92C8FC120C2D2376A131>87 D<39FFF003FFA2000FC712F015E090388001C000071480EC0300EBC0060003130E5CEBE0180001 5B5CEBF0E000005BEBF18001FBC7FC13FF137E137C1378A45BA4485AA4EA3FFE5B202276A124> 89 D97 D<137E48B4FC3803C380EA0703 EA0E07121C003CC7FC12381278A35AA45BEA7003130EEA383CEA1FF0EA0FC011157B9416>99 D<143CEB03F8A2EB0038A21470A414E0A4EB01C013F9EA01FDEA078F380F0780120E121CEA3C03 383807001278A3EAF00EA214101418EB1C30EA703C137C3838FC60383FCFC0380F078016237BA2 19>I<13F8EA03FCEA0F0EEA1E06123C1238EA780CEAF038EAFFF01380EAF0005AA413021306EA 701C1378EA3FE0EA0F800F157A9416>I<143C147F14CF1301EB03861480A3EB0700A5130EEBFF F0A2EB0E00A25BA55BA55BA55BA45B1201A2EA718012F390C7FC127E123C182D82A20F>II<136013F0 13E0A21300A8120EEA1F801233126312C3A3EA0700A2120EA35A13201330EA3860A213C01239EA 1F80EA0E000C217CA00F>105 D<13F0EA0FE0A21200A2485AA4485AA448C7FCEB01E0EB07F0EB 0E30380E1870EB30F01360EBC060381D8000121F13E0EA1CF0EA3838133CEB1C20143038703860 A21440EB18C038E01F8038600F0014237DA216>107 DI<381E0780383F1F E0EA63B8EBE070EAC3C0A21380000713E01300A3380E01C0A214C2EB0383001C1386EB0706140C EB0318003813F0381801E018157C941B>110 D<137E48B4FC3803C380380701C0120E001C13E0 123CA21278A338F003C0A21480130700701300130E5B6C5AEA1FF0EA07C013157B9419>I<3803 C1F03807E3F8380C761C137C3818781E1370A2EA00E0A43801C03CA314780003137014F014E0EB E3C038077F80EB1E0090C7FCA2120EA45AA2EAFFC0A2171F7F9419>I<381E0F80383F1FC03863 B0E013E0EAC3C1A2EB80C00007130090C7FCA3120EA45AA45A121813157C9415>114 D<13FC48B4FC38038380EA0703EA0E07A2EB0200000FC7FC13F0EA07FC6C7EEA007E130FA2EA70 07EAF00EA2485AEA7038EA3FF0EA1FC011157D9414>I<13C01201A4EA0380A4EA0700EAFFF8A2 EA0700120EA45AA45AA213101318EA7030A21360EA71C0EA3F80EA1E000D1F7C9E10>I<000F13 30381F8070EA31C0006113E012C1EAC380A2380381C0EA0701A3380E0380A214841486EB070CA2 130FEB1F183807F3F03803E1E017157C941A>I<380F01C0381F83E0EA31C3EA61C1EAC1C0EAC3 80A2000313C0EA0700A3380E0180A3EB0300A213061304EA0F1CEA07F8EA01E013157C9416>I E /Fs 53 122 df35 D<127012F812FCA2127C12 0CA41218A21230A212601240060F7C840E>44 DI<127012F8A3127005 057C840E>I48 DI51 D<00101380EA1C07381FFF 005B5B13F00018C7FCA613F8EA1BFEEA1F0F381C0780EA180314C0EA000114E0A4126012F0A214 C0EAC0031260148038300700EA1C1EEA0FFCEA03F013227EA018>53 D<137E48B4FC3803C18038 0701C0EA0E03121CEB018048C7FCA2127812701320EAF1FCEAF3FEEAF60738FC038000F813C013 0112F014E0A51270A3003813C0130300181380381C0700EA0E0EEA07FCEA01F013227EA018>I< EA01F0EA07FCEA0E0F38180780EA3803383001C01270A31278EB0380123E383F0700EA1FCEEA0F FCEA03F87FEA0F7F381C3F80EA380F387007C0130338E001E01300A5387001C0A238380380381E 0F00EA0FFEEA03F013227EA018>56 DI<497E497EA3497EA3497E130CA2EB1CF8EB1878 A2EB383C1330A2497EA3497EA348B51280A2EB800739030003C0A30006EB01E0A3000EEB00F000 1F130139FFC00FFFA220237EA225>65 DI<90380FE01090383FF8309038F81C703801E0063903C003 F03807800148C7FC121E003E1470123C127C15301278A212F81500A700781430A2127CA2003C14 60123E121E6C14C06C7E3903C001803901E003003800F80EEB3FF8EB0FE01C247DA223>II 70 D<903807F00890383FFC189038FC0E383801E0033903C001F83807800048C71278121E1538 5AA2007C14181278A212F81500A6EC1FFF1278007CEB0078A2123CA27EA27E6C7E6C6C13F83801 F0013900FC079890383FFE08903807F80020247DA226>I<39FFFC3FFFA239078001E0AD90B5FC A2EB8001AF39FFFC3FFFA220227EA125>I<3803FFF0A238000F00B3A6127012F8A3EAF01EEA60 1CEA3878EA1FF0EA07C014237EA119>74 D<39FFFC07FFA239078001F015C05D4AC7FC14065C5C 14385C5CEB81C0EB8380EB87C080138DEB98F013B0EBE078497E13808080A26E7E8114036E7EA2 6E7E4A7E3AFFFC07FF80A221227EA126>III<39FF800FFF13C00007EB01F89038E000607F12061378A27F133E131E7FA2 EB078014C01303EB01E0A2EB00F01478A2143CA2141E140FA2EC07E0A214031401A2381F8000EA FFF0156020227EA125>III82 D<3803F020380FFC60381C0EE0EA3803EA7001A2EAE000A21460A36C1300A21278127FEA3F F0EA1FFE6C7E0003138038003FC0EB07E01301EB00F0A2147012C0A46C136014E06C13C0EAF801 38EF038038C7FF00EA81FC14247DA21B>I<007FB512F8A2387C07800070143800601418A200E0 141C00C0140CA500001400B3A20003B5FCA21E227EA123>I<3BFFF03FFC07FEA23B0F0007C001 F00203EB00E01760D807806D13C0A33B03C007F001801406A216032701E00C781300A33A00F018 3C06A3903978383E0CEC301EA2161C90393C600F18A390391EC007B0A3010F14E0EC8003A36D48 6C5AA32F237FA132>87 D<387FFFFEA2EB003C007C137C0070137814F84813F0EB01E0EAC00314 C01307148038000F005B131E133E133C5B13F85B0001130313E0EA03C012071380000F13071300 001E1306003E130E003C131E007C133E007813FEB5FCA218227DA11E>90 D97 D<120E12FEA2121E120EAAEB1F80EB7FE0380FC0 F0EB0078000E1338143C141C141EA7141C143C000F1338EB8070EBC1F0380C7FC0EB1F0017237F A21B>II<14E0130FA213011300AAEA03F0EA07FEEA1F07EA3C01EA 38001278127012F0A712701278EA3801EA3C03381E0EF0380FFCFEEA03F017237EA21B>II<133C13FEEA01CFEA038FA2EA0700A9EAFFF8A2EA0700 B1EA7FF8A2102380A20F>I<14F03801F1F83807FFB8380F1F38381E0F00EA1C07003C1380A500 1C1300EA1E0FEA0F1EEA1FFCEA19F00018C7FCA2121CEA1FFF6C13C04813E0383801F038700070 481338A400701370007813F0381E03C0380FFF803801FC0015217F9518>I<120E12FEA2121E12 0EAAEB1F80EB7FC0380FC1E0EB80F0EB0070120EAE38FFE7FFA218237FA21B>I<121C121E123E 121E121CC7FCA8120E12FEA2121E120EAFEAFFC0A20A227FA10E>II<120E12FEA212 1E120EAAEB0FFCA2EB07E0EB0380EB0700130E13185B137813F8EA0F9C131EEA0E0E7F1480EB03 C0130114E014F038FFE3FEA217237FA21A>I<120E12FEA2121E120EB3ABEAFFE0A20B237FA20E> I<390E1FC07F3AFE7FE1FF809039C0F303C03A1F807E01E0390F003C00000E1338AE3AFFE3FF8F FEA227157F942A>I<380E1F8038FE7FC038FFC1E0381F80F0380F0070120EAE38FFE7FFA21815 7F941B>II<380E1F8038FE7FE0 38FFC1F0380F0078120E143CA2141EA7143CA2000F1378EB8070EBC1F0380E7FC0EB1F0090C7FC A8EAFFE0A2171F7F941B>I114 DI<1206A5120EA3121E123EEAFFF8A2 EA0E00AA130CA51308EA0718EA03F0EA01E00E1F7F9E13>I<000E137038FE07F0A2EA1E00000E 1370AC14F01301380703783803FE7FEA01F818157F941B>I<38FFC3FEA2381E00F8000E1360A2 6C13C0A338038180A213C300011300A2EA00E6A3137CA31338A217157F941A>I<39FF8FF9FFA2 391E01C07CD81C031338000EEBE030A2EB06600007EB7060A2130E39038C30C01438139C3901D8 1980141DA2EBF00F00001400A2497EEB600620157F9423>I<38FFC3FEA2381E00F8000E1360A2 6C13C0A338038180A213C300011300A2EA00E6A3137CA31338A21330A213701360A2EAF0C012F1 EAF380007FC7FC123E171F7F941A>121 D E /Ft 20 118 df45 D68 D73 D77 D80 D<90387F80203901FFE0603807C0F8390F001CE0001E130F481307003813030078130112701400 12F0A21560A37E1500127C127E7E13C0EA1FF86CB47E6C13F86C7FC613FF010F1380010013C0EC 1FE01407EC03F01401140015F8A200C01478A57E15706C14F015E07E6CEB01C06CEB038039E780 070038C1F01E38C07FFC38800FF01D337CB125>83 D97 D99 DIII<15F090387F03F83901FFCF1C3803C1FC390780F818 390F00780048137C001E133C003E133EA7001E133C001F137C6C13786C6C5A380FC1E0380DFFC0 D81C7FC7FC0018C8FCA2121CA2121E380FFFF814FF6C14804814C0391E0007E00038EB01F048EB 00701578481438A500701470007814F06CEB01E06CEB03C03907C01F003801FFFC38003FE01E2F 7E9F21>I<1207EA0F80121FA2120FEA0700C7FCABEA078012FFA2120F1207B3A6EA0FC0EAFFF8 A20D307EAF12>105 D<260781FEEB3FC03BFF87FF80FFF0903A8E07C1C0F83B0F9803E3007C27 07B001E6133C9026E000FC7F495BA3495BB3486C486C133F3CFFFC1FFF83FFF0A2341F7E9E38> 109 D<380781FE39FF87FF8090388E07C0390F9803E03807B0019038E000F05BA35BB3486C487E 3AFFFC1FFF80A2211F7E9E25>II<380783E038FF8FF8EB9C7CEA0FB0EA07F0EBE0 38EBC000A35BB3487EEAFFFEA2161F7E9E19>114 D<3801FC10380FFF30381E03F0EA38004813 705A1430A37E6C1300127EEA3FF06CB4FC6C1380000313E038003FF0EB03F8EB007800C0133CA2 141C7EA27E14186C13386C137038EF01E038C3FFC03880FE00161F7E9E1A>I<13C0A51201A312 03A21207120F121FB512E0A23803C000B01430A83801E060A23800F0C0EB7F80EB1F00142C7FAB 19>II E /Fu 5 85 df<1630167016F0A21501A21503A2150715 0FA2151B821531A2156115E115C1EC0181A2EC0301A21406A2140C141C14181430A202607FA2EC C000A249B5FC5B91C7FC1306A25BA25BA25B1370136013E01201000381D80FF01301D8FFFE9038 3FFFF0A22C337CB235>65 D<010FB6FC17C0903A003F8007F0EE01F892C7127C177E4A143E8314 7E188002FE140FA24A15C0A21301A25CA21303171F5CA2130718804A143FA2130F18004A5CA201 1F157E17FE4A5CA2013F4A5A5F91C712034C5A495D160F017E4A5A4CC7FC01FE147E16F849495A ED07E00001EC3F80B600FEC8FC15F032317CB036>68 D<010FB612FEA29039003F8000173E92C7 121EA24A140CA2147EA214FEA25CA20101151CEEC0184A1400A201031301A202F05B1503010713 0F91B5FC93C7FCECE00F010F7FA2ECC006A2011F130EA2EC800C92C8FC133FA291C9FCA25BA213 7EA213FEA25BA21201B512FCA22F317CB02F>70 D<010FB512F816FF903A003F801FC0EE07E092 380003F0EE01F84AEB00FCA2147EA214FE16015CA2010115F816034A14F0EE07E01303EE0F804A EB3F00167E0107EB03F891B512E016809138E007C0010FEB03F015014A6C7EA2011F80A25CA201 3F1301A21400A249495AA2137E170601FE150E170C5B171C000102011338B539F000FC70EE7FE0 C9EA1F802F327CB034>82 D<0007B712F8A23A0FE007F00101801400D80E00491370121E001C13 0F121800385CA20030011F1460127000605CA2023F14E000E016C0C790C8FCA25CA2147EA214FE A25CA21301A25CA21303A25CA21307A25CA2130FA25CA2131FA25C133F497E007FB512C0A22D31 74B033>84 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin %%EndSetup %%Page: 0 1 bop 795 707 a Fu(D)26 b(R)g(A)f(F)h(T)225 798 y Ft(Do)r(cumen)n(t)20 b(for)i(a)f(Standard)g(Message-P)n(assing)f(In)n(terface)685 981 y Fs(Scott)c(Berryman,)e Fr(Y)l(ale)19 b(Univ)700 1039 y Fs(James)c(Co)o(wnie,)h Fr(Meiko)i(Ltd)474 1097 y Fs(Jac)o(k)e(Dongarra,)i Fr(Univ.)23 b(of)17 b(T)l(ennesse)n(e)j(and)d(ORNL)801 1155 y Fs(Al)f(Geist,)f Fr(ORNL)795 1213 y Fs(Bill)f(Gropp,)j Fr(ANL)767 1271 y Fs(Rolf)f(Hemp)q(el,)e Fr(GMD)762 1329 y Fs(Bob)i(Knigh)o(ten,)g Fr(Intel)786 1387 y Fs(Rust)o(y)g(Lusk,)h Fr(ANL)614 1445 y Fs(Stev)o(e)e(Otto,)h Fr(Or)n(e)n(gon)h(Gr)n(aduate)g(Inst)578 1504 y Fs(T)l(on)o(y)g(Skjellum,)c Fr(Missisippi)j(State)j(Univ)653 1562 y Fs(Marc)d(Snir,)g Fr(IBM)h(T.)g(J.)g(Watson)744 1620 y Fs(Da)o(vid)f(W)l(alk)o(er,)f Fr(ORNL)626 1678 y Fs(Stev)o(e)g(Zenith,)g Fr(Kuck)k(&)f(Asso)n(ciates)844 1796 y Fs(Ma)o(y)e(9,)g(1993)87 1854 y(This)g(w)o(ork)g(w)o(as)h(supp)q(orted)g(b)o(y)f(ARP)l(A)g(and)g(NSF)g (under)g(con)o(tract)g(n)o(um)o(b)q(er)f(###,)g(b)o(y)g(the)192 1912 y(National)h(Science)f(F)l(oundation)i(Science)e(and)i(T)l(ec)o(hnology) f(Cen)o(ter)f(Co)q(op)q(erativ)o(e)650 1970 y(Agreemen)o(t)f(No.)21 b(CCR-8809615.)p eop %%Page: 1 2 bop 75 359 a Fq(Chapter)32 b(1)75 570 y Fp(P)m(oin)m(t)41 b(to)f(P)m(oin)m(t) h(Comm)m(unication)884 789 y Fo(Marc)15 b(Snir)684 838 y(William)10 b(Gropp)j(and)h(Ewing)f(Lusk)75 991 y Fn(1.1)70 b(In)n(tro)r(duction)75 1088 y Fo(This)17 b(section)h(is)e(a)h(draft)g(of)f(the)i(curren)o(t)g(prop)q (osal)f(for)g(p)q(oin)o(t-to-p)q(oin)o(t)e(comm)o(unication.)24 b(It)17 b(do)q(es)h(not)f(y)o(et)75 1137 y(include)d(a)g(description)g(of)f (the)i(F)m(ortran)f(77)f(and)h(C)f(bindings.)158 1190 y(I)h(ha)o(v)o(e)g (tried)g(to)g(indicate,)f(wherev)o(er)i(appropriate,)f(gaps)g(and)f(unresolv) o(ed)i(issues,)g(using)e(small)f(t)o(yp)q(e.)158 1321 y Fm(Discussion:)38 b Fl(The)13 b(follo)o(wing)j(sections)f(of)f(the)g(in)o(tro)q(duction)i(con)o (tain)f(general)g(information)h(on)e(the)g(design)h(of)75 1367 y(MPI)h(pro)q(cedures.)27 b(The)16 b(material)i(should)g(b)q(e)e(mo)o(v)o(ed) g(to)g(a)g(general)h(in)o(tro)q(duction)i(for)d(the)g(en)o(tire)h(do)q(cumen) o(t.)27 b(The)75 1413 y(actual)14 b(binding)i(of)d(these)g(ob)r(jects)g(in)h (F)m(ortran)g(and)f(C)g(will)h(b)q(e)g(discussed)h(in)e(the)h(language)h (binding)g(sub)q(committee.)75 1648 y Fn(1.2)70 b(Data)23 b(T)n(yp)r(es)75 1753 y Fk(1.2.1)55 b(Handle)75 1835 y Fo(MPI)17 b(pro)q(cedures)j(use)e(at)f (v)n(arious)g(places)h Fj(hand)r(les)p Fo(.)28 b(Handles)18 b(are)g(used)g(to)f(access)i(opaque)e(ob)r(jects.)30 b(Suc)o(h)75 1885 y(ob)r(ject)20 b(can)g(b)q(e)g(created,)i(up)q(dated)e(and)f(destro)o(y) o(ed)i(only)d(b)o(y)h(b)o(y)h(calling)e(suitable)h(MPI)h(pro)q(cedures,)i (and)75 1934 y(pro)o(viding)14 b(the)i(handle)f(as)g(parameter.)21 b(Opaque)16 b(ob)r(jects)g(hide)g(from)d(the)j(user)g(the)g(in)o(ternal)e (represen)o(tation)75 1984 y(used)20 b(for)f(v)n(arious)f(MPI)i(ob)r(jects,)h (th)o(us)e(allo)o(wing)e(to)i(ha)o(v)o(e)g(similar)e(calls)i(in)f(C)h(and)g (F)m(ortran,)h(allo)o(wing)d(to)75 2034 y(o)o(v)o(ercome)f(problems)g(with)h (the)g(t)o(yping)f(rules)i(in)e(these)j(languages,)d(and)h(allo)o(wing)e(for) h(future)i(extension)g(of)75 2084 y(their)13 b(functionalit)o(y)m(.)j(The)e (mec)o(hanism)c(for)j(opaque)g(ob)r(jects)h(used)g(here)g(lo)q(osely)e(follo) o(ws)g(the)h(POSIX)h(F)m(ortran)75 2134 y(binding)f(standard.)158 2186 y(An)i(opaque)h(ob)r(ject)g(can)f(b)q(e)h Fj(p)n(ersistent)f Fo(or)g Fj(ephemer)n(al)p Fo(.)22 b(A)15 b(p)q(ersisten)o(t)i(ob)r(ject)f(p)q (ersists)h(un)o(til)e(destro)o(y)o(ed)75 2236 y(b)o(y)h(an)g(explicit)g(op)q (eration.)25 b(An)17 b(ephemeral)e(ob)r(ject)j(is)e(go)q(o)q(d)g(for)g(a)g (single)g(use;)h(th)o(us)g(an)f(ephemeral)g(ob)r(ject)75 2286 y(asso)q(ciated)i(with)g(a)f(comm)o(unicatio)o(n)e(op)q(eration)i(disapp)q (ears)i(once)f(this)g(op)q(eration)f(is)h(completed)f(\(or)g(once)75 2336 y(this)d(ob)r(ject)h(is)e(not)h(needed)i(an)o(ymore)c(for)i(the)g (completion)e(of)h(the)i(op)q(eration\).)158 2389 y(Opaque)i(ob)r(ject)g(are) g(created)h(b)o(y)e(calls)g(that)g(are)h(sp)q(eci\014c)g(to)g(eac)o(h)f(ob)r (ject)i(t)o(yp)q(e.)25 b(They)17 b(are)g(destro)o(y)o(ed)75 2438 y(b)o(y)d(a)f(call)h(to)f Fi(MPI)p 363 2438 14 2 v 15 w(FREE)p Fo(.)g(Additional)f(MPI)j(functions)f(are)g(a)o(v)n(ailable)e(to)h (access)j(and)e(up)q(date)h(sp)q(eci\014c)g(opaque)75 2488 y(ob)r(jects.)158 2576 y Fh(MPI)p 257 2576 15 2 v 17 w(FREE\(handle\))75 2704 y(IN)h(handle)j Fo(handle)14 b(to)g(ob)r(ject)965 2828 y(1)p eop %%Page: 2 3 bop 75 -100 a Fo(2)733 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g (COMMUNICA)m(TION)158 45 y Fo(Mark)e(the)h(handle)g(for)f(destruction.)18 b(The)13 b(ob)r(ject)g(can)g(destro)o(y)o(ed)h(as)e(so)q(on)h(as)f(there)i (there)f(is)g(no)f(p)q(ending)75 95 y(op)q(eration)g(that)g(is)g(using)g (this)g(ob)r(ject.)18 b(This)12 b(is)g(equiv)n(alen)o(t)g(to)g(c)o(hanging)f (the)i(p)q(ersistence)i(of)c(the)i(handle)f(from)75 145 y Fi(persistent)c Fo(to)i Fi(ephemeral)p Fo(.)15 b(F)m(or)9 b(example,)g(if)g(a)h(comm)o (unication)d(ob)r(ject)j(is)g(freed,)i(after)e(the)g(comm)o(unication)75 195 y(started)15 b(but)f(b)q(efore)h(it)f(completed,)f(then)h(the)h(p)q (ending)f(op)q(eration)g(is)f(not)h(a\013ected;)h(one)g(can)f(use)h(the)f (handle)75 244 y(in)f(calls)g(to)g Fi(MPI)p 334 244 14 2 v 15 w(WAIT)f Fo(or)h Fi(MPI)p 565 244 V 15 w(STATUS)p Fo(.)f(Ho)o(w)o(ev)o (er,)h(once)h(the)g(op)q(eration)f(completes,)g(it)g(is)g(erroneous)h(to)f (reuse)75 294 y(the)h(freed)h(handle)f(for)g(a)f(new)h(comm)o(unication.)158 426 y Fm(Discussion:)158 475 y Fl(An)f(alternativ)o(e)i(design)f(p)q(oin)o(t) g(w)o(as)f(discussed,)h(whereb)o(y)g(it)f(is)h(the)e(user)i(resp)q(onsibili)q (t)o(y)i(to)d(free)f(a)h(handle)i(only)f(if)75 520 y(there)h(are)g(no)g(p)q (ending)i(op)q(erations)g(on)e(this)h(handle.)24 b(The)15 b(curren)o(t)g (design)h(w)o(as)f(deemed)g(more)g(elegan)o(t.)24 b(Note)15 b(that)75 566 y(it)f(is)g(not)f(necessary)i(to)e(destro)o(y)h(the)g(handle)h (as)e(so)q(on)h(as)g(it)f(is)h(not)g(needed)g(an)o(ymore.)19 b(A)13 b(v)n(alid)i(implemen)o(tation)i(ma)o(y)75 612 y(just)c(mark)g(freed)g (handles)i(and)f(garbage)g(collect)g(them)f(p)q(erio)q(dical)q(ly)n(.)158 660 y(On)k(the)f(other)h(hand,)h(some)f(ob)r(jects,)h(suc)o(h)f(as)g (bu\013er)g(descriptor)i(ob)r(jects,)e(can)g(b)q(e)g(used)g(b)o(y)g(sev)o (eral)h(distinct)75 706 y(concurren)o(t)c(comm)o(unication)i(op)q(erations.)j (Th)o(us,)13 b(to)g(\014nd)h(out)f(whether)h(suc)o(h)g(ob)r(ject)f(can)g(b)q (e)h(freed,)f(one)g(w)o(ould)h(need)75 752 y(a)f(reference)h(coun)o(t)g(of)f (p)q(ending)j(op)q(erations.)j(This)c(complication)h(w)o(as)d(not)h(observ)o (ed)g(at)g(our)f(last)h(meeting,)h(and)f(ma)o(y)75 797 y(b)q(e)f(a)g(go)q(o)q (d)h(argumen)o(t)g(to)f(reconsider)i(the)e(curren)o(t)g(de\014nition)j(of)d Ff(MPI)p 1130 797 12 2 v 13 w(FREE)p Fl(.)158 846 y(W)m(e)h(do)f(not)h(ha)o (v)o(e)g(y)o(et)g(a)f(mec)o(hanism)i(to)f(con)o(trol)g(where)g(from)f (storage)h(is)g(allo)q(cated)i(for)d(opaque)i(ob)r(jects.)j(Some)75 892 y(b)q(eliev)o(e)d(it)e(is)h(desirable)h(to)e(allo)o(w)h(the)g(user)f(to)g (con)o(trol)h(this)g(allo)q(cation.)158 941 y(I)e(no)o(w)h(ha)o(v)o(e)g(a)g (generic)h Ff(MPI)p 576 941 V 13 w(FREE)d Fl(function)j(for)e(all)i(kinds)g (of)f(opaque)g(ob)r(jects.)18 b(An)o(y)12 b(reason)i(to)e(ha)o(v)o(e)i(sp)q (ecialized)75 987 y(functions)h(for)d(eac)o(h)i(kind)g(of)f(ob)r(ject,)g(as)g (for)g Ff(CREATE)p Fl(?)158 1158 y Fh(MPI)p 257 1158 15 2 v 17 w(ASSOCIA)l(TED\(handle,)h(t)o(yp)q(e\))75 1286 y(IN)i(handle)j Fo(handle)14 b(to)g(ob)r(ject)75 1382 y Fh(OUT)i(t)o(yp)q(e)k Fo(state)158 1474 y(Returns)c(the)f(t)o(yp)q(e)g(of)f(the)i(ob)r(ject)f(the)h (handle)e(is)h(curren)o(tly)h(asso)q(ciated)f(with,)f(if)g(suc)o(h)i(exists.) 21 b(Returns)75 1524 y(the)14 b(sp)q(ecial)h(t)o(yp)q(e)f Fi(MPI)p 444 1524 14 2 v 15 w(NULL)f Fo(if)g(the)i(handle)e(is)h(not)g(curren)o(tly)h (asso)q(ciated)f(with)g(an)o(y)f(ob)r(ject.)158 1577 y(MPI)18 b(ma)o(y)e(pro)o(vide)h(prede\014ned)j(opaque)d(ob)r(jects)i(and)f (prede\014ned,)i(static)e(handles)g(to)f(these)i(ob)r(jects.)75 1627 y(Suc)o(h)14 b(ob)r(jects)h(ma)o(y)d(not)i(b)q(e)h(destro)o(y)o(ed.)75 1760 y Fk(1.2.2)55 b(Arra)n(y)19 b(of)g(handles)75 1843 y Fo(An)14 b(MPI)h(call)f(ma)o(y)e(need)j(a)f(parameter)g(that)g(is)h(an)f Fj(arr)n(ay)g(of)i(hand)r(les)p Fo(.)k(The)14 b(arra)o(y-of-handles)g(ob)r (ject)h(is)f(not)75 1893 y(self-delimiting;)9 b(it)i(do)q(es)i(not)f(carry)g (information)d(on)i(the)i(n)o(um)o(b)q(er)e(of)g(en)o(tries)i(in)e(the)i (arra)o(y)m(.)j(This)c(information)75 1942 y(has)i(to)g(b)q(e)g(pro)o(vided)g (b)o(y)g(an)f(additional)f Fi(len)i Fo(parameter,)f(whenev)o(er)i(a)f (list-of-handles)e(parameter)i(is)g(used.)158 2074 y Fm(Discussion:)158 2123 y Fl(The)d(reason)g(for)g(not)g(ha)o(ving)h(self-delimiti)q(ng)i(arra)o (ys)d(is)h(that)f(records)g(are)g(not)g(\014rst-class)h(citizens)g(in)g(F)m (ortran)f(and)75 2168 y(that)h(ha)o(ving)i(an)f(arra)o(y)f(where)h(one)f(en)o (try)h(enco)q(des)g(length,)g(whereas)g(the)f(other)h(en)o(tries)g(enco)q(de) g(handles)h(w)o(as)e(deemed)75 2214 y(to)k(b)q(e)g(to)q(o)g(error-prone.)27 b(This)17 b(line)g(of)f(argumen)o(tation)i(really)g(implies)g(that)e(w)o(e)g (do)g(not)g(view)h(arra)o(y-of-handles)h(as)75 2260 y(an)f(opaque)g(ob)r (ject,)h(and)f(that)f(w)o(e)g(exp)q(ect)h(the)g(user)g(to)f(build)j(these)e (ob)r(jects)f(explicitl)q(y)m(,)k(not)c(via)i(MPI)e(calls.)29 b(The)75 2305 y(alternativ)o(e)15 b(is)f(for)e(arra)o(y-of-handles)k(to)c(b)q (e)i(an)f(opaque)h(ob)r(ject,)f(manipulated)j(via)e(s)f(set)g(of)g(MPI)g (calls.)75 2521 y Fk(1.2.3)55 b(State)75 2604 y Fo(MPI)14 b(pro)q(cedures)i (use)e(at)f(v)n(arious)g(places)h(argumen)o(ts)f(with)g Fj(state)g Fo(t)o(yp)q(es.)19 b(The)14 b(v)n(alues)f(of)g(suc)o(h)h(data)g(t)o(yp)q(e)g (are)75 2654 y(all)f(iden)o(ti\014ed)h(b)o(y)g(names,)e(and)i(no)g(op)q (eration)g(is)f(de\014ned)i(on)f(them.)k(F)m(or)13 b(example,)f(the)j Fi(MPI)p 1589 2654 V 15 w(APPEND)e Fo(routine)75 2704 y(has)h(a)g(state)g(t)o (yp)q(e)h(parameter)e(with)h(v)n(alues)f Fi(MPI)p 864 2704 V 16 w(INT,)20 b(MPI)p 1054 2704 V 16 w(REAL,)12 b Fo(etc.)p eop %%Page: 3 4 bop 75 -100 a Fg(1.3.)31 b(PR)o(OCESSES)1437 b Fo(3)75 45 y Fk(1.2.4)55 b(Named)18 b(constan)n(ts)75 124 y Fo(MPI)e(pro)q(cedures)j (sometimes)14 b(assign)i(a)g(sp)q(ecial)g(meaning)e(to)i(a)g(sp)q(ecial)g(v)n (alue)g(of)f(a)h(basic)g(t)o(yp)q(e)h(parameter;)75 173 y(e.g.)i Fi(tag)14 b Fo(is)g(an)g(in)o(teger)g(v)n(alued)g(parameter)g(of)g(p)q(oin)o (t-to-p)q(oin)o(t)f(comm)o(unicatio)o(n)f(op)q(erations,)i(with)g(a)g(sp)q (ecial)75 223 y Fi(DONTCARE)g Fo(v)n(alue.)24 b(Suc)o(h)17 b(parameters)f(will)f(ha)o(v)o(e)h(a)g(range)g(of)f(regular)h(v)n(alues,)g (whic)o(h)g(is)g(a)g(prop)q(er)h(subrange)75 273 y(of)f(the)h(range)g(of)f(v) n(alues)h(of)f(the)h(corresp)q(onding)h(basic)f(t)o(yp)q(e;)h(sp)q(ecial)f(v) n(alues)f(\(suc)o(h)i(as)f(DONTCARE\))f(will)75 323 y(b)q(e)f(outside)f(the)h (regular)f(range.)19 b(The)c(range)f(of)g(regular)g(v)n(alues)g(can)g(b)q(e)h (queried,)f(and)g(sometimes)f(set,)h(using)75 373 y(en)o(vironmen)o(t)f (inquiry)g(or)h(en)o(vironmen)o(t)f(setting)h(functions)g(\(Section)g Fh(??)p Fo(\).)158 502 y Fm(Discussion:)158 549 y Fl(Need)f(to)g(agree)g(on)h (a)f(language)i(mec)o(hanism)f(for)f(named)h(constan)o(ts.)158 595 y(Implemen)o(ters)j(should)g(detect,)f(whenev)o(er)g(p)q(ossible,)i (illegal)g(uses)e(of)f(\\sp)q(ecial)j(v)n(alues".)25 b(Th)o(us,)16 b(the)g(use)g(of)f(the)75 641 y Ff(DONTCARE)10 b Fl(v)n(alue)k(to)f(tag)g(a)g (message)h(sen)o(t)f(will)h(b)q(e)g(\015agged)g(as)f(an)g(error.)75 845 y Fk(1.2.5)55 b(Choice)75 924 y Fo(MPI)15 b(functions)h(sometimes)d(use)j (parameters)f(with)g(a)f Fj(choic)n(e)i Fo(\(or)f(union\))g(data)f(t)o(yp)q (e.)23 b(I.e.,)14 b(distinct)h(calls)g(to)75 973 y(the)i(same)f(routine)h(ma) o(y)e(pass)i(b)o(y)f(reference)j(actual)d(parameters)h(of)f(di\013eren)o(t)i (t)o(yp)q(es.)27 b(The)17 b(mec)o(hanism)d(for)75 1023 y(pro)o(viding)f(suc)o (h)h(parameters)g(will)f(di\013er)h(from)e(language)h(to)h(language.)158 1153 y Fm(Discussion:)158 1199 y Fl(The)i(F)m(ortran)g(77)g(standard)h(sp)q (eci\014es)h(that)e(the)g(t)o(yp)q(e)g(of)g(actual)h(argumen)o(ts)g(need)f (to)g(agree)g(with)h(the)f(t)o(yp)q(e)g(of)75 1245 y(dumm)o(y)g(argumen)o (ts;)i(no)f(construct)f(equiv)n(alen)o(t)i(to)e(C)g(p)q(oin)o(ters)h(is)f(a)o (v)n(ailable.)29 b(Th)o(us,)16 b(it)g(w)o(ould)h(seem)f(that)g(there)g(is)75 1291 y(no)f(standard)h(conforming)g(mec)o(hanism)g(to)e(supp)q(ort)i(c)o (hoice)g(parameters.)22 b(Ho)o(w)o(ev)o(er,)14 b(most)h(F)m(ortran)g (compiler)h(either)75 1336 y(don't)g(c)o(hec)o(k)g(t)o(yp)q(e)g(consistency)i (of)e(calls)h(to)f(external)h(routines,)g(or)f(supp)q(ort)h(a)f(sp)q(ecial)h (mec)o(hanism)h(to)d(link)j(foreign)75 1382 y(\(e.g.,)d(C\))f(routines.)25 b(I)15 b(suggest)h(that)f(w)o(e)g(accept)g(this)h(nonconformit)o(y)h(with)f (F)m(ortran)f(77)h(standard.)24 b(I.e.,)15 b(w)o(e)f(accept)75 1428 y(that)f(the)g(same)h(routine)g(ma)o(y)f(b)q(e)g(passed)h(an)g(actual)g (parameter)g(of)e(a)h(di\013eren)o(t)i(t)o(yp)q(e)e(at)g(distinct)i(calls.) 158 1474 y(Generic)h(routines)g(can)f(b)q(e)g(used)g(in)g(F)m(ortran)g(90)g (to)g(pro)o(vide)h(a)e(standard)i(conforming)g(solution.)24 b(This)16 b(solution)75 1520 y(will)f(b)q(e)e(consisten)o(t)h(with)g(our)f (nonstandard)i(conforming)g(F)m(ortran)e(77)g(solution.)75 1745 y Fn(1.3)70 b(Pro)r(cesses)75 1921 y Fm(Discussion:)18 b Fl(This)c(material)g(b)q(elongs)h(to)e(an)g(in)o(tro)q(duction)j(or)d(en)o (vironmen)o(t)i(section)158 2054 y Fo(An)k(MPI)g(program)e(is)i(executed)h(b) o(y)f(sev)o(eral)g(autonomous)e(pro)q(cesses)22 b(that)c(eac)o(h)i(execute)g (their)f(o)o(wn)75 2104 y(co)q(de,)d(in)f(a)h(MIMD)f(st)o(yle.)23 b(The)16 b(co)q(des)h(executed)g(b)o(y)f(eac)o(h)g(pro)q(cess)h(need)f(not)g (b)q(e)g(iden)o(tical.)22 b(The)16 b(pro)q(cesses)75 2154 y(comm)o(unicate)d (via)h(calls)h(to)g(MPI)g(comm)o(unication)d(primitiv)o(es.)19 b(T)o(ypically)m(,)13 b(eac)o(h)j(pro)q(cessor)h(executes)g(in)e(its)75 2203 y(o)o(wn)f(address)i(space,)f(although)f(shared-memory)f(implem)o(en)o (tations)f(of)i(MPI)h(are)g(p)q(ossible.)20 b(This)14 b(do)q(cumen)o(t)75 2253 y(sp)q(eci\014es)i(the)f(b)q(eha)o(vior)f(of)g(a)g(parallel)g(program)e (assuming)h(that)i(only)e(MPI)i(calls)f(are)h(used)g(for)f(comm)o(unica-)75 2303 y(tion.)j(The)d(in)o(teraction)f(of)g(an)g(MPI)h(program)e(with)h(other) h(p)q(ossible)f(means)g(of)g(comm)o(unicatio)o(n)e(\(e.g.,)h(shared)75 2353 y(memory\))e(is)i(not)h(sp)q(eci\014ed.)19 b(In)13 b(particular,)f(it)g (is)g(assumed)g(that)h(message)f(bu\013ers)i(at)e(distinct)h(pro)q(cessors)i (are)75 2403 y(disjoin)o(t.)158 2454 y(MPI)e(do)q(es)g(not)f(sp)q(ecify)h (the)g(execution)g(mo)q(del)e(for)h(eac)o(h)h(pro)q(cess.)20 b(A)12 b(pro)q(cess)i(can)f(b)q(e)g(sequen)o(tial,)f(or)h(can)75 2503 y(b)q(e)h(m)o(ultithreaded,)f(with)g(threads)i(p)q(ossibly)e(executing)i (concurren)o(tly)m(.)j(Care)c(has)g(b)q(een)h(tak)o(en)f(to)f(mak)o(e)g(MPI) 75 2553 y(\\thread-safe",)h(b)o(y)f(a)o(v)o(oiding)f(the)j(use)f(of)g (implicit)d(global)h(states.)158 2604 y(The)h(initial)e(allo)q(cation)f(of)i (pro)q(cesses)j(to)e(an)f(MPI)h(computation)e(and)h(their)h(binding)e(to)i (ph)o(ysical)f(pro)q(ces-)75 2654 y(sors)k(is)f(not)h(sp)q(eci\014ed)g(b)o(y) g(the)f(program)f(itself.)22 b(It)16 b(is)f(exp)q(ected)i(that)f(v)o(endors)g (will)e(pro)o(vide)h(mec)o(hanisms)e(to)75 2704 y(do)h(so)g(either)h(at)e (load)g(time)g(or)h(at)g(run)g(time.)j(Suc)o(h)d(mec)o(hanisms)e(will)h(allo) o(w)f(the)i(sp)q(eci\014cation)h(of)e(the)i(initial)p eop %%Page: 4 5 bop 75 -100 a Fo(4)733 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g (COMMUNICA)m(TION)75 45 y Fo(n)o(um)o(b)q(er)i(of)g(required)i(pro)q(cesses,) i(the)d(co)q(de)h(to)f(b)q(e)g(executed)i(b)o(y)d(eac)o(h)i(initial)d(pro)q (cess,)j(and)f(the)h(allo)q(cation)75 95 y(of)g(pro)q(cesses)j(to)e(pro)q (cessors.)34 b(Also,)20 b(the)f(curren)o(t)h(prop)q(osal)e(do)q(es)i(not)e (pro)o(vide)h(for)f(dynamic)f(creation)i(or)75 145 y(deletion)14 b(of)f(pro)q(cesses)j(during)e(program)e(execution)i(\(the)h(total)e(n)o(um)o (b)q(er)g(of)g(pro)q(cesses)j(is)e(\014xed\),)g(although)e(it)75 195 y(is)k(in)o(tended)h(to)f(b)q(e)h(consisten)o(t)h(with)e(suc)o(h)h (extension.)26 b(Finally)m(,)14 b(the)j(curren)o(t)h(prop)q(osal)e(do)q(es)h (not)f(sp)q(ecify)h(a)75 244 y(naming)d(sc)o(heme)j(for)f(pro)q(cesses.)29 b(W)m(e)16 b(prop)q(ose)i(to)e(alw)o(a)o(ys)g(iden)o(tify)g(pro)q(cesses)j (according)e(to)f(their)h(relativ)o(e)75 294 y(rank)d(in)f(a)h(con)o(text)g (\(group\),)g(so)g(that,)f(e\013ectiv)o(ely)m(,)h(pro)q(cesses)j(are)d(iden)o (ti\014ed)g(b)o(y)g(consecutiv)o(e)h(in)o(tegers.)75 443 y Fk(1.3.1)55 b(Error)18 b(Handling)75 614 y Fm(Discussion:)34 b Fl(Material)14 b(of)e(this)h(section)g(should)h(b)q(e)e(mo)o(v)o(ed)h(to)f (a)g(general)h(in)o(tro)q(duction)i(section.)j(This)13 b(material)g(has)75 664 y(not)g(y)o(et)g(b)q(een)h(appro)o(v)o(ed)g(b)o(y)g(MPIF)158 802 y Fo(MPI)i(pro)o(vides)h(the)f(user)h(with)f(reliable)g(message)g (transmission:)21 b(A)16 b(message)g(sen)o(t)h(is)f(alw)o(a)o(ys)f(receiv)o (ed)75 852 y(correctly)m(,)20 b(and)f(the)g(user)h(do)q(es)f(not)g(need)g(to) g(c)o(hec)o(k)g(for)g(transmission)e(errors,)k(time-outs,)d(or)h(other)g (error)75 902 y(conditions.)f(In)c(other)g(w)o(ords,)g(MPI)g(do)q(es)h(not)f (pro)o(vide)g(mec)o(hanisms)e(for)h(dealing)g(with)h(failures)f(in)h(the)g (com-)75 952 y(m)o(unication)9 b(system.)17 b(Where)12 b(the)g(MPI)f (implemen)o(tation)d(is)j(built)f(on)h(an)g(unreliable)g(underlying)g(mec)o (hanism,)75 1002 y(then)16 b(it)f(is)g(the)h(job)f(of)g(the)h(implem)o(en)o (ter)e(of)h(the)h(MPI)f(subsystem)h(to)f(insulate)g(the)h(user)h(from)c(this) j(unrelia-)75 1051 y(bilit)o(y)m(,)11 b(or)j(to)f(re\015ect)j(unreco)o(v)o (erable)f(errors)g(as)e(global)f(program)g(failures.)18 b(Similarly)10 b(MPI)k(itself)f(pro)o(vides)h(no)75 1101 y(mec)o(hanisms)e(for)h(handling)g (no)q(de)h(failures.)158 1157 y(Of)c(course,)i(MPI)e(programs)f(ma)o(y)f (still)i(b)q(e)h(erroneous.)18 b(A)10 b Fh(program)g(error)f Fo(can)h(o)q(ccur)i(when)e(an)g(MPI)h(call)75 1207 y(is)i(called)g(with)g(an) g(incorrect)h(parameter)f(\(non-existing)g(destination)g(in)f(a)h(send)h(op)q (eration,)f(bu\013er)h(to)q(o)f(small)75 1257 y(in)k(a)f(receiv)o(e)j(op)q (eration,)e(etc.\))29 b(This)17 b(t)o(yp)q(e)g(of)g(error)h(w)o(ould)e(o)q (ccur)i(in)f(an)o(y)f(implemen)o(tation.)24 b(In)18 b(addition,)75 1307 y(a)f Fh(resource)i(error)e Fo(ma)o(y)f(o)q(ccur)i(when)h(a)e(program)f (exceeds)k(the)e(amoun)o(t)e(of)i(a)o(v)n(ailable)d(system)j(resources)75 1357 y(\(n)o(um)o(b)q(er)d(of)h(p)q(ending)g(messages,)g(system)g(bu\013ers,) h(etc.\).)26 b(The)16 b(o)q(ccurrence)j(of)c(this)h(t)o(yp)q(e)h(of)e(error)i (dep)q(ends)75 1406 y(on)h(the)g(amoun)o(t)e(of)h(a)o(v)n(ailable)f (resources)k(in)e(the)g(system)g(and)f(the)i(resource)h(allo)q(cation)c(mec)o (hanism)g(used;)75 1456 y(this)g(ma)o(y)d(di\013er)j(from)e(system)h(to)h (system.)22 b(The)16 b(recommended)f(implemen)o(tation)d(pro\014le)k(pro)o (vides)f(sev)o(eral)75 1506 y(mec)o(hanisms)d(to)h(alleviate)g(the)h(p)q (ortabilit)o(y)e(problem)g(this)i(represen)o(ts.)21 b(One)14 b(can)g(also)f(write)h Fh(safe)f Fo(programs,)75 1556 y(that)h(are)g(not)g (sub)r(ject)h(to)f(resource)i(errors.)158 1612 y(All)e(MPI)h(pro)q(cedure)i (calls)e(return)h(an)f(error)h(parameter)f(that)g(indicates)g(successful)i (completion)c(of)h(the)75 1662 y(op)q(eration,)f(or)h(the)h(error)f (condition)g(that)g(o)q(ccurred,)h(otherwise.)158 1718 y(The)e(recommended)f (implem)o(en)o(tation)e(pro\014le)i(in)g(a)g(POSIX)i(en)o(vironmen)o(t)d(is)h (for)g(an)o(y)g(MPI)h(routine)g(that)75 1767 y(encoun)o(ters)18 b(a)f(reco)o(v)o(erable)g(error)g(to)g(generate)g(an)g Fj(MPI)g(err)n(or)f (signal)p Fo(,)h(using)f(a)g(sp)q(ecial)h(signal)e(v)n(alue.)26 b(The)75 1817 y(default)10 b(handler)h(for)g(this)g(signal)f(terminates)g (the)i(execution)f(of)f(all)g(in)o(v)o(olv)o(ed)g(pro)q(cesses,)j(with)e(a)f (suitable)h(error)75 1867 y(message)k(b)q(eing)g(returned)i(to)e(the)h(user.) 22 b(Ho)o(w)o(ev)o(er,)16 b(the)g(user)g(can)f(pro)o(vide)g(his)g(or)g(her)h (o)o(wn)f(signal)f(handling)75 1917 y(routine.)j(In)11 b(particular,)f(the)i (user)g(can)e(sp)q(ecify)i(a)e(\\no)q(op")g(signal)g(handler,)h(th)o(us)g (relegating)f(all)g(error)h(handling)75 1967 y(to)j(the)g(user)h(co)q(de,)f (using)g(the)h(error)f(parameters)g(returned)i(b)o(y)e(the)g(MPI)g(calls.)158 2023 y(MPI)h(calls)g(ma)o(y)e(initiate)h(op)q(erations)h(that)g(con)o(tin)o (ue)g(async)o(hronously)g(after)g(the)h(call)e(returned.)23 b(Th)o(us,)75 2072 y(the)d(op)q(eration)e(ma)o(y)f(return)k(with)d(a)h(co)q (de)h(indicating)d(successful)k(completion,)d(y)o(et)h(later)g(cause)h(an)f (error)75 2122 y(exception)d(to)f(b)q(e)g(raised.)22 b(If)15 b(there)h(is)f(a)g(subsequen)o(t)i(call)d(that)h(relates)h(to)f(the)h(same)e (op)q(eration)h(\(e.g.,)f(a)h(call)75 2172 y(that)i(v)o(eri\014es)i(that)e (an)g(async)o(hronous)i(op)q(eration)e(has)h(completed\))f(then)h(the)g (error)g(parameter)f(asso)q(ciated)75 2222 y(with)e(this)g(call)g(will)f(b)q (e)i(used)g(to)f(indicate)h(the)g(nature)g(of)e(the)i(error.)24 b(In)15 b(a)g(few)h(cases,)g(the)g(error)g(ma)o(y)e(o)q(ccur)75 2272 y(after)h(all)f(calls)h(that)g(relate)h(to)e(the)i(op)q(eration)f(ha)o (v)o(e)g(completed,)f(so)h(that)h(no)e(error)i(parameter)f(can)g(b)q(e)h (used)75 2321 y(to)e(indicate)f(the)i(nature)f(of)f(the)i(error)g(\(e.g.,)d (an)i(error)h(in)e(a)g(send)i(with)e(the)i(ready)f(mo)q(de\).)j(In)d(suc)o(h) g(cases,)h(an)75 2371 y(error)g(will)d(b)q(e)j(undetected,)g(if)e(the)i(user) g(disabled)f(the)g(MPI)g(error)h(signal.)158 2506 y Fm(Discussion:)35 b Fl(The)13 b(alternativ)o(e)i(c)o(hoice)f(is)g(to)f(ha)o(v)o(e)g(fatal)h (and)g(non-fatal)g(signals.)158 2558 y(One)f(migh)o(t)h(w)o(an)o(t)f (di\013eren)o(t)h(signals)i(for)c(di\013eren)o(t)j(mo)q(dules.)158 2614 y(The)e(details)i(of)e(suc)o(h)g(prop)q(osal)i(need)f(b)q(e)f(elab)q (orated)i(in)f(an)f(appropriate)i(\\pro\014le")g(sub)q(committee.)p eop %%Page: 5 6 bop 75 -100 a Fg(1.4.)31 b(MESSA)o(GES)1456 b Fo(5)75 45 y Fn(1.4)70 b(Messages)75 136 y Fo(A)14 b(message)g(consists)g(of)g(an)f Fj(envelop)n(e)i Fo(and)f Fj(data)p Fo(.)75 252 y Fk(1.4.1)55 b(Data)75 329 y Fo(The)17 b(data)f(part)h(of)f(a)g(message)g(consists)h(of)f (a)g(sequence)j(of)d(v)n(alues,)g(eac)o(h)h(of)f(a)g(basic)h(datat)o(yp)q(e)g (in)f(the)h(host)75 379 y(language.)29 b(Th)o(us,)19 b(in)e(F)m(ortran)h(77,) f(a)h(message)f(consists)i(of)e(a)h(sequence)i(of)d(v)n(alues)h(that)g(are)g (eac)o(h)g(of)f(t)o(yp)q(e)75 428 y Fi(INTEGER)p Fo(,)12 b Fi(REAL)p Fo(,)h Fi(DOUBLE)20 b(PRECISION)p Fo(,)12 b Fi(COMPLEX)p Fo(,)g Fi(DOUBLE)20 b(PRECISION)g(COMPLEX)p Fo(,)12 b Fi(LOGICAL)p Fo(,)g(or)i(\(length)g(1\))75 478 y Fi(CHARACTER)p Fo(.)i(A)i(message)f(ma)o (y)f(also)h(con)o(tain)h(a)f(v)n(alue)h(of)f(t)o(yp)q(e)h Fi(BYTE)p Fo(,)f(whic)o(h)h(consists)h(of)e(8)h(binary)f(digits.)75 528 y(A)e(b)o(yte)g(is)g(di\013eren)o(t)h(from)d(a)i(c)o(haracter.)23 b(Di\013eren)o(t)15 b(mac)o(hines)f(ma)o(y)f(ha)o(v)o(e)i(di\013eren)o(t)h (represen)o(tations)h(for)d(the)75 578 y(same)e(c)o(haracter,)i(or)f(ma)o(y)e (use)i(more)f(than)h(one)g(b)o(yte)g(to)g(represen)o(t)i(a)e(c)o(haracter.)19 b(On)13 b(the)g(other)h(hand,)e(a)h(b)o(yte)75 628 y(has)h(the)h(same)e (binary)g(v)n(alue)g(on)h(an)o(y)f(mac)o(hine.)k(A)d(message)g(ma)o(y)d(mix)h (v)n(alues)i(of)f(di\013eren)o(t)i(t)o(yp)q(es.)158 756 y Fm(Missing:)158 802 y Fl(Need)g(to)h(agree)g(on)f(the)h(C)f(t)o(yp)q(es)h(\(including)i (handling)g(of)e(signed/unsign)q(ed\),)i(and)e(on)g(F)m(ortran)g(90)f(t)o(yp) q(es)h(\(in-)75 847 y(cluding)f(handling)h(of)d(kinds\).)75 1046 y Fk(1.4.2)55 b(En)n(v)n(elop)r(e)75 1123 y Fo(The)14 b(follo)o(wing)e(information)f(is)i(asso)q(ciated)i(with)e(eac)o(h)i (message:)75 1205 y Fh(source)20 b Fo(The)14 b(rank)g(of)f(the)i(sending)f (pro)q(cess)75 1288 y Fh(destinatio)o(n)k Fo(The)c(rank)g(of)f(the)i (receiving)f(pro)q(cess)75 1370 y Fh(tag)20 b Fo(User)15 b(de\014ned)75 1453 y Fh(con)o(text)k Fo(handle)158 1536 y(The)14 b(range)g(of)f(v)n(alid)f (v)n(alues)h(for)g(the)h Fh(source)f Fo(and)g Fh(destination)d Fo(\014elds)k(is)f Fi(0)22 b(...)43 b(n-1)p Fo(,)12 b(where)j Fi(n)e Fo(is)g(the)75 1585 y(n)o(um)o(b)q(er)e(of)h(pro)q(cesses)j(in)c(the)i (sp)q(eci\014ed)h(con)o(text.)k Fi(source)j(=)g(destination)10 b Fo(is)i(allo)o(w)o(ed:)k(a)c(pro)q(cess)i(can)e(send)75 1635 y(a)i(message)f(to)h(itself.)158 1685 y(The)g(ranges)h(of)e(v)n(alid)f(v)n (alues)i(for)f Fi(tag)g Fo(is)h(implem)o(en)o(tation)d(dep)q(enden)o(t,)k (and)f(can)g(b)q(e)g(found)f(b)o(y)h(calling)e(a)75 1735 y(suitable)i(query)g (function,)f(as)h(describ)q(ed)i(in)d(Section)i Fh(??)p Fo(.)158 1785 y(The)f Fi(tag)e Fo(\014eld)i(can)f(b)q(e)h(arbitrarily)e(set)j(b)o(y)e (the)h(application,)d(and)i(can)h(b)q(e)g(used)g(to)f(distinguish)g (di\013eren)o(t)75 1835 y(messages.)158 1884 y Fi(Context)f Fo(should)i(b)q(e)h(a)e(con)o(text)i(shared)g(b)o(y)e(b)q(oth)h(source)h(and) f(destination.)158 1934 y(The)g(actual)g(mec)o(hanism)d(used)k(to)f(asso)q (ciate)g(an)g(en)o(v)o(elop)q(e)g(with)g(a)f(message)h(is)g(implem)o(en)o (tation)d(dep)q(en-)75 1984 y(den)o(t;)h(some)f(of)g(the)i(information)8 b(\(e.g.,)j Fh(sender)f Fo(or)i Fh(receiv)o(er)p Fo(\))e(ma)o(y)g(b)q(e)i (implicit,)d(and)i(need)i(not)f(b)q(e)g(explicitly)75 2034 y(carried)j(b)o(y)e(a)h(message.)75 2171 y Fn(1.5)70 b(Comm)n(unication)20 b(Bu\013ers)75 2262 y Fo(The)h(basic)f(p)q(oin)o(t)g(to)g(p)q(oin)o(t)g(comm) o(unicatio)o(n)e(op)q(erations)i(are)h Fh(send)e Fo(and)h Fh(receiv)o(e)p Fo(.)36 b(A)20 b Fh(send)f Fo(op)q(eration)75 2312 y(creates)c(a)d(message;)g (the)i(message)e(data)h(is)f(tak)o(en)h(from)e(the)j Fh(send)f(bu\013er)p Fo(.)j(A)d Fh(receiv)o(e)e Fo(op)q(eration)i(consumes)75 2361 y(a)j(message;)h(the)g(message)f(data)g(is)g(put)h(in)o(to)e(the)i Fh(receiv)o(e)h(bu\013er)p Fo(.)23 b(The)17 b(sp)q(eci\014cation)g(of)f(send) h(or)g(receiv)o(e)75 2411 y(bu\013ers)e(uses)g(the)g(same)e(syn)o(tax.)75 2527 y Fk(1.5.1)55 b(Comm)n(unication)21 b(bu\013er)d(comp)r(onen)n(ts)75 2604 y Fo(A)d(bu\013er)g(consists)h(of)d(a)i(sequence)h(of)e Fh(bu\013er)h(comp)q(onen)o(ts)p Fo(.)j(Eac)o(h)d(comp)q(onen)o(t)e(consists) j(of)e(a)g(sequence)j(of)75 2654 y(\(not)f(necessarily)g(distinct\))g(v)n (ariables)f(of)g(the)h(same)f(basic)h(datat)o(yp)q(e.)23 b(The)16 b(datat)o(yp)q(e)g(of)f(suc)o(h)h(a)g(v)n(ariable)e(is)75 2704 y(sp)q(eci\014ed)g(b)o(y)f(a)f(state)i(parameter.)j(The)d(p)q(ossible)f(v)n (alues)f(for)h(F)m(ortran)f(are)h Fi(MPI)p 1357 2704 14 2 v 15 w(INT)p Fo(,)f Fi(MPI)p 1528 2704 V 15 w(REAL)p Fo(,)g Fi(MPI)p 1721 2704 V 15 w(DOUBLE)p Fo(,)p eop %%Page: 6 7 bop 75 -100 a Fo(6)733 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g (COMMUNICA)m(TION)75 45 y Fi(MPI)p 144 45 14 2 v 15 w(COMPLEX)p Fo(,)k Fi(MPI)p 409 45 V 15 w(DCOMPLEX)p Fo(,)g Fi(MPI)p 696 45 V 15 w(LOGICAL)p Fo(,)h Fi(MPI)p 962 45 V 15 w(CHAR)g Fo(and)h Fi(MPI)p 1237 45 V 15 w(BYTE)p Fo(,)f(corresp)q(onding)i(to)f(datat)o(yp)q (es)75 95 y Fi(INTEGER)p Fo(,)e Fi(REAL)p Fo(,)g Fi(DOUBLE)j(PRECISION)p Fo(,)c Fi(COMPLEX)p Fo(,)h Fi(DOUBLE)i(PRECISION)g(COMPLEX)p Fo(,)e Fi(LOGICAL)p Fo(,)g Fi(CHARACTER)75 145 y Fo(and)j(to)g(un)o(t)o(yp)q (ed)g Fi(BYTE)p Fo(.)f(The)h(p)q(ossible)g(v)n(alues)g(for)g(C)g(are)g Fi(MPI)p 1162 145 V 15 w(CHAR)p Fo(,)f Fi(MPI)p 1363 145 V 15 w(SHORT)p Fo(,)f Fi(MPI)p 1585 145 V 15 w(INT)p Fo(,)h Fi(MPI)p 1764 145 V 15 w(LONG)p Fo(,)75 195 y Fi(MPI)p 144 195 V 15 w(FLOAT)p Fo(,)12 b Fi(MPI)p 359 195 V 15 w(DOUBLE)h Fo(and)h Fi(MPI)p 666 195 V 15 w(BYTE)p Fo(.)158 325 y Fm(Discussion:)158 372 y Fl(Ma)o(y)g(need)g(to)g(tak)o(e)g(care)f(of)h(kinds)h(in)f(F)m(ortran)g (90)g(and)g(of)g(signed/unsign)q(ed)j(in)d(C.)f(More)h(elab)q(orate)h (de\014nitions)75 418 y(b)q(elong)g(to)e(the)g(language)i(binding.)158 552 y Fo(There)g(are)f(\014v)o(e)g(kinds)g(of)g(bu\013er)g(comp)q(onen)o(ts:) 75 670 y Fh(Con)o(tiguous)f(comp)q(onen)o(t)75 750 y Fo(A)h(sequence)i(of)d (con)o(tiguous)h(v)n(alues)g(of)f(the)h(same)f(basic)h(t)o(yp)q(e,)g(sp)q (eci\014ed)i(b)o(y)75 839 y Fh(start)k Fo(Initial)12 b(elemen)o(t)75 930 y Fh(coun)o(t)19 b Fo(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\()p Fi(count)c Fe(\025)i Fi(0)p Fo(\))75 1020 y Fh(datat)o(yp)q(e)19 b Fo(T)o(yp)q(e)c(of)e(elemen)o(ts)158 1109 y(Th)o(us,)h(if)e Fi(A)i Fo(is)f(an)h(arra)o(y)f(of)g(in)o(tegers,)h(then)h Fi(A\(1\),)21 b(4,)g(MPI)p 1127 1109 V 15 w(INT)13 b Fo(is)h(the)g(bu\013er)h(comp)q(onen)o (t)e(with)g(en)o(tries)75 1159 y Fi(A\(1\),)21 b(A\(2\),)g(A\(3\),)f(A\(4\))p Fo(.)75 1277 y Fh(V)l(ector)15 b(comp)q(onen)o(t)75 1357 y Fo(A)f(sequence)j(of)c(equally)h(spaced)h(and)f(equally)f(sized)i(blo)q(c)o (ks)f(of)g(elemen)o(ts)g(of)g(the)g(same)g(basic)g(t)o(yp)q(e,)g(sp)q (eci\014ed)75 1407 y(b)o(y)75 1495 y Fh(start)20 b Fo(Initial)12 b(elemen)o(t)75 1586 y Fh(coun)o(t)19 b Fo(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\()p Fi(count)c Fe(\025)i Fi(0)p Fo(\))75 1677 y Fh(stride)18 b Fo(Num)o(b)q(er)c(of)f(elemen)o(ts)h(b)q(et)o(w)o(een)h (the)g(start)f(of)f(eac)o(h)i(blo)q(c)o(k)75 1767 y Fh(len)o(blk)j Fo(Num)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(eac)o(h)g(blo)q(c)o(k)g(\()p Fi(lenblk)c Fd(>)i Fi(0)p Fo(\))75 1858 y Fh(datat)o(yp)q(e)19 b Fo(T)o(yp)q(e)c(of)e(elemen)o(ts)158 1947 y(Assume)20 b(that)h(an)f(elemen) o(t)g(of)f(t)o(yp)q(e)i Fi(datatype)e Fo(o)q(ccupies)i Fi(m)f Fo(b)o(ytes.)38 b(Then,)22 b(in)e(a)g(b)o(yte)h(addressable)75 1996 y(mac)o(hine,)11 b(the)i(bu\013er)h(comp)q(onen)o(t)d(describ)q(ed)k(b)o (y)d(these)i(\014v)o(e)f(parameters)f(consists)i(of)e(the)h Fi(count)e Fo(v)n(ariables)h(at)75 2046 y(addresses)k Fi(start,)21 b(start+m,)f(...)43 b(,)21 b(start+m*\(lenblk-1\),)d(start+m*stride,)75 2096 y(start+m*\(stride+1)o(\),)h(...,)i(start+m*\(stride+l)o(enbl)o(k-1\),)d (...)p Fo(.)i(F)m(or)14 b(example,)f(if)h Fi(A)g Fo(is)h(an)f(arra)o(y)h(of) 75 2146 y(in)o(tegers,)f(then)137 2234 y Fe(\017)21 b Fi(A\(1\),)g(5,)g(3,)g (2,)h(MPI)p 575 2234 V 15 w(INT)c Fo(is)h(the)h(bu\013er)g(comp)q(onen)o(t)e (with)h(en)o(tries)h Fi(A\(1\),)h(A\(2\),)f(A\(4\),)h(A\(5\),)179 2284 y(A\(7\))p Fo(.)137 2375 y Fe(\017)g Fi(A\(1\),)g(5,)g(0,)g(2,)h(MPI)p 575 2375 V 15 w(INT)c Fo(is)h(the)h(bu\013er)g(comp)q(onen)o(t)e(with)h(en)o (tries)h Fi(A\(1\),)h(A\(2\),)f(A\(1\),)h(A\(2\),)179 2425 y(A\(1\))p Fo(.)137 2515 y Fe(\017)g Fi(A\(10\),)f(5,)i(-1,)f(2,)g(MPI)p 618 2515 V 15 w(INT)9 b Fo(is)g(the)h(bu\013er)g(comp)q(onen)o(t)f(with)g(en) o(tries)h Fi(A\(10\),)21 b(A\(11\),)f(A\(9\),)h(A\(10\),)179 2565 y(A\(8\))p Fo(.)158 2654 y Fi(Contiguous)13 b Fo(is)h(a)g(sp)q(ecial)h (case)h(of)e Fi(vector)p Fo(.)19 b(A)c(v)o(ector)g(bu\013er)h(comp)q(onen)o (t)d(can)i(b)q(e)h(used)f(to)f(extract)i(an)75 2704 y(arbitrary)e(submatrix)e (of)i(a)f(t)o(w)o(o-dimensional)e(matrix.)p eop %%Page: 7 8 bop 75 -100 a Fg(1.5.)31 b(COMMUNICA)m(TION)14 b(BUFFERS)1091 b Fo(7)75 45 y Fh(Indexed)15 b(comp)q(onen)o(t)75 122 y Fo(A)f(sequence)i(of) d(elemen)o(ts)h(of)f(the)i(same)e(basic)h(t)o(yp)q(e,)g(sp)q(eci\014ed)h(b)o (y)75 207 y Fh(start)20 b Fo(initial)12 b(elemen)o(t)75 291 y Fh(coun)o(t)19 b Fo(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\()p Fi(count)c Fe(\025)i Fi(0)p Fo(\))75 376 y Fh(arra)o(y)p 188 376 15 2 v 17 w(of)p 244 376 V 16 w(indices)19 b Fo(Arra)o(y)14 b(of)f(displacemen)o(ts)h(of)f(the)i(elemen)o(ts)e(in)h(the)h(bu\013er)g (comp)q(onen)o(ts,)e(relativ)o(e)h(to)f(the)179 425 y(initial)f(elemen)o(t.) 75 510 y Fh(datat)o(yp)q(e)19 b Fo(T)o(yp)q(e)c(of)e(elemen)o(ts)158 594 y(F)m(or)f(example,)f(if)h Fi(A)g Fo(is)g(an)g(arra)o(y)g(of)g(in)o (tegers,)h(and)g Fi(I)f Fo(is)g(an)g(arra)o(y)h(with)f(v)n(alues)g Fi(2,)21 b(-1,)g(2,)h(1)12 b Fo(then)h Fi(A\(3\),)75 644 y(4,)21 b(I,)h(MPI)p 275 644 14 2 v 15 w(INT)13 b Fo(is)h(the)g(bu\013er)h(comp)q (onen)o(t)e(with)h(en)o(tries)h Fi(A\(5\),)21 b(A\(2\),)f(A\(5\),)h(A\(4\))p Fo(.)158 694 y Fi(Vector)13 b Fo(can)h(b)q(e)g(though)o(t)g(of)f(as)h(a)g(sp) q(ecial)g(case)h(of)e Fi(indexed)g Fo(in)g(whic)o(h)h(the)g(indices)h(are)f (generated)h(with)75 744 y(a)f(constan)o(t)g(stride.)75 854 y Fh(Heterogeneous)f(v)o(ector)i(\(h)o(v)o(ector\))f(comp)q(onen)o(t)75 931 y Fo(Same)d(as)h Fi(vector)p Fo(,)f(except)j(that)e Fi(stride)f Fo(is)h(giv)o(en)g(in)f(m)o(ultiple)f(of)i(b)o(ytes,)h(rather)g(than)f(m)o (ultiples)e(of)i(elemen)o(ts:)75 1024 y Fh(start)20 b Fo(Initial)12 b(elemen)o(t)75 1108 y Fh(coun)o(t)19 b Fo(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\()p Fi(count)c Fe(\025)i Fi(0)p Fo(\))75 1193 y Fh(stride)18 b Fo(Num)o(b)q(er)c(of)f(b)o(ytes)i(b)q(et)o(w)o(een)g (the)g(start)f(of)f(eac)o(h)i(blo)q(c)o(k)75 1278 y Fh(len)o(blk)j Fo(Num)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(eac)o(h)g(blo)q(c)o(k)g(\()p Fi(lenblk)c Fd(>)i Fi(0)p Fo(\))75 1362 y Fh(datat)o(yp)q(e)19 b Fo(T)o(yp)q(e)c(of)e(elemen)o(ts)158 1455 y(Assume)20 b(that)h(an)f(elemen) o(t)g(of)f(t)o(yp)q(e)i Fi(datatype)e Fo(o)q(ccupies)i Fi(m)f Fo(b)o(ytes.)38 b(Then,)22 b(in)e(a)g(b)o(yte)h(addressable)75 1505 y(mac)o(hine,)15 b(the)i(bu\013er)g(comp)q(onen)o(t)f(describ)q(ed)i(b)o (y)e(these)i(\014v)o(e)e(parameters)h(consists)g(of)f(the)g Fi(count)g Fo(v)n(ariables)75 1554 y(at)c(addresses)j Fi(start,)20 b(start+m,)h(...)43 b(,)21 b(start+m*\(lenblk-1\))o(,)e(start+stride,)g (start+stride+m,)75 1604 y(...,)i(start+stride+m*\(l)o(enblk)o(-1\),)d(...)p Fo(.)158 1654 y(Consider)c(a)g(C)g(arra)o(y)f(declared)i(as)75 1748 y Fi(struct)21 b({)140 1798 y(char)g(info[3];)140 1847 y(short)g(number;)140 1897 y(double)g(val;)228 1947 y(})65 b(a[100])158 2040 y Fo(Assume)19 b(that)f(all)g(structure)j(comp)q(onen)o(ts) d(are)h(b)o(yte)h(aligned,)e(a)h Fi(short)e Fo(o)q(ccupies)j(t)o(w)o(o)e(b)o (ytes,)j(and)d(a)75 2090 y Fi(double)i Fo(o)q(ccupies)i(eigh)o(t)e(b)o(ytes.) 40 b(Then)21 b(the)h(bu\013er)g(comp)q(onen)o(t)e Fi(&a,)h(100,)g(13,)g(2,)g (MPI)p 1610 2090 V 16 w(CHAR)f Fo(extracts)75 2139 y(the)i(\014rst)h(t)o(w)o (o)e(c)o(haracters)i(of)e(eac)o(h)h(structure.)43 b(The)22 b(bu\013er)h(comp)q(onen)o(t)e Fi(&\(a.number\),)e(100,)i(13,)g(1,)75 2189 y(MPI)p 144 2189 V 15 w(SHORT)16 b Fo(extracts)i(the)f Fi(short)f Fo(\014eld)g(of)g(eac)o(h)i(structure;)h(and)e(the)g(bu\013er)h (comp)q(onen)o(t)e Fi(&\(a.val\),)k(100,)75 2239 y(13,)h(1)e Fo(extracts)i(the)f Fi(double)e Fo(\014eld)h(of)g(eac)o(h)h(structure.)36 b(Th)o(us,)21 b(heterogeneous)g(v)o(ectors)f(can)g(b)q(e)g(used)g(to)75 2289 y(extract)15 b(v)o(ectors)g(of)e(homogeneous)g(v)n(alues)h(from)e (heterogeneous)k(data.)158 2339 y Fi(Vector)d Fo(is)g(a)h(sp)q(ecial)g(case)h (of)e Fi(heterogeneous)19 b(vector)p Fo(.)75 2449 y Fh(Heterogeneous)13 b(indexed)h(\(hindexed\))f(comp)q(onen)o(t)75 2526 y Fo(Same)g(as)h(indexed,) g(except)h(that)f(displacemen)o(ts)f(are)i(in)e(b)o(ytes,)h(rather)h(than)f (m)o(ultiples)e(of)h(elemen)o(t)g(size.)75 2619 y Fh(start)20 b Fo(initial)12 b(elemen)o(t)75 2704 y Fh(coun)o(t)19 b Fo(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\()p Fi(count)c Fe(\025)i Fi(0)p Fo(\))p eop %%Page: 8 9 bop 75 -100 a Fo(8)733 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g (COMMUNICA)m(TION)75 45 y Fh(arra)o(y)p 188 45 15 2 v 17 w(of)p 244 45 V 16 w(indices)19 b Fo(Arra)o(y)12 b(of)g(b)o(yte)g(displacemen)o(ts)g (of)g(the)g(elemen)o(ts)h(in)e(the)i(bu\013er)g(comp)q(onen)o(ts,)f(relativ)o (e)g(to)179 95 y(the)i(initial)e(elemen)o(t.)75 189 y Fh(datat)o(yp)q(e)19 b Fo(T)o(yp)q(e)c(of)e(elemen)o(ts)158 292 y(F)m(or)18 b(example,)f(if)h Fi(a)g Fo(is)g(the)h(C)f(arra)o(y)g(of)f(the)i(previous)g(example,)e(and)i Fi(I)e Fo(is)i(an)f(arra)o(y)g(with)f(v)n(alues)h Fi(0,)75 342 y(13,)j(13,)g(27,)h(26)16 b Fo(then)i Fi(&a,)k(5,)f(I,)g(MPI)p 782 342 14 2 v 15 w(CHAR)c Fo(is)g(the)h(bu\013er)h(comp)q(onen)o(t)d(with)h (en)o(tries)i Fi(a[0].info[0],)75 392 y(a[1].info[0],)g(a[1].info[0],)g (a[1].info[1],)g(a[1].info[0])p Fo(.)158 444 y(Lik)o(e)e Fi(heterogeneous)i (vector)p Fo(,)d Fi(heterogeneous)j(indexed)c Fo(extends)k(the)e Fi(indexed)f Fo(facilit)o(y)f(so)i(as)g(to)75 494 y(allo)o(w)12 b(the)j(extraction)f(of)f(homogeneous)g(data)h(from)e(a)h(heterogeneous)j(la) o(y)o(out.)158 625 y Fm(Discussion:)158 674 y Fl(The)d(use)h(of)e (displacemen)o(ts)k(in)e(indexed)h(bu\013er)f(comp)q(onen)o(ts)h(has)e(the)h (same)f(e\013ect)g(as)h(indirect)h(addressing)g(in)o(to)75 720 y(an)d(arra)o(y)g(with)g(initial)j(index)e(zero.)j(I.e.,)11 b(if)h Ff(a)f Fl(is)h(an)g(arra)o(y)g(with)g(initial)j(index)e(zero,)e(then)h (a)g(bu\013er)h(comp)q(onen)o(t)g(de\014ned)75 765 y(b)o(y)g(parameters)g Ff(a[0],)k(N,)i(i,)g(type)11 b Fl(extract)h(the)h(elemen)o(ts)g Ff(a[i[0]],)j(a[i[1]],)h(...,)h(a[i[n]])p Fl(.)c(This)f(\014ts)f(w)o(ell)h(C) 75 811 y(usage)g(but)f(ill-\014ts)i(F)m(ortran)e(usage)h(\(initial)h(index)g (one\).)i(W)m(e)c(to)q(ok)h(a)f(decision)i(of)d(principle)k(to)d(force)g (\\start)g(from)f(zero")75 857 y(index)j(arithmetic)f(on)g(F)m(ortran)f (users,)h(rather)f(than)h(ha)o(v)o(e)f(di\013eren)o(t)i(index)g(arithmetic)f (in)g(the)f(C)g(and)h(F)m(ortran)f(binding)75 902 y(of)h(MPI.)158 951 y(The)19 b(use)f(of)h(b)q(oth)g(bu\013er)h(comp)q(onen)o(ts)f(with)h(b)o (yte)e(and)i(elemen)o(t)f(size)g(displacemen)o(ts)j(is)d(motiv)n(ated)h(b)o (y)f(the)75 996 y(observ)n(ation)c(that)f(some)g(applications)j(require)d (the)g(more)f(general)i(b)o(yte)f(displacemen)o(t,)h(but)f(that)g(most)f (applicatio)q(ns)75 1042 y(w)o(ould)d(use)g(elemen)o(t)g(size)h(displacemen)o (t,)h(whic)o(h)e(is)g(m)o(uc)o(h)g(more)g(natural)g(and)g(con)o(v)o(enien)o (t)i(\(and)e(mac)o(hine)h(indep)q(enden)o(t\).)75 1088 y(Th)o(us,)f(rather)f (than)h(forcing)g(all)g(users)g(to)f(do)g(b)o(yte)h(arithmetics,)h(w)o(e)e (decided)h(to)f(pro)o(vide)i(separately)g(the)e(more)g(adv)n(anced)75 1133 y(function)14 b(to)f(\\exp)q(ert)h(programmers".)158 1182 y(Leslie)g(Hart)f(prop)q(oses)i(to)d(replace)j(\\heterogeneous")g(with)e (\\general".)75 1396 y Fk(1.5.2)55 b(Bu\013er)18 b(op)r(erations)75 1478 y Fo(A)c(bu\013er)i(is)e(describ)q(ed)i(b)o(y)e(an)g(opaque)g Fh(bu\013er)h(descriptor)c Fo(ob)r(ject)k(accessed)h(via)e(a)g Fh(bu\013er)h(handle)p Fo(.)i(Suc)o(h)75 1528 y(an)f(ob)r(ject)g(is)g (created)h(b)o(y)f Fi(MPI)p 582 1528 V 15 w(CREATE)p 729 1528 V 15 w(BUFFER)p Fo(.)e(It)i(is)f(asso)q(ciated)i(with)f(successiv)o(e)i (bu\013er)f(comp)q(onen)o(ts)e(b)o(y)75 1577 y(calling)g(in)i(succession)h (one)f(of)f(the)i(functions)e Fi(MPI)p 914 1577 V 15 w(APPEND)p 1061 1577 V 15 w(xxx)p Fo(,)g(where)i Fi(xxx)e Fo(iden)o(ti\014es)h(the)h(t)o (yp)q(e)f(of)f(bu\013er)75 1627 y(comp)q(onen)o(t)e(app)q(ended)h(to)f(the)h (bu\013er.)20 b(It)14 b(b)q(ecomes)h(a)o(v)n(ailable)d(for)i(use)h(in)f(comm) o(unicatio)o(n)e(op)q(erations)i(once)75 1677 y(committed)c(b)o(y)h(the)i (function)e Fi(MPI)p 635 1677 V 15 w(COMMIT)p 782 1677 V 15 w(BUFFER)p Fo(.)f(Once)j(committed,)c(the)k(descriptor)g(cannot)f(b)q(e)g(mo) q(di\014ed)75 1727 y(an)o(y)h(more.)158 1780 y(Note)18 b(that)g(the)g(commit) c(op)q(eration)k(do)q(es)g(not)g(create)h(y)o(et)f(a)f(message,)h(and)f(do)q (es)h(not)g(bind)f(the)h(data)75 1829 y(to)d(b)q(e)h(sen)o(t.)23 b(It)16 b(is)f(a)g(commit)d(for)j(the)h(bu\013er)g(descriptor,)h(not)e(a)g (commit)d(for)j(the)h(bu\013er)h(con)o(ten)o(t.)23 b(A)15 b(bu\013er)75 1879 y(descriptor)f(is)e(destro)o(y)o(ed)i(after)f(the)g(completion)d(of)i (the)h(\014rst)h(comm)o(unicati)o(on)c(op)q(eration)i(that)h(uses)g(it,)f(if) g(it)g(is)75 1929 y(ephemeral,)g(or)h(after)g(it)g(is)g(freed)h(b)o(y)e(a)h (call)f(to)h Fi(MPI)p 899 1929 V 15 w(FREE)f Fo(and)h(an)o(y)f(p)q(ending)i (comm)o(uni)o(cation)c(op)q(eration)j(that)75 1979 y(uses)j(it)e(has)h (completed,)f(if)f(it)i(is)f(p)q(ersisten)o(t.)22 b(After)15 b(the)h(bu\013er)f(descriptor)h(ob)r(ject)f(has)g(b)q(een)h(destro)o(y)o(ed,) f(the)75 2029 y(handle)f(is)f(unde\014ned.)158 2117 y Fh(MPI)p 257 2117 15 2 v 17 w(CREA)l(TE)p 471 2117 V 19 w(BUFFER\()i(bu\013er,)g(p)q (ersistence)e(\))158 2205 y Fo(Create)i(a)e(new)i(bu\013er)g(descriptor)g(ob) r(ject.)k(The)14 b(parameters)g(are:)75 2296 y Fh(OUT)i(bu\013er)j Fo(bu\013er)c(handle)75 2391 y Fh(IN)h(p)q(ersistence)i Fo(bu\013er)11 b(p)q(ersistence)h(\(state)e(t)o(yp)q(e)g(with)f(p)q(ossible)g(v)n(alues)g Fi(MPI)p 1350 2391 14 2 v 15 w(PERSISTENT)e Fo(or)j Fi(MPI)p 1705 2391 V 15 w(EPHEMERAL)p Fo(\).)158 2561 y Fm(Discussion:)158 2609 y Fl(Ma)o(y)j(w)o(an)o(t)g(to)g(pro)o(vide)i(information)g(on)e(maxim)o (um)h(ob)r(ject)g(size,)f(and)h(p)q(erhaps)g(user)f(space)h(for)f(the)g(ob)r (ject.)158 2658 y(Leslie)20 b(Hart)e(prop)q(oses)i Ff(MPI)p 592 2658 12 2 v 13 w(CREATE)p 725 2658 V 12 w(BUFFER)p 857 2658 V 12 w(DESC)p Fl(,)c(rather)j(than)g Ff(MPI)p 1252 2658 V 13 w(CREATE)p 1385 2658 V 12 w(BUFFER)p Fl(.)c(More)k(accurate,)h(but)75 2704 y(longer.)e(P)o(erhaps)c Ff(MPI)p 414 2704 V 13 w(CREATE)p 547 2704 V 12 w(BUFDESC)p Fl(?)p eop %%Page: 9 10 bop 75 -100 a Fg(1.5.)31 b(COMMUNICA)m(TION)14 b(BUFFERS)1091 b Fo(9)158 45 y Fh(MPI)p 257 45 15 2 v 17 w(APPEND)p 481 45 V 17 w(CONTIGUOUS\()15 b(bu\013er,)g(start,)g(coun)o(t,)f(datat)o(yp)q(e\)) 158 130 y Fo(App)q(end)h(a)e(blo)q(c)o(k)h(comp)q(onen)o(t)f(to)h(bu\013er.) 19 b(The)14 b(parameters)g(are:)75 208 y Fh(INOUT)i(bu\013er)j Fo(bu\013er)c(handle)75 289 y Fh(IN)h(start)k Fo(bu\013er)15 b(comp)q(onen)o(t)e(initial)f(elemen)o(t)h(\(c)o(hoice\))75 369 y Fh(IN)j(coun)o(t)j Fo(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\(in)o (teger\))75 450 y Fh(IN)i(datat)o(yp)q(e)k Fo(datat)o(yp)q(e)14 b(iden)o(ti\014er)g(\(status\))158 563 y Fh(MPI)p 257 563 V 17 w(APPEND)p 481 563 V 17 w(VEC\()j(bu\013er,)d(coun)o(t,)h(stride,)f(len)o (blk,)f(datat)o(yp)q(e)i(\))158 648 y Fo(App)q(end)g(a)e(v)o(ector)i (bu\013er)g(comp)q(onen)o(t)e(to)h(bu\013er.)19 b(The)14 b(parameters)g(are:) 75 726 y Fh(INOUT)i(bu\013er)j Fo(bu\013er)c(handle)75 807 y Fh(IN)h(start)k Fo(bu\013er)15 b(comp)q(onen)o(t)e(initial)f(elemen)o(t)h (\(c)o(hoice\))75 887 y Fh(IN)j(coun)o(t)j Fo(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\(in)o(teger\))75 967 y Fh(IN)i(stride)j Fo(Num)o(b)q(er)13 b(of)h(elemen)o(ts)f(b)q(et)o(w)o(een)j(the)e(start)h(of)e (eac)o(h)h(blo)q(c)o(k)g(\(in)o(teger\))75 1048 y Fh(IN)i(len)o(blk)i Fo(Num)o(b)q(er)c(of)f(elemen)o(ts)h(in)f(eac)o(h)i(blo)q(c)o(k)e(\(in)o (teger\))75 1128 y Fh(IN)j(datat)o(yp)q(e)k Fo(datat)o(yp)q(e)14 b(iden)o(ti\014er)g(\(status\))158 1242 y Fh(MPI)p 257 1242 V 17 w(APPEND)p 481 1242 V 17 w(INDEXED\()i(bu\013er,)f(start,)g(coun)o(t,)f (arra)o(y)p 1294 1242 V 17 w(of)p 1350 1242 V 17 w(indices,)f(datat)o(yp)q (e\))158 1327 y Fo(App)q(end)i(an)e(indexed)i(bu\013er)g(comp)q(onen)o(t)e (to)h(bu\013er.)19 b(The)14 b(parameters)g(are:)75 1405 y Fh(INOUT)i (bu\013er)j Fo(bu\013er)c(handle)75 1485 y Fh(IN)h(start)k Fo(initial)12 b(p)q(osition)h(for)h(indexing)f(\(c)o(hoice\))75 1566 y Fh(IN)j(coun)o(t)j Fo(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\(in)o (teger\))75 1646 y Fh(IN)i(arra)o(y)p 259 1646 V 17 w(of)p 315 1646 V 17 w(indices)i Fo(arra)o(y)c(of)f(displacemen)o(ts)g(of)h(en)o (tries)h(relativ)o(e)e(to)h(start)h(\(arra)o(y)e(of)h(in)o(tegers\))75 1727 y Fh(IN)i(datat)o(yp)q(e)k Fo(datat)o(yp)q(e)14 b(iden)o(ti\014er)g (\(status\))158 1805 y(The)g(index)f(v)n(alues)g(used)h(to)f(create)i(this)f (comp)q(onen)o(t)e(are)i(the)g(v)n(alues)f(pro)o(vided)g(at)g(the)h(call;)e (subsequen)o(t)75 1854 y(up)q(dates)j(of)e(the)i(arra)o(y)e(of)h(indices)g (will)e(not)i(a\013ect)h(the)f(bu\013er)h(descriptor)g(comp)q(onen)o(t.)158 1940 y Fh(MPI)p 257 1940 V 17 w(APPEND)p 481 1940 V 17 w(HVEC\()i(bu\013er,)e (start,)g(coun)o(t,)f(stride,)g(len)o(blk,)f(datat)o(yp)q(e)i(\))158 2025 y Fo(App)q(end)g(a)e(heterogeneous)j(v)o(ector)f(bu\013er)g(comp)q(onen) o(t)e(to)h(bu\013er.)19 b(The)14 b(parameters)g(are:)75 2103 y Fh(INOUT)i(bu\013er)j Fo(bu\013er)c(handle)75 2183 y Fh(IN)h(start)k Fo(bu\013er)15 b(comp)q(onen)o(t)e(initial)f(elemen)o(t)h(\(c)o(hoice\))75 2264 y Fh(IN)j(coun)o(t)j Fo(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\(in)o (teger\))75 2344 y Fh(IN)i(stride)j Fo(Num)o(b)q(er)13 b(of)h(b)o(ytes)g(b)q (et)o(w)o(een)h(the)g(start)f(of)g(eac)o(h)g(blo)q(c)o(k)g(\(in)o(teger\))75 2425 y Fh(IN)i(len)o(blk)i Fo(Num)o(b)q(er)c(of)f(elemen)o(ts)h(in)f(eac)o(h) i(blo)q(c)o(k)e(\(in)o(teger\))75 2505 y Fh(IN)j(datat)o(yp)q(e)k Fo(datat)o(yp)q(e)14 b(iden)o(ti\014er)g(\(status\))158 2618 y Fh(MPI)p 257 2618 V 17 w(APPEND)p 481 2618 V 17 w(HINDEXED\()i(bu\013er,)f (start,)g(coun)o(t,)g(arra)o(y)p 1332 2618 V 16 w(of)p 1387 2618 V 17 w(indices,)f(datat)o(yp)q(e\))158 2704 y Fo(App)q(end)h(an)e (heterogeneous)j(indexed)f(bu\013er)g(comp)q(onen)o(t)e(to)h(bu\013er.)19 b(The)14 b(parameters)g(are:)p eop %%Page: 10 11 bop 75 -100 a Fo(10)716 b Fg(CHAPTER)14 b(1.)27 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m(TION)75 45 y Fh(INOUT)i(bu\013er)j Fo(bu\013er)c(handle)75 124 y Fh(IN)h(start)k Fo(initial)12 b(p)q(osition)h(for)h(indexing)f(\(c)o(hoice\))75 203 y Fh(IN)j(coun)o(t)j Fo(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\(in)o(teger\))75 282 y Fh(IN)i(arra)o(y)p 259 282 15 2 v 17 w(of)p 315 282 V 17 w(indices)i Fo(arra)o(y)c(of)f(b)o(yte)h(displacemen)o(ts)g(of)f(en)o(tries)i (relativ)o(e)f(to)f(start)i(\(arra)o(y)f(of)f(in)o(tegers\))75 361 y Fh(IN)j(datat)o(yp)q(e)k Fo(datat)o(yp)q(e)14 b(iden)o(ti\014er)g (\(status\))158 471 y Fh(MPI)p 257 471 V 17 w(COMMIT)p 485 471 V 19 w(BUFFER\()h(bu\013erin)e(\))158 557 y Fo(Commi)o(t)7 b(a)j(bu\013er)h(descriptor)g(ob)r(ject.)18 b(The)10 b(ob)r(ject)h(cannot)f (b)q(e)g(c)o(hanged)h(an)o(ymore)d(after)i(it)g(is)g(committed.)75 606 y(A)h(bu\013er)g(descriptor)h(ob)r(ject)g(can)f(b)q(e)g(used)g(for)g (comm)o(uni)o(cation)d(only)i(after)g(it)h(is)f(committed.)15 b(The)c(parameters)75 656 y(are:)75 731 y Fh(INOUT)16 b(bu\013er)j Fo(bu\013er)c(descriptor)g(handle)158 885 y Fm(Discussion:)158 930 y Fl(The)i(commit)h(function)g(ma)o(y)g(do)f(prepro)q(cessing)j(of)d(the) g(bu\013er)h(descriptor)h(to)e(simplify)i(its)f(use)f(and)h(commit)75 976 y(resources)12 b(in)g(the)g(comm)o(unication)i(subsystem)e(for)f(its)h (use.)k(The)c(commit)f(function)i(ma)o(y)f(generate)g(a)f(new)g(ob)r(ject)g (and,)75 1022 y(hence,)i(mo)q(dify)h(the)g(ob)r(ject)f(handle.)158 1067 y(The)18 b(curren)o(t)g(prop)q(osal)h(do)q(es)g(not)f(allo)o(w)h(to)e (create)h(a)g(bu\013er)g(descriptor,)i(use)e(it)h(in)f(a)g(comm)o(unication)i (after)75 1113 y(committed,)14 b(then)g(mo)q(dify)g(it)g(partially)i(and)e (use)g(it)g(for)f(another)h(comm)o(unication)i({)d(a)h(new)f(bu\013er)h (descriptor)h(has)f(to)75 1159 y(b)q(e)h(created.)21 b(Of)13 b(course,)i(the)g(same)f(bu\013er)h(descriptor)h(can)f(b)q(e)f(reused)h(man)o (y)g(times)g(in)g(successiv)o(e)h(comm)o(unications)75 1204 y(that)d(send)g(data)g(from)f(the)g(same)h(bu\013ers;)g(e.g.,)f(a)g(new)h (send)g(with)g(a)f(bu\013er)i(descriptor)g(handle)g(can)f(b)q(e)f(initiated)j (once)75 1250 y(the)e(previous)i(send)f(with)f(this)h(handle)h(has)e (completed.)158 1382 y Fo(Example)f(1:)75 1432 y(Consider)i(the)h(follo)o (wing)c(fragmen)o(t)h(of)i(F)m(ortran)f(co)q(de)75 1513 y Fi(DOUBLE)21 b(PRECISION)e(A\(10,20\))75 1563 y(INTEGER)h(B,)i(C\(5,10\))75 1613 y(INTEGER)e(BH)75 1663 y(...)75 1713 y(CALL)h(MPI_CREATE_BUFFER)o(\(BH,) d(MPI_PERSISTENT\))75 1763 y(CALL)j(MPI_APPEND_BLOCK)d(\(BH,)j(B,)h(1,)f (MPI_INT\))75 1812 y(CALL)g(MPI_APPEND_VEC)e(\(BH,)i(A\(1,3\),)f(11,)h(4,)h (2,)f(MPI_DOUBLE\))75 1862 y(CALL)g(MPI_APPEND_INDEX\()o(BH,)e(C\(3,7\),)h (\(4,2,1\),)g(MPI_INT\))158 1943 y Fo(Then)14 b(the)h(bu\013er)g(asso)q (ciated)f(with)g(the)g(handle)g Fi(BH)g Fo(consists)h(of)e(the)h(sequence)i (of)e(v)n(ariables)158 1993 y Fi(B,)21 b(A\(1,3\),)g(A\(2,3\),)f(A\(5,3\),)g (A\(6,3\),)h(A\(9,3\),)f(A\(10,3\),)g(A\(3,4\),)h(A\(4,4\),)f(A\(7,4\),)g (A\(8,4\),)75 2043 y(A\(1,5\),)g(C\(2,8\),)h(C\(5,7\),)f(C\(4,7\))p Fo(.)158 2093 y(A)14 b(message)f(created)j(from)c(this)i(bu\013er)h(will)d (consist)j(of)e(a)g(sequence)j(of)e(one)g(in)o(teger,)f(follo)o(w)o(ed)g(b)o (y)g(elev)o(en)75 2143 y(double)h(precision)g(reals,)g(follo)o(w)o(ed)e(b)o (y)i(three)h(in)o(tegers.)158 2192 y(Example)d(2:)75 2242 y(Supp)q(ose)j(one) f(has)g(an)g(arra)o(y)f(of)g(data)h(declared)h(in)e(C)h(as)75 2323 y Fi(struct)21 b({)140 2373 y(int)h(type;)140 2423 y(double)f (position[3];)140 2473 y(double)g(velocity[3];)228 2523 y(})g(particle[1000]) 158 2604 y Fo(and)d(one)g(w)o(an)o(ts)h(to)f(transfer)h(in)f(one)g(message)g (the)h(n)o(um)o(b)q(er)e(of)h(particles)g(of)g(t)o(yp)q(e)h(1,)f(the)h(n)o (um)o(b)q(er)f(of)75 2654 y(particles)g(of)g(t)o(yp)q(e)g(2,)g(and)g(the)g(p) q(ositions)g(of)f(all)g(the)h(particles)h(of)e(these)i(t)o(w)o(o)f(t)o(yp)q (es.)31 b(The)18 b(follo)o(wing)d(co)q(de)75 2704 y(fragmen)o(t)d(creates)k (a)e(suitable)f(bu\013er)i(descriptor.)p eop %%Page: 11 12 bop 75 -100 a Fg(1.5.)31 b(COMMUNICA)m(TION)14 b(BUFFERS)1070 b Fo(11)75 45 y Fi(int)21 b(count1)g(=)g(0;)75 95 y(int)g(count2)g(=)g(0;)75 145 y(...)75 195 y(mpi_create_buffer)o(\()e(&handle,)h(MPI_PERSISTENT\);)75 244 y(mpi_append_contig)o(uous\()e(&handle,)i(&count1,)h(MPI_INT\);)75 294 y(mpi_append_contig)o(uous\()d(&handle,)i(&count2,)h(MPI_INT\);)75 344 y(for)g(\(i=0;)g(i<1000;)f(i++\))h({)140 394 y(if)h(\(particle[i].typ)o (e)d(==)i(1\))h({)206 444 y(count1++;)206 493 y(mpi_append_conti)o(guous)o (\()d(&handle,)h(&particle[i].posit)o(ion,)e(3,)k(MPI_DOUBLE\);)206 543 y(})140 593 y(elseif)f(\(particle[i].type)d(==)k(2\))f({)206 643 y(count2++;)206 693 y(mpi_append_conti)o(guous)o(\()e(&handle,)h (&particle[i].posit)o(ion,)e(3,)k(MPI_DOUBLE\);)206 742 y(})140 792 y(})75 842 y(mpi_commit_buffer)o(\(hand)o(le\))75 892 y(...)158 980 y Fo(The)14 b(bu\013er)h(descriptor)f(handle)g(can)g(no)o(w)f(b)q(e)h (used)g(to)g(send)g(\(rep)q(eatedly\))h(the)f(particle)g(coun)o(ts)g(for)f(t) o(yp)q(e)75 1030 y(1)g(and)f(2,)h(and)g(the)g(p)q(ositions)g(of)f(the)i (particles)f(with)g(these)h(t)o(w)o(o)f(t)o(yp)q(es,)g(as)g(long)f(as)h(the)h (t)o(yp)q(es)g(of)e(the)h(particles)75 1079 y(is)h(not)g(mo)q(di\014ed.)158 1129 y(Supp)q(ose)i(the)f(incoming)d(data)i(is)h(to)f(b)q(e)h(stored)h(at)e (the)h(receiv)o(er)h(in)o(to)e(the)h(t)o(w)o(o)g(coun)o(t)f(v)n(ariables)g Fi(count1)75 1179 y Fo(and)h Fi(count2)f Fo(and)h(in)o(to)f(the)i Fi(position)d Fo(comp)q(onen)o(ts)i(of)f(the)i(initial)d(en)o(tries)j(of)f (the)g(arra)o(y)g(of)f(particles.)23 b(The)75 1229 y(follo)o(wing)10 b(co)q(de)i(generates)i(a)e(suitable)g(bu\013er)h(descriptor)h(for)d(the)i (receiv)o(e)g(op)q(eration)f(\(w)o(e)h(assume)f(4)f(b)o(ytes)i(for)75 1279 y(in)o(teger)h(and)g(8)g(b)o(ytes)g(for)g(double\).)75 1367 y Fi(...)75 1416 y(mpi_create_buffer)o(\()19 b(&handle,)h (MPI_PERSISTENT\);)75 1466 y(mpi_append_contig)o(uous\()e(&handle,)i (&count1,)h(MPI_INT\);)75 1516 y(mpi_append_contig)o(uous\()d(&handle,)i (&count2,)h(MPI_INT\);)75 1566 y(mpi_append_hvec\()d(&handle,)j (&particle[0].pos)o(itio)o(n,)e(1000,)i(52,)g(3,)g(MPI_DOUBLE\);)75 1616 y(mpi_commit_buffer)o(\()e(&handle\))75 1665 y(...)158 1753 y Fo(A)c(bu\013er)g(handle)f(can)h(b)q(e)g(used)g(for)f(comm)o (unication,)d(ev)o(en)k(if)f(it)g(is)g(not)g(asso)q(ciated)i(with)e(an)o(y)g (v)n(ariables)75 1803 y(\(i.e.,)e(ev)o(en)i(if)e(it)h(w)o(as)g(not)g(set)h(b) o(y)f(an)o(y)g Fi(MPI)p 761 1803 14 2 v 15 w(APPEND)p 908 1803 V 15 w(xxx)f Fo(call\).)17 b(Suc)o(h)d(a)f(handle)g(is)g(asso)q(ciated)h (with)f(an)g(empt)o(y)75 1853 y(bu\013er,)i(and)f(a)g(message)g(created)i (from)c(it)i(con)o(tains)g(no)g(data.)19 b(The)c(handle)f(still)f(need)i(b)q (e)g(committed)d(b)q(efore)75 1903 y(it)i(is)f(used.)158 1953 y(The)h(same)e(en)o(try)i(ma)o(y)d(app)q(ear)i(more)f(than)i(once)f(in)g(a)g (bu\013er)h(comp)q(onen)o(t,)e(and)h(distinct)g(bu\013er)h(comp)q(o-)75 2002 y(nen)o(ts)i(ma)o(y)d(o)o(v)o(erlap.)21 b(If)15 b(a)g(bu\013er)h (descriptor)g(with)f(replicated)h(en)o(tries)g(is)f(used)h(to)f(receiv)o(e)h (a)f(message,)f(then)75 2052 y(the)g(\014nal)g(v)n(alue)f(of)g(an)h(en)o(try) g(that)g(o)q(ccurs)i(more)d(than)g(once)i(in)e(the)i(bu\013er)g(descriptor)g (is)f(unde\014ned.)158 2180 y Fm(Discussion:)158 2226 y Fl(It)f(is)g(not)h (clear)g(to)f(me)g(that)g(o)o(v)o(erlaps)i(ha)o(v)o(e)e(an)o(y)h(useful)g (purp)q(ose,)g(and)g(they)g(ma)o(y)f(complicate)i(implemen)o(tation.)75 2272 y(But)e(w)o(e)g(had)g(an)h(o)o(v)o(erwhelming)h(v)o(ote)e(in)h(fa)o(v)o (or)f(of)g(this)h(feature.)75 2470 y Fk(1.5.3)55 b(T)n(yp)r(e)18 b(Matc)n(hing)i(and)f(Data)g(Con)n(v)n(ersion)75 2547 y Fh(T)o(yp)q(e)d(matc) o(hing)75 2623 y Fo(One)f(can)f(think)f(of)h(message)f(transmission)g(as)h (consisting)g(of)f(three)i(phases:)126 2704 y(1.)20 b(Data)13 b(is)h(pulled)f(out)h(of)f(the)i(send)g(bu\013er)f(and)g(a)g(message)f(is)h (assem)o(bled)p eop %%Page: 12 13 bop 75 -100 a Fo(12)712 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m(TION)126 45 y Fo(2.)20 b(A)14 b(message)f(is)h (transferred)i(from)c(sender)j(to)f(receiv)o(er)126 127 y(3.)20 b(Data)13 b(is)h(pulled)f(from)f(the)j(incoming)c(message)j(and)g(disassem)o (bled)f(in)o(to)g(the)i(receiv)o(e)g(bu\013er)158 209 y(T)o(yp)q(e)c(matc)o (hing)e(has)i(to)f(b)q(e)h(observ)o(ed)h(at)f(eac)o(h)g(of)f(these)i(three)g (phases:)17 b(The)11 b(t)o(yp)q(e)g(of)f(eac)o(h)i(v)n(ariable)d(in)h(the)75 259 y(sender)15 b(bu\013er)g(has)f(to)g(matc)o(h)f(the)h(t)o(yp)q(e)g(of)g (the)g(corresp)q(onding)h(en)o(try)f(in)g(the)g(bu\013er)h(descriptor;)g(the) f(t)o(yp)q(e)h(of)75 309 y(an)e(en)o(try)g(in)f(the)h(bu\013er)h(descriptor)g (used)g(to)e(send)i(a)f(message)f(has)h(to)f(matc)o(h)g(the)h(t)o(yp)q(e)h (of)e(the)h(corresp)q(onding)75 358 y(en)o(try)f(in)e(the)i(bu\013er)g (descriptor)g(used)g(to)f(receiv)o(e)h(the)g(message;)f(and)g(the)h(t)o(yp)q (e)f(of)g(eac)o(h)g(v)n(ariable)f(in)h(the)h(receiv)o(e)75 408 y(bu\013er)i(has)f(to)g(matc)o(h)e(the)j(corresp)q(onding)f(en)o(try)h (in)e(the)i(bu\013er)g(descriptor)g(for)e(this)h(bu\013er.)19 b(A)13 b(program)e(that)75 458 y(fails)i(to)g(observ)o(e)i(these)h(three)f (rules)f(is)g(erroneous.)158 508 y(T)m(o)g(de\014ne)h(t)o(yp)q(e)g(matc)o (hing)d(more)h(precisely)m(,)i(w)o(e)f(need)h(to)g(deal)f(with)g(t)o(w)o(o)f (issues:)20 b(matc)o(hing)13 b(of)g(t)o(yp)q(es)j(of)75 558 y(the)h(host)g(language)f(with)g(bu\013er)h(descriptor)h(t)o(yp)q(es;)g(and)f (matc)o(hing)d(of)i(bu\013er)i(descriptor)g(t)o(yp)q(es)f(at)f(sender)75 607 y(and)c(receiv)o(er.)18 b(W)m(e)12 b(restrict)h(ourselv)o(es)g(to)f(the)g (case)h(where)g(the)f(sending)g(and)g(receiving)g(programs)e(are)i(written)75 657 y(in)h(the)i(same)e(language.)158 707 y(The)21 b(t)o(yp)q(e)h(of)e(an)g (en)o(try)i(in)e(the)h(sender)i(bu\013er)f(descriptor)g(matc)o(hes)e(the)i(t) o(yp)q(e)f(of)f(an)h(en)o(try)g(in)f(the)75 757 y(receiv)o(er)13 b(bu\013er)g(descriptor)g(if)e(they)h(ha)o(v)o(e)g(iden)o(tical)f(names:)16 b Fi(MPI)p 1131 757 14 2 v 15 w(INT)11 b Fo(matc)o(hes)h Fi(MPI)p 1448 757 V 15 w(INT)p Fo(,)e Fi(MPI)p 1617 757 V 15 w(REAL)h Fo(matc)o(hes)75 807 y Fi(MPI)p 144 807 V 15 w(REAL)p Fo(,)i(and)g(so)h(on.) 158 857 y(The)i(t)o(yp)q(e)g(of)e(a)h(v)n(ariable)g(in)g(a)g(host)g(program)f (matc)o(hes)h(the)h(t)o(yp)q(e)g(of)e(the)i(en)o(try)g(in)f(a)g(bu\013er)i (descriptor)75 906 y(if)e(the)i(name)e(of)g(the)i(bu\013er)g(descriptor)g(en) o(try)g(t)o(yp)q(e)g(corresp)q(onds)h(to)e(the)g(basic)g(t)o(yp)q(e)h(of)f (the)g(host)h(program)75 956 y(v)n(ariable:)k(an)15 b(en)o(try)i(with)e(t)o (yp)q(e)h(name)f Fi(MPI)p 794 956 V 15 w(INT)g Fo(matc)o(hes)h(a)f(F)m (ortran)h(v)n(ariable)e(of)i(t)o(yp)q(e)g Fi(INTEGER)p Fo(,)e(an)h(en)o(try) 75 1006 y(with)10 b(t)o(yp)q(e)i(name)d Fi(MPI)p 433 1006 V 15 w(REAL)h Fo(matc)o(hes)h(a)f(F)m(ortran)h(v)n(ariable)e(of)i(t)o(yp)q(e)g Fi(REAL)p Fo(,)e(and)i(so)g(on.)17 b(There)12 b(is)e(one)h(exception)75 1056 y(to)h(this)h(last)f(rule:)18 b(An)13 b(en)o(try)g(with)f(t)o(yp)q(e)h (name)e Fi(MPI)p 917 1056 V 15 w(BYTE)h Fo(can)h(b)q(e)g(used)g(to)g(matc)o (h)e(an)o(y)h(b)o(yte)h(of)f(storage)h(\(on)f(a)75 1106 y(b)o (yte-addressable)j(mac)o(hine\),)d(irresp)q(ectiv)o(e)j(of)e(the)g(datat)o (yp)q(e)h(of)f(the)h(v)n(ariable)e(that)i(con)o(tains)f(this)g(b)o(yte.)19 b(The)75 1155 y(v)n(alue)13 b(of)h(the)g(message)g(en)o(try)g(will)f(b)q(e)h (the)g(binary)g(v)n(alue)f(of)g(the)i(corresp)q(onding)g(b)o(yte)f(in)f (memory)m(.)158 1205 y(W)m(e)h(th)o(us)g(ha)o(v)o(e)g(t)o(w)o(o)f(cases:)137 1287 y Fe(\017)21 b Fo(Comm)o(uni)o(cation)6 b(of)j(t)o(yp)q(ed)h(v)n(alues)f (\(e.g.,)g(with)g(datat)o(yp)q(e)g(di\013eren)o(t)h(from)e Fi(MPI)p 1430 1287 V 15 w(BYTE)p Fo(,)g(where)i(the)g(datat)o(yp)q(es)179 1337 y(of)k(the)h(corresp)q(onding)g(en)o(tries)g(in)f(the)h(sender)h (program,)d(in)h(the)h(sender)h(bu\013er)f(descriptor,)h(in)d(the)i(re-)179 1386 y(ceiv)o(er)g(bu\013er)f(descriptor)i(and)d(in)h(the)g(receiv)o(er)i (program)c(should)i(all)e(matc)o(h.)137 1469 y Fe(\017)21 b Fo(Comm)o(uni)o(cation)13 b(of)j(un)o(t)o(yp)q(ed)h(v)n(alues)f(\(e.g.,)g(of) g(datat)o(yp)q(e)g Fi(MPI)p 1224 1469 V 15 w(BYTE)p Fo(,)f(where)j(the)f(the) g(corresp)q(onding)179 1518 y(en)o(tries)e(in)e(the)i(sender)g(and)f(the)g (receiv)o(er)h(bu\013er)g(descriptors)h(ha)o(v)o(e)d(b)q(oth)h(datat)o(yp)q (e)g Fi(MPI)p 1631 1518 V 16 w(BYTE)p Fo(.)e(In)i(this)179 1568 y(case,)k(there)g(are)f(no)g(requiremen)o(ts)g(on)g(the)g(datat)o(yp)q (es)h(of)e(the)i(corresp)q(onding)f(en)o(tries)h(in)f(the)g(sender)179 1618 y(and)c(the)i(receiv)o(er)g(programs,)e(nor)h(is)f(it)h(required)h(that) f(they)g(b)q(e)g(the)h(same.)75 1734 y Fk(1.5.4)55 b(Data)19 b(con)n(v)n(ersion)75 1811 y Fo(W)m(e)13 b(use)i(the)g(follo)o(wing)c (terminology:)75 1892 y Fh(t)o(yp)q(e)k(con)o(v)o(ersion)j Fo(c)o(hanges)d(the)f(datat)o(yp)q(e)g(of)f(a)h(v)n(alue,)f(e.g.)18 b(b)o(y)13 b(rounding)h(a)f Fi(REAL)g Fo(to)h(an)g Fi(INTEGER)p Fo(.)75 1974 y Fh(represen)o(tati)o(on)e(con)o(v)o(ersion)18 b Fo(c)o(hanges)e(the)g(binary)f(represen)o(tation)i(of)e(a)g(v)n(alue,)f (e.g.)23 b(from)13 b(Hex)j(\015oating)179 2024 y(p)q(oin)o(t)d(to)h(IEEE)g (\015oating)f(p)q(oin)o(t.)158 2106 y(Represen)o(tation)20 b(con)o(v)o(ersion)f(is)g(needed)h(when)g(data)e(is)h(mo)o(v)o(ed)e(across)j (mac)o(hines)e(that)h(use)h(di\013eren)o(t)75 2156 y(binary)9 b(enco)q(dings)h(for)f(the)h(same)f(datat)o(yp)q(e.)16 b(Represen)o(tation)11 b(con)o(v)o(ersion)e(preserv)o(es,)k(to)c(the)h(greatest)h(p)q(ossible)75 2205 y(extend,)h(the)e Fh(v)m(alue)g Fo(represen)o(ted.)20 b(Ho)o(w)o(ev)o(er,)11 b(rounding)e(errors)j(and)e(o)o(v)o(er\015o)o(w)g(and) g(under\015o)o(w)h(exceptions)g(ma)o(y)75 2255 y(o)q(ccur)17 b(during)e(\015oating)f(p)q(oin)o(t)h(con)o(v)o(ersions;)h(con)o(v)o(ersion)g (of)f(in)o(tegers)h(ma)o(y)e(also)g(lead)h(to)h(exceptions)g(when)g(a)75 2305 y(v)n(alue)d(that)h(can)g(represen)o(ted)j(in)c(one)h(system)g(cannot)g (b)q(e)h(represen)o(ted)h(in)e(the)g(other)h(system.)i(MPI)d(do)q(es)h(not)75 2355 y(sp)q(ecify)f(rules)h(for)e(represen)o(tation)j(con)o(v)o(ersion.)158 2405 y(The)j(t)o(yp)q(e)f(matc)o(hing)e(rules)j(imply)d(that)i(MPI)g(comm)o (unication)d(nev)o(er)k(en)o(tails)e(t)o(yp)q(e)i(con)o(v)o(ersion.)31 b(On)75 2455 y(the)15 b(other)g(hand,)f(MPI)g(requires)i(that)e(a)g(represen) o(tation)i(con)o(v)o(ersion)e(b)q(e)h(p)q(erformed)f(when)h(a)f(t)o(yp)q(ed)g (v)n(alue)g(is)75 2504 y(transferred)g(across)g(en)o(vironmen)o(ts)d(that)i (use)g(di\013eren)o(t)g(represen)o(tations)i(for)d(the)h(datat)o(yp)q(e)g(of) e(this)i(v)n(alue.)k(An)75 2554 y(exception)f(o)q(ccurring)g(during)f (represen)o(tation)i(con)o(v)o(ersion)f(results)g(in)f(a)h(failure)e(of)h (the)h(comm)o(unicatio)o(n;)d(an)75 2604 y(error)i(o)q(ccurs)g(either)g(in)e (the)i(send)g(op)q(eration,)e(or)h(the)g(receiv)o(e)h(op)q(eration,)f(or)g(b) q(oth.)158 2654 y(If)g(a)f(v)n(alue)h(sen)o(t)h(in)e(a)h(message)g(is)f(un)o (t)o(yp)q(ed)i(\(i.e.,)e(of)g(t)o(yp)q(e)i Fi(MPI)p 1180 2654 V 15 w(BYTE)p Fo(\),)e(then)h(the)h(binary)e(represen)o(tation)75 2704 y(of)h(the)g(b)o(yte)h(stored)g(at)f(the)h(receiv)o(er)h(is)e(iden)o (tical)f(to)h(the)h(binary)f(represen)o(tation)i(of)d(the)i(b)o(yte)g(loaded) e(at)h(the)p eop %%Page: 13 14 bop 75 -100 a Fg(1.5.)31 b(COMMUNICA)m(TION)14 b(BUFFERS)1070 b Fo(13)75 45 y(sender.)24 b(This)16 b(holds)f(true,)h(whether)h(sender)g (and)e(receiv)o(er)i(run)f(in)f(the)h(same)f(or)g(in)g(distinct)h(en)o (vironmen)o(ts;)75 95 y(no)e(represen)o(tation)h(con)o(v)o(ersion)f(is)g (required.)158 145 y(Note)f(that)g(no)g(con)o(v)o(ersions)g(ev)o(er)h(o)q (ccur)g(when)g(an)e(MPI)h(program)f(executes)j(in)d(a)h(homogeneous)f (system,)75 195 y(where)g(all)e(pro)q(cesses)j(run)e(in)g(the)g(same)f(en)o (vironmen)o(t.)16 b(Also)11 b(note)g(the)g(di\013eren)o(t)h(b)q(eha)o(vior)f (of)f Fi(MPI)p 1654 195 14 2 v 15 w(BYTE)g Fo(and)h(of)75 244 y Fi(MPI)p 144 244 V 15 w(CHAR)p Fo(.)g(A)h(bu\013er)h(descriptor)h(en)o(try) f(with)e(datat)o(yp)q(e)i(of)e Fi(MPI)p 1106 244 V 16 w(CHAR)g Fo(can)h(only)g(matc)o(h)f(a)g(F)m(ortran)h(v)n(ariable)g(of)75 294 y(t)o(yp)q(e)i Fi(CHAR)p Fo(;)e(and)h(represen)o(tation)i(con)o(v)o (ersion)f(ma)o(y)d(o)q(ccur)j(when)g(v)n(alues)f(of)g(t)o(yp)q(e)h Fi(MPI)p 1473 294 V 15 w(CHAR)e Fo(are)i(transferred.,)75 344 y(e.g.,)f(from)f(an)h(EBCDIC)i(enco)q(ding)f(to)f(an)h(ASCI)q(I)g(enco)q (ding.)158 394 y(Consider)g(the)h(follo)o(wing)c(examples.)75 481 y Fi(!)43 b(FIRST)21 b(EXAMPLE)75 531 y(INTEGER)f(A\(100\))75 581 y(...)75 630 y(CALL)h(MPI_CREATE_BUFFER)o(\(BH,)d(MPI_PERSISTENT\))75 680 y(CALL)j(MPI_APPEND_CONTIG)o(UOUS)d(\(BH,)j(A\(1\),)g(1,)h(MPI_CHAR\))158 767 y Fo(This)15 b(program)f(is)i(erroneous,)g(since)h(the)f(t)o(yp)q(e)g(of) e(the)j(bu\013er)f(descriptor)h(do)q(es)f(not)f(matc)o(h)g(the)h(t)o(yp)q(e)g (of)75 817 y(the)g(v)n(ariable)f(in)g(the)i(bu\013er.)25 b(The)16 b(b)q(eha)o(vior)f(or)h(this)g(program)e(is)i(unde\014ned.)25 b(A)16 b(desirable)g(b)q(eha)o(vior)g(is)f(for)75 867 y(that)f(error)h(to)e (b)q(e)i(detected)h(\(at)e(compile)e(time)h(or)h(run)g(time\))e(and)i(result) h(in)e(an)h(error)h(condition.)75 954 y Fi(!)43 b(SECOND)21 b(EXAMPLE)75 1004 y(INTEGER)f(A\(100\))75 1054 y(...)75 1104 y(CALL)h(MPI_CREATE_BUFFER)o(\(BH,)d(MPI_PERSISTENT\))75 1153 y(CALL)j(MPI_APPEND_CONTIG)o(UOUS)d(\(BH,)j(A\(1\),)g(1,)h(MPI_BYTE\))158 1241 y Fo(This)11 b(program)f(is)h(correct.)19 b(The)12 b(b)o(yte)g(that)f (will)f(b)q(e)i(transmitted)f(in)g(a)g(message)g(created)i(with)e(this)h (bu\013er)75 1290 y(descriptor)h(is)f(the)g(\014rst)h(b)o(yte)f(in)g(the)g (storage)h(o)q(ccupied)f(b)o(y)g Fi(A\(1\))p Fo(.)17 b(The)12 b(v)n(alue)f(of)h(this)f(b)o(yte)i(is)e(implemen)o(tation)75 1340 y(dep)q(enden)o(t,)17 b(and)e(will)f(v)n(ary)h(according)g(to)g(the)h(n) o(um)o(b)q(er)f(of)f(b)o(ytes)i(used)h(to)e(store)h(F)m(ortran)f(v)n (ariables)g(of)f(t)o(yp)q(e)75 1390 y Fi(INTEGER)e Fo(and)i(the)h(b)o(yte)f (ordering)g(on)f(the)i(host)f(mac)o(hine.)75 1477 y Fi(!)43 b(THIRD)21 b(EXAMPLE)75 1527 y(INTEGER)f(A\(100\))75 1577 y(...)75 1627 y(CALL)h(MPI_CREATE_BUFFER)o(\(BH1,)d(MPI_PERSISTENT\))75 1677 y(CALL)j(MPI_APPEND_CONTIG)o(UOUS)d(\(BH1,)j(A\(1\),)g(1,)g (MPI_INTEGER\))75 1726 y(...)75 1776 y(CALL)g(MPI_CREATE_BUFFER)o(\(BH2,)d (MPI_PERSISTENT\))75 1826 y(CALL)j(MPI_APPEND_CONTIG)o(UOUS)d(\(BH2,)j (A\(1\),)g(4,)g(MPI_BYTE\))158 1913 y Fo(Consider)14 b(the)h(follo)o(wing)c (three)k(cases:)126 1993 y(1.)20 b(A)10 b(message)g(is)g(sen)o(t)h(using)e (bu\013er)j(descriptor)f(handle)f Fi(BH1)f Fo(and)h(receiv)o(ed)i(using)e (bu\013er)h(descriptor)g(handle)179 2043 y Fi(BH2)p Fo(.)19 b(The)d(program)d(is)h(erroneous,)i(since)f(the)h(sender)g(and)e(receiv)o(er) i(bu\013er)g(descriptors)g(don't)f(matc)o(h.)179 2093 y(The)g(b)q(eha)o(vior) f(of)g(the)h(program)d(is)j(unde\014ned,)g(and)f(a)g(p)q(ossible)h (\(desirable)g(but)g(unlik)o(ely\))e(b)q(eha)o(vior)h(is)179 2142 y(for)f(an)h(error)h(condition)e(to)h(o)q(ccur.)126 2224 y(2.)20 b(Both)d(sender)h(and)f(receiv)o(er)i(use)e(bu\013er)h(descriptor)g (handle)f Fi(BH1)p Fo(.)26 b(The)18 b(program)d(is)i(correct,)h(and)f(an)179 2274 y(in)o(teger)12 b(v)n(alue)f(is)g(transmitted)g(from)f(sender)j(to)f (receiv)o(er.)19 b(The)12 b(v)n(alue)f(assigned)h(to)f Fi(A\(1\))g Fo(at)g(the)h(receiv)o(er)179 2323 y(is)h(iden)o(tical)f(to)h(the)g(v)n(alue) g(of)f Fi(A\(1\))g Fo(at)h(the)h(sender,)g(but)f(the)h(binary)f(represen)o (tation)h(of)e(this)i(v)n(alue)e(ma)o(y)179 2373 y(b)q(e)i(di\013eren)o(t,)h (if)e(sender)i(and)f(receiv)o(er)h(run)g(in)e(di\013eren)o(t)i(en)o(vironmen) o(ts.)126 2455 y(3.)20 b(Both)c(sender)h(and)e(receiv)o(er)i(use)g(bu\013er)f (descriptor)h(handle)f Fi(BH2)p Fo(.)22 b(The)16 b(program)e(is)h(correct.)25 b(Assume)179 2504 y(that)14 b(b)q(oth)h(sender)h(and)e(receiv)o(er)i(use)f (four)f(b)o(ytes)h(to)f(store)i(in)o(tegers.)k(Then)15 b(the)g(v)n(alue)f (stored)h(in)f Fi(A\(1\))179 2554 y Fo(at)i(the)h(receiv)o(er)h(has)e(the)h (same)f(binary)g(represen)o(tation)i(as)e(the)h(v)n(alue)f(of)f Fi(A\(1)h Fo(at)g(the)h(sender.)27 b(If)16 b(the)179 2604 y(same)c(b)o(yte)h (ordering)g(and)g(the)h(same)e(enco)q(ding)h(of)g(in)o(teger)g(v)n(alues)g (is)g(used)g(at)g(b)q(oth)g(ends)h(then)g Fi(A\(1\))e Fo(at)179 2654 y(the)j(receiv)o(er)h(is)f(set)g(to)g(the)g(v)n(alue)f(of)g Fi(A\(1\))g Fo(at)h(the)g(sender;)h(otherwise)g(the)f(same)f(32)g(bits)h(ma)o (y)e(enco)q(de)179 2704 y(di\013eren)o(t)i(v)n(alues.)p eop %%Page: 14 15 bop 75 -100 a Fo(14)712 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m(TION)158 45 y Fm(Implemen)o(tation)f(note:)158 94 y Fl(The)d(curren)o(t)g(de\014nition)j(do)q(es)d(not)h(require)g(messages) g(to)e(carry)i(t)o(yp)q(e)f(information.)18 b(A)9 b(message)i(can)f(b)q(e)h (comp)q(osed)75 140 y(and)g(sen)o(t)f(using)i(only)f(the)f(information)i(in)f (the)g(sender)g(bu\013er)g(descriptor,)h(and)e(can)h(b)q(e)f(receiv)o(ed)i (and)e(stored)h(using)h(only)75 186 y(the)i(information)i(in)e(the)g(receiv)o (er)h(bu\013er)g(descriptor.)21 b(If)13 b(messages)h(are)g(sen)o(t)g(b)q(et)o (w)o(een)g(di\013eren)o(t)i(mac)o(hines)f(then)f(one)75 231 y(can)e(either)h(use)g(a)f(\\univ)o(ersal")j(data)d(enco)q(ding)i(for)e (messages,)h(use)f(kno)o(wledge)i(of)d(the)i(receiv)o(er)g(en)o(vironmen)o(t) h(in)e(order)75 277 y(to)i(con)o(v)o(ert)h(data)f(at)g(the)h(sender,)f(or)h (use)f(kno)o(wledge)i(of)e(the)g(sender)h(en)o(vironmen)o(t)h(in)f(order)f (to)g(con)o(v)o(ert)h(data)g(at)f(the)75 323 y(receiv)o(er.)k(In)c(either)g (case)f(the)h(lo)q(cal)g(bu\013er)g(descriptor)h(can)f(b)q(e)f(used)h(to)f (deriv)o(e)i(the)e(t)o(yp)q(es)h(of)f(the)g(v)n(alues)h(transferred.)75 368 y(Ho)o(w)o(ev)o(er,)f(additional)k(t)o(yp)q(e)c(information)j(carried)e (b)o(y)g(messages)g(can)g(pro)o(vide)h(b)q(etter)f(error)f(detection.)19 b(It)13 b(is)h(p)q(ossible)75 414 y(to)d(add)g(alw)o(a)o(ys)h(suc)o(h)f (additional)j(t)o(yp)q(e)d(information,)i(for)e(b)q(etter)g(error)g(c)o(hec)o (king,)h(or)f(add)h(it)f(when)g(running)i(in)e(\\debug")75 460 y(mo)q(de,)i(where)g(some)g(p)q(erformance)h(is)g(sacri\014ced)h(for)d (added)i(error)g(c)o(hec)o(king,)g(etc.)158 675 y Fm(Discussion:)158 724 y Fl(There)h(is)g(no)g(agreemen)o(t)h(y)o(et)f(on)g(the)g(need)g(for)g(a) g(de\014nition)i(and,)f(if)f(needed,)h(on)f(the)g(righ)o(t)g(de\014nition)j (of)c(t)o(yp)q(e)75 770 y(matc)o(hing)k(and)e(con)o(v)o(ersion)j(b)q(et)o(w)o (een)d(programs)h(written)g(in)g(di\013eren)o(t)g(languages.)29 b(The)16 b(natural)h(in)o(terpretation)i(of)75 815 y(\\send)11 b(from)e(C)g(and)i(receiv)o(e)f(in)h(F)m(ortran")f(is)g(either)h(\\send)g (from)e(C,)g(receiv)o(e)i(in)f(C)f(and)i(con)o(v)o(ert)f(to)g(F)m(ortran",)g (or)g(\\con)o(v)o(ert)75 861 y(to)i(F)m(ortran,)h(send)g(from)f(F)m(ortran)g (and)h(receiv)o(e)g(in)g(F)m(ortran".)18 b(Ho)o(w)o(ev)o(er,)11 b(the)i(corresp)q(ondence)h(b)q(et)o(w)o(een)f(F)m(ortran)f(t)o(yp)q(es)75 907 y(and)g(C)f(t)o(yp)q(es)g(is)h(implemen)o(tation)j(dep)q(enden)o(t.)j(Th) o(us,)11 b(the)h(diagram)g(sho)o(wn)g(b)q(elo)o(w)g(is)g(not)f(comm)o(utativ) o(e,)i(and)f(the)f(t)o(w)o(o)75 952 y(in)o(terpretations)16 b(of)c(C)h(to)g(F)m(ortran)g(comm)o(unication)j(ma)o(y)d(lead)h(to)f (di\013eren)o(t)i(results.)238 999 y(mac)o(hine)g(1)285 b(mac)o(hine)15 b(2)513 1045 y(send)232 1091 y(C)e(program)118 b Fc(!)g Fl(C)13 b(program)313 1182 y Fc(#)436 b(#)513 1228 y Fl(send)183 1273 y(F)m(ortran)13 b(program)69 b Fc(!)g Fl(F)m(ortran)13 b(program)158 1321 y(A)h(go)q(o)q(d)h(solution)h(w)o(ould)f(mak)o(e)f(this)h(diagram)h (comm)o(utativ)o(e.)21 b(Otherwise,)15 b(one)g(needs)g(to)f(pic)o(k)h(one)f (of)g(the)g(t)o(w)o(o)75 1366 y(paths)k(as)f(the)g(canonical)i(de\014nition)h (of)d(in)o(terlanguage)j(comm)o(unication.)31 b(If)16 b(the)h(upp)q(er)h (path)g(is)f(c)o(hosen)h(\(message)75 1412 y(carries)c(sender)g(datat)o(yp)q (es\),)h(then)e(the)h(receiv)o(er)g(of)f(a)g(message)h(needs)g(to)g(kno)o(w)f (the)h(language)h(the)f(message)f(w)o(as)h(sen)o(t)75 1458 y(from;)h(if)g(the)f(lo)o(w)o(er)h(path)h(is)f(c)o(hosen,)g(\(message)h (carries)f(receiv)o(er)h(datat)o(yp)q(es\),)g(then)f(the)f(sender)i(needs)f (to)g(kno)o(w)g(the)75 1503 y(language)g(the)e(message)h(is)f(receiv)o(ed)i (in)o(to.)158 1553 y(A)c(p)q(ossible)i(solution)h(\(prop)q(osed)e(b)o(y)g (Dann)o(y)g(Nessett\))f(is)h(to)f(lea)o(v)o(e)h(to)f(the)g(user)h(the)f(c)o (hoice)h(of)f(sp)q(ecifying)i(whether)75 1599 y(to)d(send)g(\\F)m(ortran)g (datat)o(yp)q(es")i(or)d(\\C)h(datat)o(yp)q(es",)h(in)g(the)f(ab)q(o)o(v)o(e) g(situation.)18 b(One)10 b(w)o(ould)h(still)g(require)g(exact)f(matc)o(hing) 75 1644 y(of)16 b(send)h(bu\013er)g(descriptors)h(and)f(receiv)o(e)g (bu\013er)g(descriptors.)28 b(Ho)o(w)o(ev)o(er,)17 b(a)f(F)m(ortran)h (program)g(could)g(use)g(a)f(bu\013er)75 1690 y(descriptor)f(with)e(\\C)g (datat)o(yp)q(es",)h(and)g(vice)g(v)o(ersa.)j(The)c(standard)h(w)o(ould)g(sp) q(ecify)h(whic)o(h)e(matc)o(hing)i(of)e(F)m(ortran)g(and)75 1735 y(C)i(datat)o(yp)q(es)j(are)e(legal.)27 b(F)m(or)16 b(example,)i(it)e(w) o(ould)h(b)q(e)f(correct)g(to)g(send)h(a)f(F)m(ortran)g(v)n(ariable)i(of)e(t) o(yp)q(e)g Ff(REAL)e Fl(using)k(a)75 1781 y(bu\013er)c(descriptor)g(of)f(t)o (yp)q(e)g Ff(MPI)p 556 1781 12 2 v 13 w(FLOAT)p Fl(.)d(Data)j(con)o(v)o (ersion)i(migh)o(t)f(b)q(e)f(required)h(if)f(C)f(\015oat)i(n)o(um)o(b)q(ers)f (and)h(F)m(ortran)f(real)75 1827 y(n)o(um)o(b)q(ers)h(do)f(not)h(use)f(the)g (same)g(presen)o(tation.)158 1876 y(Another)j(\(non\)solution)i(is)f(to)e (ignore)i(the)e(issue)i(and)f(let)g(the)g(b)q(eha)o(vior)h(of)e(C)g(to)h(F)m (ortran)f(comm)o(unication)k(b)q(e)75 1922 y(unde\014ned.)32 b(The)18 b(user)g(can)g(alw)o(a)o(ys)g(matc)o(h)g(language)h(at)f(sending)h (and)f(receiving)i(end)e(if)f(he)h(or)g(she)g(is)g(willing)i(to)75 1968 y(struggle)14 b(with)g(the)f(issue)h(of)f(calling)i(MPI)e(C)g(functions) i(from)d(F)m(ortran)i(or)f(vice-v)o(ersa.)75 2207 y Fn(1.6)70 b(Receiv)n(e)20 b(Criteria)75 2305 y Fo(The)e(selection)g(of)f(a)g(message)g (b)o(y)g(a)g(receiv)o(e)i(op)q(eration)e(is)h(done)f(uniquely)g(according)h (to)f(the)h(v)n(alue)f(of)f(the)75 2355 y(message)c(en)o(v)o(elop)q(e.)18 b(The)13 b(receiv)o(e)h(op)q(eration)e(sp)q(eci\014es)i(an)e Fh(en)o(v)o(elop)q(e)g(pattern)p Fo(;)f(a)h(message)g(can)h(b)q(e)g(receiv)o (ed)75 2405 y(b)o(y)20 b(that)g(receiv)o(e)h(op)q(eration)f(only)f(if)g(its)h (en)o(v)o(elop)q(e)h(matc)o(hes)e(that)h(pattern.)38 b(A)20 b(pattern)g(sp)q(eci\014es)i(v)n(alues)75 2455 y(for)c(the)h Fi(source)p Fo(,)f Fi(tag)g Fo(and)h Fi(context)e Fo(\014elds)i(of)f(the)h (message)f(en)o(v)o(elop)q(e.)32 b(In)19 b(addition,)f(the)h(v)n(alue)f(for)g (the)75 2504 y Fi(dest)g Fo(\014eld)g(is)g(set,)i(implicitly)m(,)c(to)i(b)q (e)h(equal)f(to)h(the)g(receiving)f(pro)q(cess)j(id.)31 b(The)19 b(receiv)o(er)h(ma)o(y)c(sp)q(ecify)j(a)75 2554 y Fi(DONTCARE)11 b Fo(v)n(alue)i(for)f Fi(source)p Fo(,)g(or)h Fi(tag)p Fo(,)f(indicating)g (that)h(an)o(y)g(source)h(and/or)f(tag)f(are)i(acceptable.)19 b(It)13 b(cannot)75 2604 y(sp)q(ecify)k(a)f(DONTCARE)h(v)n(alue)e(for)h Fi(context)f Fo(or)i Fi(dest)p Fo(.)25 b(Th)o(us,)17 b(a)f(message)g(can)g(b) q(e)i(receiv)o(ed)f(b)o(y)g(a)f(receiv)o(e)75 2654 y(op)q(eration)e(only)f (if)g(it)g(is)h(addressed)h(to)f(the)g(receiving)g(task,)g(has)f(a)h(matc)o (hing)e(con)o(text,)i(has)g(matc)o(hing)e(source)75 2704 y(unless)18 b(source=DONTCARE)i(in)d(the)h(pattern,)h(and)e(has)h(a)f(matc)o(hing)f(tag)h (unless)i(tag=DONTCARE)e(in)p eop %%Page: 15 16 bop 75 -100 a Fg(1.7.)31 b(COMMUNICA)m(TION)14 b(MODE)1136 b Fo(15)75 45 y(the)18 b(pattern.)28 b(DONTCARE)16 b(v)n(alues)h(for)g (source)h(or)f(tag)f(are)i(sp)q(eci\014ed)g(via)e(a)h(named)f(constan)o(t)h (for)g(source)75 95 y(don)o(tcare)e(and)e(a)h(named)f(constan)o(t)h(for)g (tag)f(don)o(tcare.)158 145 y(The)i(length)f(of)g(the)g(receiv)o(ed)i (message)e(m)o(ust)f(b)q(e)i(less)g(then)g(or)f(equal)g(the)h(length)f(of)g (the)h(receiv)o(e)g(bu\013er.)75 195 y(I.e.,)d(all)f(incoming)f(data)i(m)o (ust)f(\014t,)i(without)f(truncation,)g(in)o(to)g(the)h(receiv)o(e)g (bu\013er.)19 b(It)12 b(is)h(erroneous)g(to)g(receiv)o(e)75 244 y(a)h(message)f(whic)o(h)h(length)g(exceed)i(the)e(receiv)o(e)h (bu\013er,)g(and)e(the)i(outcome)e(of)g(program)g(where)i(this)f(o)q(ccurs)h (is)75 294 y(undetermined.)75 432 y Fn(1.7)70 b(Comm)n(unication)20 b(Mo)r(de)75 523 y Fo(A)14 b(send)h(op)q(eration)f(can)g(o)q(ccur)h(in)e(one) h(of)f(three)j(mo)q(des:)75 606 y Fh(ST)l(AND)o(ARD)k Fo(The)14 b(send)h(ma)o(y)d(start)i(whether)i(or)d(not)h(a)g(matc)o(hing)e(receiv)o(e)j (has)f(b)q(een)h(p)q(osted.)75 690 y Fh(READ)o(Y)20 b Fo(The)15 b(send)g(ma)o(y)c(start)k(only)e(if)g(a)h(matc)o(hing)e(receiv)o(e)j(has)f(b) q(een)h(p)q(osted.)75 773 y Fh(SECURE)21 b Fo(The)12 b(send)g(op)q(eration)f (will)f(not)h(complete)g(successfully)h(un)o(til)e(it)h(is)g(guaran)o(teed)h (that)f(the)h(message)179 823 y(sen)o(t)i(will)f(b)q(e)h(receiv)o(ed.)158 906 y(The)k(start)g(and)f(completion)e(of)i(a)g Fh(standard)e Fo(send)j(do)q(es)g(not)f(dep)q(end)h(on)f(the)h(status)g(of)f(a)g(matc)o (hing)75 956 y(receiv)o(e.)158 1006 y(A)e Fh(ready)h(send)e Fo(can)h(start)g(only)f(if)g(a)h(matc)o(hing)e(receiv)o(e)j(is)f(already)f(p) q(osted;)i(otherwise)g(the)f(op)q(eration)75 1056 y(is)e(erroneous)h(and)e (its)h(outcome)f(is)g(unde\014ned.)19 b(In)13 b(some)f(systems,)g(this)h (allo)o(ws)f(the)h(remo)o(v)n(al)d(of)i(a)h(hand-shak)o(e)75 1106 y(op)q(eration)h(that)g(is)f(otherwise)i(required,)g(and)e(results)i(in) f(impro)o(v)o(ed)e(p)q(erformance.)18 b(Its)c(completion)e(do)q(es)j(not)75 1155 y(dep)q(end)g(on)f(the)g(status)h(of)e(a)h(matc)o(hing)e(receiv)o(e.)158 1205 y(A)20 b Fh(secure)h(send)d Fo(will)g(complete)h(if)g(a)g(matc)o(hing)f (receiv)o(e)j(is)e(p)q(osted,)i(and)f(the)g(receiv)o(e)h(op)q(eration)e(is)75 1255 y(committed)c(to)h(receiv)o(e)j(the)e(message)g(sen)o(t)h(b)o(y)f(the)g (secure)i(send.)28 b(\(I.e.,)17 b(the)h(send)g(and)e(receiv)o(e)j(op)q (erations)75 1305 y(ha)o(v)o(e)h(already)h(b)q(een)g(matc)o(hed,)h(the)f (receiv)o(e)h(op)q(eration)e(has)h(already)f(started)i(and)e(cannot)h(b)q(e)g (cancelled)75 1355 y(an)o(ymore.\))c(The)d(start)h(of)e(a)g(secure)j(send)f (do)q(es)g(not)f(dep)q(end)h(on)e(the)i(status)g(of)e(a)g(matc)o(hing)f (receiv)o(e.)158 1405 y(A)j(receiv)o(e)i(op)q(eration)e(can)h(o)q(ccur)g(in)f (one)h(of)f(t)o(w)o(o)g(mo)q(des:)20 b Fh(ST)l(AND)o(ARD)14 b Fo(and)h Fh(SECURE)p Fo(.)h(A)f(secure)75 1454 y(send)i(can)g(b)q(e)g(matc) o(hed)f(only)g(b)o(y)g(a)g Fh(secure)i(receiv)o(e)p Fo(;)e(a)g(program)f (where)j(a)e(secure)i(send)g(can)f(b)q(e)g(matc)o(hed)75 1504 y(b)o(y)e(a)f(standard)h(receiv)o(e)h(is)f(erroneous)h(and)f(its)f(b)q(eha)o (vior)h(is)g(unde\014ned.)22 b(The)15 b(secure)i(receiv)o(e)f(can)f(b)q(e)g (p)q(osted)75 1554 y(either)g(b)q(efore)f(or)g(after)g(the)h(matc)o(hing)d (send.)158 1604 y(A)f(standard)g(send)g(and)f(a)h(ready)f(send)i(can)f(b)q (oth)f(b)q(e)h(matc)o(hed)f(only)g(b)o(y)g(a)g Fh(standard)h(receiv)o(e)p Fo(;)e(a)i(program)75 1654 y(where)k(a)f(secure)i(receiv)o(e)g(can)e(matc)o (h)f(a)h(nonsecure)i(send)f(is)g(erroneous)g(and)f(its)g(b)q(eha)o(vior)g(is) g(unde\014ned.)21 b(The)75 1704 y(receiv)o(e)14 b(has)g(to)f(b)q(e)h(p)q (osted)g(ahead)f(of)f(the)i(send)g(for)f(a)g(ready)g(send)h(and)f(can)h(b)q (e)g(p)q(osted)g(either)g(b)q(efore)g(or)f(after)75 1753 y(the)h(send)h(is)f (p)q(osted)h(for)e(a)h(standard)g(send.)158 1882 y Fm(Discussion:)158 1932 y Fl(I)h(got)h(criticisms)h(ab)q(out)f(the)g(previous)h(notation)g (\(REGULAR\))e(and)h(the)g(curren)o(t)g(one)g(\(ST)m(AND)o(ARD\).)e(I)h(can) 75 1982 y(also)g(try)e(NORMAL,)g(DEF)l(A)o(UL)m(T,)f(...)18 b(W)m(e)c(should)h(probably)h(v)o(ote,)d(at)h(some)g(p)q(oin)o(t.)19 b(Also,)14 b(Lyndon)h(prop)q(oses)g(to)e(use)75 2031 y(SYNCHR)o(ONOUS,)f (rather)i(than)g(secure,)g(since)h(the)e(other)h(mo)q(des)g(can)g(also)h (o\013er)e(secure)i(comm)o(unication)h(with)e(the)75 2081 y(righ)o(t)g(use.) 158 2292 y Fm(Implemen)o(tation)f(note:)158 2338 y Fl(The)g(curren)o(t)h (de\014nition)i(assumes)e(it)g(is)g(the)f(user)h(resp)q(onsibil)q(it)o(y)i (to)e(prev)o(en)o(t)g(matc)o(hing)g(of)f(a)h(regular)g(send)g(with)75 2384 y(a)f(secure)h(receiv)o(e)g(and)f(vice-v)o(ersa.)18 b(Otherwise,)c (messages)g(need)f(b)q(e)g(tagged)h(as)f(standard)i(or)e(secure.)158 2429 y(Assume)i(an)g(implemen)o(tation)j(of)d(MPI)f(comm)o(unication)k(that)d (a)o(v)o(oids)h(retries.)23 b(What)15 b(proto)q(cols)i(can)e(b)q(e)g(used?)75 2475 y(Since)j(a)e(receiv)o(e)i(ma)o(y)f(ha)o(v)o(e)g(a)g(wildcard)h(source)f (\014eld,)i(the)d(\014rst)h(step)g(in)h(the)e(proto)q(col)i(m)o(ust)f(b)q(e)g (executed)h(b)o(y)f(the)75 2521 y(sender.)158 2567 y(F)m(or)c(a)g(ready)h (send,)f(this)h(\014rst)f(step)h(ma)o(y)f(in)o(v)o(olv)o(e)i(sending)g(the)e (data,)h(since)g(the)f(receiv)o(er)h(is)g(guaran)o(teed)g(to)f(ha)o(v)o(e)75 2612 y(space)h(to)f(store)g(it.)k(The)c(proto)q(col)h(is)g(send,)f(p)q (ossibly)j(follo)o(w)o(ed)e(b)o(y)g(an)f(ac)o(k-send.)158 2658 y(The)g(proto)q(col)h(for)e(secure)i(send)f(is)g(lik)o(ely)i(to)e(b)q(e)g (the)g(follo)o(wing:)19 b(The)13 b(sender)g(sends)h(a)e(request-to-send)i (message.)75 2704 y(The)f(receiv)o(er)h(stores)f(this)g(request)h(\(whic)o(h) g(con)o(tains)g(a)f(cop)o(y)g(of)g(the)g(message)g(en)o(v)o(elop)q(e\).)19 b(When)14 b(a)f(matc)o(hing)h(receiv)o(e)p eop %%Page: 16 17 bop 75 -100 a Fo(16)712 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m(TION)75 45 y Fl(is)g(p)q(osted,)h(the)f(receiv)o (er)h(sends)f(bac)o(k)h(an)f(ac)o(kno)o(wledge,)i(and)e(the)g(sender)h(no)o (w)e(sends)i(the)f(message.)20 b(The)14 b(proto)q(col)h(is)75 91 y(\(req,)e(ac)o(k-req,)g(send\),)g(p)q(ossibly)j(follo)o(w)o(ed)e(b)o(y)f (an)h(ac)o(k-send.)158 136 y(What)g(proto)q(col)h(is)f(used)g(for)f(standard) i(sends?)k(Either)14 b(the)g(second)g(proto)q(col)h(\(req,)e(ac)o(k-req,)g (send\))h(or)g(a)f(mix)h(of)75 182 y(the)f(\014rst)g(and)h(the)f(second)h (\(e.g.)j(\014rst)c(used)h(for)e(short)i(messages,)f(second)h(for)f(long)h (messages\).)158 228 y(When)h(the)g(receiv)o(er)g(do)q(es)g(not)f(wildcard)i (the)f(source)g(\014eld)g(then)g(the)f(proto)q(col)i(can)f(b)q(e)f(rev)o (ersed:)21 b(The)14 b(receiv)o(er)75 273 y(sends)h(a)f(clear-to-send)h (message,)g(and)f(the)h(sender)f(can)h(pro)q(ceed)g(with)f(the)g(short)h (proto)q(col)g(if)f(it)h(has)f(a)g(clear-to-send)75 319 y(when)f(the)g(send)h (is)g(executed.)158 365 y(What)f(do)g(w)o(e)g(gain)g(from)g(the)g(\\secure)g (receiv)o(e"?)19 b(Some)13 b(optimizations)i(for)e(regular)h(receiv)o(es)g (\(e.g.,)d(in)j(a)e(regular)75 410 y(receiv)o(e)18 b(of)f(a)g(short)g (message)h(the)f(receiv)o(er)h(kno)o(ws)g(the)f(short)g(proto)q(col)i(will)f (b)q(e)g(used,)g(and)g(need)f(not)h(c)o(hec)o(k)f(for)g(a)75 456 y(request-to-send\).)h(Q:)12 b(is)i(that)f(enough)i(justi\014cation)g (for)e(the)g(secure)g(receiv)o(e?)158 502 y(A)k(coun)o(ter)g(argumen)o(t)h (\(Leslie)g(Hart\))f(is)h(that)f(one)g(could)i(conceiv)o(e)f(of)f(an)g (implemen)o(tation)j(where)d(a)g(regular)75 547 y(receiv)o(e)12 b(reserv)o(es)g(bu\013er)g(space,)f(so)h(that)f(a)g(program)h(that)f(use)h (regular)g(receiv)o(es)g(is)g(not)f(safe,)g(ev)o(en)h(if)f(only)h(secure)g (sends)75 593 y(are)h(used.)18 b(I)12 b(doubt)i(this)g(is)f(a)g(strong)h (argumen)o(t.)k(On)13 b(the)g(other)g(hand,)h(it)f(indicates)i(that,)e(if)g (the)g(exp)q(ectation)i(is)f(that)75 639 y(programs)i(written)g(using)g(only) h(secure)e(sends)h(are)g(safe,)f(then)h(this)g(is)f(an)h(additional)i (correctness)e(condition)i(on)e(the)75 684 y(implemen)o(tation)g(that)d(need) h(b)q(e)f(made)h(explicit.)75 903 y Fn(1.8)70 b(Comm)n(unication)20 b(Ob)t(jects)75 994 y Fo(An)15 b(opaque)g(comm)o(unication)d(ob)r(ject)k (iden)o(ti\014es)g(v)n(arious)e(prop)q(erties)j(of)e(a)g(comm)o(uni)o(cation) d(op)q(eration,)j(suc)o(h)75 1043 y(as)h(the)g(bu\013er)h(descriptor)h(that)d (is)h(asso)q(ciated)h(with)e(it,)h(its)g(con)o(text,)g(the)h(tag)e(and)h (destination)g(parameters)75 1093 y(to)f(b)q(e)h(used)g(for)f(a)g(send,)h(or) f(the)h(tag)f(and)g(source)h(parameters)g(to)f(b)q(e)g(used)i(for)d(a)h (receiv)o(e.)24 b(In)15 b(addition,)f(this)75 1143 y(ob)r(ject)g(stores)h (information)10 b(ab)q(out)j(the)h(status)g(of)e(the)i(last)f(comm)o (unication)d(op)q(eration)j(that)g(w)o(as)g(p)q(erformed)75 1193 y(with)h(this)f(ob)r(ject.)19 b(This)14 b(ob)r(ject)h(is)f(accessed)i (using)d(a)h(comm)o(unicatio)o(n)d(handle.)158 1243 y(One)k(can)f(consider)h (comm)o(uni)o(cation)c(op)q(erations)j(to)g(consist)g(of)g(the)g(follo)o (wing)d(sub)q(op)q(erations:)75 1324 y Fh(INIT\(op)q(eration,)j(params,)i (handle\))i Fo(Pro)q(cess)c(pro)o(vides)f(all)f(relev)n(an)o(t)g(parameters)h (for)f(its)h(participation)179 1374 y(in)i(the)i(comm)o(unication)c(op)q (eration)j(\(t)o(yp)q(e)g(of)g(op)q(eration,)g(data)g(bu\013er,)h(tag,)e (participan)o(ts,)i(etc.\).)25 b(An)179 1424 y(ob)r(ject)14 b(is)g(created)i(that)e(iden)o(ti\014es)g(the)g(op)q(eration.)75 1503 y Fh(ST)l(AR)l(T\(handle\))k Fo(The)c(comm)o(unication)c(op)q(eration)k (is)g(started)75 1582 y Fh(COMPLETE\(handle\))19 b Fo(The)14 b(comm)o(unication)c(op)q(eration)k(is)g(completed.)75 1661 y Fh(FREE\(handle\))k Fo(The)d(comm)o(unicati)o(on)c(ob)r(ject,)j(and)g(asso) q(ciated)h(resources)h(are)e(freed.)75 1743 y(Correct)h(in)o(v)o(o)q(cation)e (of)g(these)i(sub)q(op)q(erations)g(is)f(a)f(sequence)k(of)c(the)h(form)587 1825 y Fh(INIT)j Fo(\()p Fh(ST)l(AR)l(T)f(COMPLETE)p Fo(\))1187 1808 y Fb(\003)1223 1825 y Fh(FREE)p Fd(:)75 1907 y Fo(I.e.,)10 b(an)h(ob)r(ject)g(needs)h(b)q(e)f(created)i(b)q(efore)e(comm)o(unication)c (o)q(ccurs;)13 b(it)d(can)h(b)q(e)g(reused)i(only)c(after)i(the)h(previous)75 1956 y(use)k(has)g(completed;)f(and)g(it)g(needs)i(to)e(b)q(e)h(freed)h(ev)o (en)o(tually)d(\(of)h(course,)i(one)f(can)f(assume)g(that)h(all)e(ob)r(jects) 75 2006 y(are)g(freed)h(at)f(program)e(termination,)g(b)o(y)h(default\).)158 2056 y(The)21 b(ab)q(o)o(v)o(e)g(scenario)g(p)q(ertains)g(to)g Fj(p)n(ersistent)f Fo(ob)r(jects.)39 b(One)22 b(can)f(also)f(create)i Fj(ephemer)n(al)e Fo(ob)r(jects.)75 2106 y(Suc)o(h)c(an)g(ob)r(ject)h(p)q (ersists)h(only)d(un)o(til)g(the)i(comm)o(uni)o(cation)c(op)q(eration)j(is)g (completed,)f(at)h(whic)o(h)g(p)q(oin)o(t)f(it)h(is)75 2156 y(destro)o(y)o(ed.)i(Th)o(us)12 b(the)g(correct)i(in)o(v)o(o)q(cation)c(of)h (sub)q(op)q(erations)i(with)e(an)g(ephemeral)g(ob)r(ject)i(is)e Fh(INIT)j(ST)l(AR)l(T)75 2205 y(COMPLETE)p Fo(.)158 2255 y(A)19 b(user)i(ma)o(y)c(directly)j(in)o(v)o(ok)o(e)f(these)h(sub)q(op)q(erations.) 36 b(This)19 b(w)o(ould)g(allo)o(w)e(the)j(amortization)e(of)g(the)75 2305 y(o)o(v)o(erhead)d(of)g(setting)g(up)g(a)g(comm)o(unication)d(o)o(v)o (er)j(man)o(y)e(successiv)o(e)k(uses)f(of)f(the)g(same)g(handle,)f(and)h (allo)o(ws)75 2355 y(the)d(o)o(v)o(erlap)f(of)g(comm)o(unication)e(and)i (computation.)16 b(Simpler)10 b(comm)o(unicatio)o(n)f(op)q(erations)j(com)o (bine)e(sev)o(eral)75 2405 y(of)17 b(these)i(sub)q(op)q(erations)g(in)o(to)e (one)h(op)q(eration,)g(th)o(us)g(simplifying)d(the)j(use)h(of)e(comm)o (unicatio)o(n)e(primitiv)o(es.)75 2455 y(Th)o(us,)h(one)h(only)e(needs)j(to)e (sp)q(ecify)g(precisely)h(the)g(seman)o(tics)f(of)f(these)j(sub)q(op)q (erations)f(in)f(order)g(to)g(sp)q(ecify)75 2504 y(the)e(seman)o(tics)g(of)f (MPI)h(p)q(oin)o(t)g(to)f(p)q(oin)o(t)h(comm)o(unicatio)o(n)d(op)q(erations.) 158 2554 y(W)m(e)i(sa)o(y)g(that)g(a)g(comm)o(unicatio)o(n)d(op)q(eration)j (\(send)i(or)e(receiv)o(e\))h(is)f Fh(p)q(osted)f Fo(once)i(a)f Fh(start)e Fo(sub)q(op)q(eration)75 2604 y(has)18 b(b)q(een)h(in)o(v)o(ok)o (ed;)g(the)g(op)q(eration)f(is)g Fh(completed)e Fo(once)i(the)h Fh(complete)d Fo(sub)q(op)q(eration)j(has)f(completed.)75 2654 y(A)e(send)h(and)f(a)g(receiv)o(e)i(op)q(eration)e Fh(matc)o(h)f Fo(if)g(the)i(receiv)o(e)h(pattern)f(sp)q(eci\014ed)g(b)o(y)f(the)h(receiv)o (e)h(matc)o(hes)d(the)75 2704 y(message)f(en)o(v)o(elop)q(e)g(created)h(b)o (y)f(the)g(send.)p eop %%Page: 17 18 bop 75 -100 a Fg(1.8.)31 b(COMMUNICA)m(TION)14 b(OBJECTS)1073 b Fo(17)75 45 y Fk(1.8.1)55 b(Comm)n(unication)21 b(Ob)s(ject)d(Creation)75 206 y Fm(Discussion:)35 b Fl(This)14 b(section)g(has)f(not)h(y)o(et)f(b)q (een)g(appro)o(v)o(ed)158 339 y Fo(An)c(ob)r(ject)i(for)d(a)h(send)i(op)q (eration)e(is)g(created)i(b)o(y)e(a)g(call)f(to)h Fh(MPI)p 1180 339 15 2 v 18 w(INIT)p 1304 339 V 17 w(SEND)p Fo(.)g(A)g(call)g(to)g Fh(MPI)p 1728 339 V 17 w(INIT)p 1851 339 V 18 w(RECV)75 389 y Fo(is)17 b(similarly)c(used)18 b(for)f(creating)g(an)f(ob)r(ject)i(for)e(a) h(receiv)o(e)h(op)q(eration.)26 b(The)18 b(creation)f(of)f(a)g(comm)o (unication)75 439 y(ob)r(ject)f(is)e(a)h(lo)q(cal)f(op)q(eration)h(that)g (need)h(not)e(in)o(v)o(olv)o(e)g(comm)o(unicatio)o(n)e(with)j(a)f(remote)h (pro)q(cess.)158 525 y Fh(MPI)p 257 525 V 17 w(INIT)p 380 525 V 18 w(SEND)i(\(handle,)e(bu\013er)p 856 525 V 15 w(handle,)g(dest,)h(tag,)h (con)o(text,)f(mo)q(de,)g(p)q(ersistence\))158 611 y Fo(Creates)g(a)f(send)h (comm)o(unicati)o(on)c(ob)r(ject.)19 b(P)o(arameters)14 b(are)75 696 y Fh(OUT)i(handle)i Fo(message)g(handle.)28 b(The)18 b(handle)g(should)f (not)g(b)q(e)h(asso)q(ciated)h(with)e(an)o(y)g(ob)r(ject)h(b)q(efore)g(the) 179 746 y(call.)75 832 y Fh(IN)e(bu\013er)p 273 832 V 16 w(handle)j Fo(handle)13 b(to)h(send)h(bu\013er)g(descriptor)75 918 y Fh(IN)h(dest)k Fo(rank)14 b(in)f(con)o(text)i(of)e(destination)h(\(in)o(teger\))75 1004 y Fh(IN)i(tag)21 b Fo(user)15 b(tag)e(for)h(messages)g(sen)o(t)g(with)g (this)g(handle)g(\(in)o(teger\))75 1090 y Fh(IN)i(con)o(text)k Fo(con)o(text)14 b(of)f(messages)h(sen)o(t)h(with)f(this)g(handle)75 1176 y Fh(IN)i(mo)q(de)21 b Fo(send)14 b(mo)q(de)f(\(state)i(t)o(yp)q(e,)f (with)f(three)j(v)n(alues:)h Fi(MPI)p 1119 1176 14 2 v 15 w(STANDARD)p Fo(,)12 b Fi(MPI)p 1400 1176 V 15 w(READY)h Fo(and)h Fi(MPI)p 1685 1176 V 15 w(SECURE)p Fo(\))75 1262 y Fh(IN)i(p)q(ersistence)i Fo(handle)10 b(p)q(ersistence)i(\(state)e(t)o(yp)q(e,)g(with)f(t)o(w)o(o)g(v) n(alues:)15 b Fi(MPI)p 1318 1262 V 15 w(PERSISTENT)8 b Fo(and)h Fi(MPI)p 1703 1262 V 15 w(EPHEMERAL)p Fo(\))158 1383 y Fh(MPI)p 257 1383 15 2 v 17 w(INIT)p 380 1383 V 18 w(RECV)15 b(\(handle,)e(bu\013er)p 859 1383 V 15 w(handle,)g(source,)h(tag,)g(con)o(text,)f(mo)q(de,)i(p)q (ersistence\))158 1469 y Fo(Create)g(a)e(receiv)o(e)j(handle.)i(P)o (arameters)c(are)75 1555 y Fh(OUT)i(handle)i Fo(message)g(handle.)28 b(The)18 b(handle)g(should)f(not)g(b)q(e)h(asso)q(ciated)h(with)e(an)o(y)g (ob)r(ject)h(b)q(efore)g(the)179 1604 y(call.)75 1690 y Fh(IN)e(bu\013er)p 273 1690 V 16 w(handle)j Fo(handle)13 b(to)h(receiv)o(e)h(bu\013er)g (descriptor.)75 1777 y Fh(IN)h(source)k Fo(rank)14 b(in)f(con)o(text)i(of)e (source,)i(or)f Fi(MPI)p 897 1777 14 2 v 15 w(ANY)p 978 1777 V 15 w(SOURCE)f Fo(\(in)o(teger\).)75 1863 y Fh(IN)j(tag)21 b Fo(user)15 b(tag)e(for)h(messages)g(receiv)o(ed)h(with)f(this)f(handle,)h (or)g Fi(MPI)p 1232 1863 V 15 w(ANY)p 1313 1863 V 15 w(TAG)f Fo(\(in)o(teger\).)75 1949 y Fh(IN)j(con)o(text)k Fo(con)o(text)14 b(of)f(messages)h(receiv)o(ed)i(with)d(this)h(handle.)75 2035 y Fh(IN)i(mo)q(de)21 b Fo(receiv)o(e)15 b(mo)q(de)e(\(state)h(t)o(yp)q(e,)g (with)g(t)o(w)o(o)f(v)n(alues:)18 b Fi(MPI)p 1133 2035 V 15 w(STANDARD)p Fo(,)12 b(and)i Fi(MPI)p 1495 2035 V 15 w(SECURE)p Fo(\))75 2121 y Fh(IN)i(p)q(ersistence)i Fo(handle)10 b(p)q(ersistence)i (\(state)e(t)o(yp)q(e,)g(with)f(t)o(w)o(o)g(v)n(alues:)15 b Fi(MPI)p 1318 2121 V 15 w(PERSISTENT)8 b Fo(and)h Fi(MPI)p 1703 2121 V 15 w(EPHEMERAL)p Fo(\))158 2206 y(See)15 b(Section)f(1.4.2)e(for) i(a)f(discussion)i(of)e(source,)i(tag)e(and)h(con)o(text.)158 2335 y Fm(Discussion:)35 b Fl(I)12 b(ha)o(v)o(e)h(not)g(included)i(prop)q (osals)f(for)f(partially)i(sp)q(eci\014ed)f(message)f(handles,)i(that)d(some) h(p)q(eoples)75 2381 y(seem)g(to)g(desire.)158 2427 y(I)g(ha)o(v)o(e)g (merged)h(all)g(handle)h(setup)e(in)o(to)h(one)g(call.)158 2474 y(The)c(functions)i Ff(MPI)p 459 2474 12 2 v 13 w(INIT)p 552 2474 V 12 w(SEND)c Fl(and)j Ff(MPI)p 784 2474 V 13 w(INIT)p 877 2474 V 13 w(RECV)d Fl(ha)o(v)o(e)j(the)f(same)g(list)h(of)f(parameters.) 17 b(Jim)10 b(Co)o(wnie)h(suggests)75 2519 y(to)i(ha)o(v)o(e)g(only)i(one)e (function)i(with)e(an)g(extra)h(argumen)o(t.)158 2566 y(Ephemeral)f(\\p)q (ersisten)o(t)h(handles")f(are)f(not)g(terribly)h(useful)g(as)f(suc)o(h.)17 b(But)12 b(their)g(existence)h(allo)o(w)g(us)f(to)f(describ)q(e)75 2612 y(all)j(the)f(higher)i(lev)o(el)f(send/receiv)o(e)h(op)q(erations)g(in)f (terms)f(of)g(basic)h(op)q(erations)h(on)e(handles.)p eop %%Page: 18 19 bop 75 -100 a Fo(18)712 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m(TION)75 45 y Fk(1.8.2)55 b(Comm)n(unication)21 b(Start)75 157 y Fh(MPI)p 174 157 15 2 v 17 w(ST)l(AR)l(T\(handle\))75 274 y(IN)16 b(handle)j Fo(comm)o(unication)10 b(handle)158 355 y(The)16 b Fi(MPI)p 314 355 14 2 v 15 w(START)f Fo(function)h(starts)g (the)h(execution)f(of)g(a)f(comm)o(unication)e(op)q(eration)i(\(send)i(or)f (receiv)o(e\).)75 404 y(A)i(sender)i(should)d(not)h(up)q(date)h(the)f(send)h (bu\013er)g(after)f(a)g(send)h(op)q(eration)f(has)g(started)h(un)o(til)e (after)h(it)g(has)75 454 y(completed.)31 b(A)18 b(receiv)o(er)i(should)e(not) h(access)h(the)f(receiv)o(e)h(bu\013er)f(after)g(a)f(receiv)o(e)h(op)q (eration)g(w)o(as)f(started)75 504 y(un)o(til)c(after)h(it)f(has)h (completed.)21 b(A)15 b(program)e(that)i(do)q(es)g(not)g(satisfy)g(these)h (conditions)e(is)h(erroneous)h(and)f(its)75 554 y(outcome)e(is)h (undetermined.)75 670 y Fk(1.8.3)55 b(Comm)n(unication)21 b(Completion)75 782 y Fh(MPI)p 174 782 15 2 v 17 w(W)-5 b(AIT)17 b(\()e(handle,)f(return)p 675 782 V 15 w(status)p 816 782 V 16 w(handle\))75 898 y(IN)i(handle)j Fo(comm)o(unication)10 b(handle)75 980 y Fh(IN)16 b(return)p 284 980 V 15 w(handle)j Fo(handle)13 b(that)h(is)f(asso)q(ciated)i(with)e (return)i(status)f(ob)r(ject.)19 b(the)14 b(return)g(status)h(ob)r(ject)f(is) 179 1030 y(up)q(dated)g(b)o(y)g(call.)158 1111 y(A)g(call)e(to)i Fh(MPI)p 428 1111 V 17 w(W)-5 b(AIT)14 b Fo(returns)h(when)f(the)g(send)g(op) q(eration)g(iden)o(ti\014ed)f(b)o(y)h Fi(handle)e Fo(is)h(complete.)18 b(The)75 1161 y(completion)13 b(of)h(a)g(send)h(op)q(eration)g(indicates)g (that)f(the)h(sender)i(is)d(no)o(w)g(free)i(to)e(up)q(date)h(the)g(lo)q (cations)f(in)g(the)75 1211 y(send)e(bu\013er,)h(or)e(an)o(y)g(other)h(lo)q (cation)e(that)i(can)f(b)q(e)h(referenced)i(b)o(y)d(the)h(send)g(op)q (eration.)17 b(Ho)o(w)o(ev)o(er,)12 b(it)f(do)q(es)h(not)75 1261 y(indicate)f(that)h(the)g(message)f(has)h(b)q(een)g(receiv)o(ed;)h (rather)g(it)e(ma)o(y)e(ha)o(v)o(e)j(b)q(een)g(bu\013ered)h(b)o(y)e(the)h (comm)o(unication)75 1310 y(subsystem.)158 1360 y(The)20 b(completion)e(of)i (a)f(receiv)o(e)i(op)q(eration)f(indicates)g(that)g(the)h(receiv)o(er)g(is)f (no)o(w)f(free)i(to)f(access)h(the)75 1410 y(lo)q(cations)15 b(in)g(the)h(receiv)o(e)h(bu\013er,)f(whic)o(h)f(con)o(tain)g(the)h(receiv)o (ed)h(message,)e(or)h(an)o(y)f(other)h(lo)q(cation)e(that)i(can)75 1460 y(b)q(e)g(referenced)i(b)o(y)d(the)h(receiv)o(e)h(op)q(eration.)23 b(It)15 b(do)q(es)h(not)g(indicate)f(that)h(the)g(matc)o(hing)d(send)j(op)q (eration)g(has)75 1510 y(completed.)158 1559 y(The)k(call)e(returns)j(a)e (handle)h(to)f(an)g(opaque)g(ob)r(ject)h(that)g(con)o(tains)f(information)d (on)k(the)g(completed)75 1609 y(op)q(eration)14 b({)f(the)i Fh(return)f(status)e Fo(ob)r(ject.)158 1694 y Fh(MPI)p 257 1694 V 17 w(ST)l(A)l(TUS)j(\(handle,)f(\015ag,)i(return)p 898 1694 V 15 w(handle\))75 1811 y(IN)g(handle)j Fo(comm)o(unication)10 b(handle)75 1893 y Fh(OUT)16 b(\015ag)k Fo(logical)75 1975 y Fh(OUT)c(return)p 335 1975 V 15 w(handle)i Fo(handle)c(that)g(is)g(asso)q (ciated)g(with)g(return)h(status)g(ob)r(ject.)158 2056 y(A)e(call)f(to)h Fh(MPI)p 426 2056 V 17 w(ST)l(A)l(TUS)f Fo(returns)j Fi(flag=true)c Fo(if)h(the)h(op)q(eration)g(iden)o(ti\014ed)g(b)o(y)g Fi(handle)e Fo(is)i(complete.)75 2106 y(In)j(suc)o(h)g(case,)g(the)h(return)f(handle)g(p) q(oin)o(ts)f(to)h(an)f(opaque)h(ob)r(ject)g(that)g(con)o(tains)f(information) e(on)i(the)h(com-)75 2156 y(pleted)f(op)q(eration.)k(It)c(returns)h Fi(flag=false)p Fo(,)c(otherwise.)20 b(In)15 b(suc)o(h)g(case,)g(the)g(v)n (alue)f(of)f(the)i(return)h(handle)e(is)75 2205 y(unde\014ned.)158 2255 y(A)f(call)g(to)g Fi(MPI)p 397 2255 14 2 v 15 w(WAIT)f Fo(blo)q(c)o(ks)h(only)f(the)i(executing)g(thread.)k(If)13 b(the)h(executing)g(pro)q(cess)h(is)e(m)o(ultithreaded,)75 2305 y(then)i(other)f(threads)h(within)e(the)i(pro)q(cess)g(can)f(b)q(e)h(sc) o(heduled)g(for)f(execution.)158 2355 y(The)h(use)h(of)f(a)f(blo)q(c)o(king)h (receiv)o(e)h(op)q(eration)f(\()p Fi(MPI)p 977 2355 V 15 w(WAIT)p Fo(\))f(allo)o(ws)g(the)i(op)q(erating)f(system)f(to)h(desc)o(hedule)75 2405 y(the)j(blo)q(c)o(k)o(ed)g(thread)h(and)e(sc)o(hedule)i(another)g (thread)f(for)f(execution,)j(if)c(suc)o(h)j(is)f(a)o(v)n(ailable.)27 b(The)18 b(use)h(of)e(a)75 2455 y(non)o(blo)q(c)o(king)12 b(receiv)o(e)i(op)q (eration)f(\()p Fi(MPI)p 710 2455 V 15 w(STATUS)p Fo(\))f(allo)o(ws)g(the)i (user)g(to)f(sc)o(hedule)h(alternativ)o(e)f(activities)g(within)75 2504 y(a)h(single)f(thread)i(of)e(execution.)158 2554 y(The)g(in)o(tended)g (implemen)o(tation)c(of)j Fi(MPI)p 820 2554 V 16 w(STATUS)f Fo(is)h(for)h(that)f(op)q(eration)h(to)f(return)i(as)f(so)q(on)f(as)h(p)q (ossible.)75 2604 y(Ho)o(w)o(ev)o(er,)h(if)f(rep)q(eatedly)i(called)f(for)f (an)h(op)q(eration)g(that)f(is)h(enabled,)g(it)f(m)o(ust)g(ev)o(en)o(tually)h (succeed.)158 2654 y(The)g(return)h(status)g(ob)r(ject)g(for)e(a)h(send)h(op) q(eration)e(carries)i(no)f(information.)h(The)f(return)h(status)g(ob)r(ject) 75 2704 y(for)h(a)g(receiv)o(e)i(op)q(eration)f(carries)g(information)d(on)i (the)i(source,)g(tag)e(and)g(length)h(of)f(the)h(receiv)o(ed)h(message.)p eop %%Page: 19 20 bop 75 -100 a Fg(1.8.)31 b(COMMUNICA)m(TION)14 b(OBJECTS)1073 b Fo(19)75 45 y(These)13 b(\014elds)g(are)f(required)h(b)q(ecause)h(the)e (receiv)o(e)i(op)q(eration)e(ma)o(y)e(ha)o(v)o(e)h(sp)q(eci\014ed)j Fi(DONTCARE)c Fo(in)i(either)h(source)75 95 y(or)h(tag)f(\014eld,)h(and)g (the)g(message)g(ma)o(y)e(ha)o(v)o(e)h(b)q(een)i(shorter)g(than)f(the)h (receiv)o(e)g(bu\013er.)158 180 y Fh(MPI)p 257 180 15 2 v 17 w(QUER)l(Y\()h(handle,)e(coun)o(t,)g(source,)h(tag\))75 295 y(IN)h(handle)j Fo(handle)14 b(to)g(return)h(status)f(ob)r(ject)75 376 y Fh(OUT)i(coun)o(t)j Fo(n)o(um)o(b)q(er)13 b(of)g(elemen)o(ts)h(receiv)o (ed.)75 457 y Fh(OUT)i(source)j Fo(rank)14 b(of)g(message)f(sender)j(in)d (message)h(con)o(text)g(\(in)o(teger\).)75 538 y Fh(OUT)i(tag)k Fo(tag)14 b(of)f(receiv)o(ed)i(message)f(\(in)o(teger\).)158 652 y Fh(MPI)p 257 652 V 17 w(CREA)l(TE)p 471 652 V 19 w(ST)l(A)l(TUS\()h (handle\))158 738 y Fo(Creates)g(a)f(new)g(return)h(status)g(ob)r(ject.)75 817 y Fh(OUT)h(handle)i Fo(handle)c(to)g(return)h(status)g(ob)r(ject.)158 974 y Fm(Discussion:)158 1020 y Fl(The)c(use)g(of)f(a)h(return)g(status)g(ob) r(ject,)g(rather)g(than)g(a)g(list)h(of)e(parameters)i(ma)o(y)f(simplify)i (the)e(use)g(of)f(MPI)h(routines,)75 1066 y(if)i(the)f(v)n(alues)i(stored)f (in)g(the)f(ob)r(ject)h(are)f(seldom)i(c)o(hec)o(k)o(ed.)j(A)12 b(prede\014ned)i(return)f(status)g(ob)r(ject)f(should)j(b)q(e)d(pro)o(vided,) 75 1111 y(to)h(ease)g(programming.)19 b(The)12 b(main)i(reason)g(for)e(the)h (use)g(of)g(a)f(return-status)i(ob)r(ject)f(is)g(that)g(one)h(w)o(an)o(ts)e (to)h(b)q(e)g(able)h(to)75 1157 y(use)c(the)h(same)f Ff(MPI)p 357 1157 12 2 v 13 w(WAIT)e Fl(and)j Ff(MPI)p 590 1157 V 13 w(STATUS)c Fl(calls)12 b(for)e(c)o(hec)o(king)h(on)g(di\013eren)o(t)g(t)o(yp) q(es)g(of)f(op)q(erations,)i(e.g.)k(b)q(oth)10 b(sends)h(and)75 1203 y(receiv)o(es,)k(and)f(p)q(erhaps)h(new)e(op)q(erations)j(in)e(the)g (future.)19 b(These)14 b(di\013eren)o(t)h(op)q(erations)g(return)g (di\013eren)o(t)g(information)75 1248 y(\(alb)q(eit)g(w)o(e)d(could)j(decree) e(that)g(a)g(send)h(returns)g(the)f(same)g(information)i(as)e(a)g(receiv)o (e\).)158 1294 y(The)h(issue)h(of)f(memory)g(allo)q(cation)j(for)d(return)p 868 1294 V 15 w(status)g(handles)i(is)f(not)f(y)o(et)g(solv)o(ed.)22 b(There)14 b(is)g(a)g(desire)h(that)g(the)75 1340 y(user)e(will)i(b)q(e)e (able)h(to)f(allo)q(cate)i(space)f(for)e(them)i(\(e.g.,)e(on)h(the)g(stac)o (k\).)158 1385 y(W)m(e)j(return)g(n)o(um)o(b)q(er)g(of)f(elemen)o(ts)i (receiv)o(ed,)g(rather)e(than)i(b)o(ytes)e(receiv)o(ed)i(b)q(ecause)g(\\n)o (um)o(b)q(er)f(of)f(elemen)o(ts")i(is)75 1431 y(closer)c(to)f(the)g (application)j(seman)o(tic)f(lev)o(el)f(\(user)f(do)q(es)h(not)f(need)h(to)f (b)q(e)g(a)o(w)o(are)g(of)g(the)g(size)h(of)f(elemen)o(ts\),)g(and)h(is)g (more)75 1477 y(in)o(v)n(arian)o(t.)23 b(The)14 b(return)h(status)f(ob)r (ject)h(returns)g(the)f(n)o(um)o(b)q(er)h(of)f Fm(elemen)o(ts)g Fl(receiv)o(ed.)22 b(If)13 b(there)i(w)o(as)f(no)g(truncation,)75 1522 y(then)h(this)g(is)g(equal)g(to)f(the)h(n)o(um)o(b)q(er)g(of)f(elemen)o (ts)h(sen)o(t.)21 b(On)14 b(the)g(other)h(hand,)g(the)f(n)o(um)o(b)q(er)h(of) f(b)o(ytes)h(collected)h(from)75 1568 y(the)d(sender)i(memory)m(,)e(the)g(n)o (um)o(b)q(er)h(of)f(b)o(ytes)h(sen)o(t)g(o)o(v)o(er)f(the)h(wire,)f(and)h (the)f(n)o(um)o(b)q(er)i(of)e(b)o(ytes)g(stored)h(in)g(the)g(receiv)o(er)75 1614 y(memory)c(ma)o(y)g(all)i(b)q(e)e(di\013eren)o(t,)i(in)e(a)g (heterogeneous)i(en)o(vironmen)o(t.)18 b(The)10 b(n)o(um)o(b)q(er)g(of)g (elemen)o(ts)h(sen)o(t)f(can)h(b)q(e)f(computed)75 1659 y(form)k(the)g(send)h (bu\013er)h(descriptor;)g(the)e(n)o(um)o(b)q(er)h(of)f(elemen)o(ts)i(receiv)o (ed)f(can)g(b)q(e)g(computed)g(from)f(the)h(receiv)o(e)g(bu\013er)75 1705 y(descriptor)g(and)e(the)g(length)i(\(in)e(b)o(ytes\))h(of)f(the)g (receiv)o(ed)h(message.)158 1751 y Ff(MPI)p 220 1751 V 13 w(STATUS)g Fl(and)j Ff(MPI)p 505 1751 V 13 w(QUERY)d Fl(are)i(not)h(go)q(o)q(d)g(names)g ({)g(they)f(can)h(b)q(e)g(easily)h(confused.)28 b(Leslie)18 b(Hart)e(suggests)75 1796 y Ff(MPI)p 137 1796 V 13 w(TEST)d Fl(or)i Ff(MPI)p 352 1796 V 13 w(CHECK)e Fl(for)h(the)h(\014rst.)23 b(P)o(erhaps)16 b(w)o(e)e(should)j(replace)f Ff(MPI)p 1229 1796 V 13 w(STATUS)c Fl(with)j Ff(MPI)p 1523 1796 V 13 w(TEST)e Fl(and)j Ff(MPI)p 1766 1796 V 13 w(QUERY)75 1842 y Fl(with)d Ff(MPI)p 224 1842 V 13 w(STATUS)p Fl(.)75 2040 y Fk(1.8.4)55 b(Multiple)20 b(Completions)75 2117 y Fo(It)c(is)g(con)o(v)o(enien)o(t)h(to)f (b)q(e)h(able)e(to)h(w)o(ait)g(for)g(the)g(completion)f(of)g(an)o(y)h(or)g (all)f(the)i(op)q(erations)f(in)g(a)g(set,)h(rather)75 2166 y(than)c(ha)o(ving)f(to)g(w)o(ait)g(for)h(a)g(sp)q(eci\014c)h(message.)j(A)c (call)g(to)f Fi(MPI)p 1096 2166 14 2 v 15 w(WAITANY)g Fo(or)h Fi(MPI)p 1393 2166 V 15 w(STATUSANY)e Fo(can)i(b)q(e)g(used)h(to)75 2216 y(w)o(ait)f(for)h(the)h(completion)d(of)h(one)i(out)f(of)f(sev)o(eral)i (op)q(erations;)e(a)h(call)f(to)h Fi(MPI)p 1340 2216 V 15 w(WAITALL)f Fo(can)h(b)q(e)h(used)g(to)f(w)o(ait)75 2266 y(for)g(all)e(p)q(ending)i(op)q (erations)g(in)g(a)f(list.)158 2351 y Fh(MPI)p 257 2351 15 2 v 17 w(W)-5 b(AIT)l(ANY)17 b(\()f(list)p 628 2351 V 15 w(of)p 682 2351 V 16 w(handles,)e(coun)o(t,)h(index,)f(return)p 1306 2351 V 15 w(handle\))158 2436 y Fo(Blo)q(c)o(ks)g(un)o(til)f(one)h(of)f(the)h (op)q(erations)g(asso)q(ciated)h(with)e(the)i(comm)o(uni)o(cation)c(handles)j (in)f(the)h(arra)o(y)g(has)75 2486 y(completed.)19 b(Returns)c(the)g(index)g (of)e(that)i(handle)f(in)g(the)h(arra)o(y)m(,)e(and)i(returns)h(the)f(status) g(of)f(that)g(op)q(eration)75 2536 y(in)f(the)i(ob)r(ject)g(asso)q(ciated)f (with)g(the)g(return)p 798 2536 13 2 v 16 w(handle.)k(The)d(parameters)f (are:)75 2623 y Fh(IN)i(list)p 215 2623 15 2 v 15 w(of)p 269 2623 V 17 w(handles)j Fo(list)13 b(of)g(handles)h(to)g(comm)o(unication)d(ob) r(jects.)75 2704 y Fh(IN)16 b(coun)o(t)j Fo(list)14 b(length)g(\(in)o (teger\))p eop %%Page: 20 21 bop 75 -100 a Fo(20)712 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m(TION)75 45 y Fh(OUT)i(index)j Fo(index)14 b(of)f(handle)h(for)f(op)q(eration)h(that)g(completed)f(\(in)o(teger\).)75 129 y Fh(OUT)j(return)p 335 129 15 2 v 15 w(handle)i Fo(handle)d(that)g(is)g (asso)q(ciated)h(with)e(return)i(status)g(ob)r(ject.)22 b(Set)16 b(to)f(return)h(status)g(of)179 179 y(op)q(eration)e(that)f(completed.)158 271 y(The)h(successful)i(execution)e(of)f Fi(MPI)p 733 271 14 2 v 16 w(WAITANY\(list)p 1013 271 V 13 w(of)p 1070 271 V 15 w(handles,)20 b(index,)h(return)p 1566 271 V 14 w(handle\))12 b Fo(has)i(the)75 321 y(same)i(e\013ect)i(as)f(the)h(successful)g(execution)g (of)e Fi(MPI)p 922 321 V 15 w(WAIT\(handle[i],)j(return)p 1418 321 V 14 w(handle\))p Fo(,)d(where)i Fi(i)f Fo(is)f(the)75 371 y(v)n(alue)g(returned)i(b)o(y)e Fi(index)g Fo(and)g Fi(handle[i])f Fo(is)h(the)h Fi(i)p Fo(-th)f(handle)h(in)f(the)h(list,)f(and)g(the)h (cancellation)f(of)g(all)75 421 y(remaining)c(w)o(ait)h(op)q(erations.)158 471 y(If)i(more)e(then)j(one)f(op)q(eration)g(is)g(enabled)g(and)g(can)g (terminate,)f(one)h(is)g(arbitrarily)f(c)o(hosen)i(\(sub)r(ject)g(to)75 521 y(the)e(restrictions)i(on)d(op)q(eration)h(termination)e(order,)i(and)g (fairness,)g(see)h(Section)f(1.13\).)158 571 y Fi(MPI)p 227 571 V 15 w(WAITANY)20 b(\()i(list)p 548 571 V 15 w(of)p 607 571 V 15 w(handles,)e(count,)h(index,)f(return)p 1255 571 V 15 w(status)p 1402 571 V 14 w(handle\))13 b Fo(is)140 655 y Fi({MPI_WAIT)20 b(\(handle[0],)g(return_handle\);)f(index)h(=)i(0})f(||)h (...)75 705 y(||)140 755 y({MPI_WAIT)e(\(handle[count-1],)f(return_handle\);) f(index)j(=)h(count-1})158 838 y Fo(\(\\)p Fe(jj)p Fo(")13 b(indicates)h(c)o(hoice;)g(one)g(of)f(the)i(alternativ)o(es)f(is)f(c)o (hosen,)i(nondeterministically)m(.\))158 924 y Fh(MPI)p 257 924 15 2 v 17 w(ST)l(A)l(TUSANY)h(\()g(list)p 685 924 V 15 w(of)p 739 924 V 16 w(handles,)e(coun)o(t,)h(index,)f(return)p 1363 924 V 15 w(handle\))158 1009 y Fo(Causes)20 b(either)h(one)f(or)f(none)h (of)f(the)h(op)q(erations)g(asso)q(ciated)g(with)f(the)h(comm)o(unication)c (handles)k(to)75 1059 y(return.)f(In)13 b(the)h(former)d(case,)j(it)f(has)g (the)h(same)e(return)i(seman)o(tics)e(as)i(a)e(call)g(to)h Fi(MPI)p 1449 1059 14 2 v 15 w(WAIT)p 1552 1059 V 15 w(ANY)p Fo(.)f(In)h(the)h(latter)75 1109 y(case,)g(it)g(returns)h(a)f(v)n(alue)f(of)g ({1)h(in)f Fi(index)g Fo(and)h Fi(return)p 991 1109 V 14 w(handle)f Fo(is)h(unde\014ned.)19 b(The)14 b(parameters)g(are:)75 1201 y Fh(IN)i(list)p 215 1201 15 2 v 15 w(of)p 269 1201 V 17 w(handles)j Fo(list)13 b(of)g(handles)h(to)g(comm)o(unication)d(ob)r(jects.)75 1286 y Fh(IN)16 b(coun)o(t)j Fo(list)14 b(length)g(\(in)o(teger\))75 1370 y Fh(OUT)i(index)j Fo(index)14 b(of)f(handle)h(for)f(op)q(eration)h (that)g(completed,)f(or)h(-1)f(if)g(none)h(completed)g(\(in)o(teger\).)75 1454 y Fh(OUT)i(return)p 335 1454 V 15 w(handle)i Fo(handle)d(that)g(is)g (asso)q(ciated)h(with)e(return)i(status)g(ob)r(ject.)22 b(Set)16 b(to)f(return)h(status)g(of)179 1504 y(op)q(eration)e(that)f(completed,)g(if) g(an)o(y;)g(unde\014ned)i(when)g Fi(index)21 b(=)g(-1)p Fo(.)158 1631 y Fh(MPI)p 257 1631 V 17 w(W)-5 b(AIT)l(ALL\(list)p 580 1631 V 15 w(of)p 634 1631 V 17 w(handles,)13 b(coun)o(t,)i(list)p 1047 1631 V 15 w(of)p 1101 1631 V 17 w(return)p 1253 1631 V 15 w(handles\))158 1717 y Fo(Blo)q(c)o(ks)k(un)o(til)e(all)g(comm)o(unicati)o (on)e(op)q(erations)k(asso)q(ciated)g(with)f(handles)g(in)g(the)g(list)g (complete,)g(and)75 1767 y(return)d(the)g(status)f(of)g(all)e(these)j(op)q (erations.)k(The)14 b(parameters)g(are:)75 1859 y Fh(IN)i(list)p 215 1859 V 15 w(of)p 269 1859 V 17 w(handles)j Fo(list)13 b(of)g(handles)h (to)g(comm)o(unication)d(ob)r(jects.)75 1943 y Fh(IN)16 b(coun)o(t)j Fo(lists)14 b(length)g(\(in)o(teger\))75 2027 y Fh(OUT)i(list)p 266 2027 V 15 w(of)p 320 2027 V 16 w(return)p 471 2027 V 15 w(handles)j Fo(Must)h(ha)o(v)o(e)g(the)g(same)f(length)h(as)f(the)i(\014rst)f (list.)35 b(Eac)o(h)20 b(return)h(status)179 2077 y(ob)r(ject)14 b(is)g(set)h(to)f(the)g(return)h(status)g(of)e(the)i(corresp)q(onding)g(op)q (eration)e(in)h(the)g(\014rst)h(list.)158 2248 y Fm(Discussion:)158 2294 y Fl(The)9 b(fairness)i(requiremen)o(t)f(giv)o(en)h(in)f(Section)g(1.13) g(implies)h(that)f(the)f(use)h(of)f Ff(WAIT)p 1376 2294 12 2 v 12 w(ANY)f Fl(cannot)i(lead)g(to)g(starv)n(ation:)75 2340 y(If)15 b(the)i(sending)g(pro)q(cess)g(has)g(issued)g(a)f(send)h(complete)g (op)q(eration,)i(and)d(the)g(receiving)j(pro)q(cess)d(rep)q(eatedly)i(p)q (ost)f(a)75 2385 y(receiv)o(e)i(for)f(the)h(message)g(sen)o(t,)g(then)g(that) f(message)h(m)o(ust)f(b)q(e)h(ev)o(en)o(tually)i(receiv)o(ed.)34 b(Section)19 b(1.13)g(has)f(not)h(y)o(et)75 2431 y(b)q(een)14 b(discussed.)19 b(The)12 b(fairness)i(requiremen)o(t)h(can)e(b)q(e)g(attac)o (k)o(ed)h(either)g(for)e(b)q(eing)j(to)q(o)e(w)o(eak)g(\(ev)o(en)o(tually)i (is)f(not)f(go)q(o)q(d)75 2477 y(enough\),)g(or)f(to)q(o)g(strong)g(\(hard)h (to)f(implemen)o(t\).)18 b(In)12 b(the)g(later)g(case,)g(if)g(the)g (requiremen)o(t)i(of)d(fairness)i(is)g(dropp)q(ed)g(from)75 2522 y(MPI)g(implemen)o(tation)q(s,)j(then)d(some)h(mec)o(hanism)h(need)e(b)q (e)h(pro)o(vided)h(to)e(the)h(user)f(to)h(ac)o(hiev)o(e)g(fairness)h(b)o(y)e (his)h(or)g(her)75 2568 y(o)o(wn)g(devices.)22 b(One)14 b(suc)o(h)h(prop)q (osal)h(is)f(to)f(b)q(e)g(able)i(to)e(sp)q(ecify)h(a)f(rotating)h(priorit)o (y)h(order)f(on)g(the)f(op)q(erations)i(p)q(osted)75 2614 y(b)o(y)d(a)g Ff(MPI)p 222 2614 V 13 w(WAITANY)p Fl(:)d(searc)o(h)k(the)f(list)h(sequen)o (tially)n(,)h(starting)f(from)f(a)g(user)g(sp)q(eci\014ed)i(p)q(osition.)p eop %%Page: 21 22 bop 75 -100 a Fg(1.9.)31 b(BLOCKING)14 b(COMMUNICA)m(TION)1038 b Fo(21)75 45 y Fn(1.9)70 b(Blo)r(c)n(king)21 b(Comm)n(unication)75 140 y Fo(Blo)q(c)o(king)14 b(send)i(and)f(receiv)o(e)h(op)q(erations)g(com)o (bine)d(all)h(comm)o(unication)e(sub)q(op)q(erations)j(in)o(to)g(one)g(call.) 21 b(The)75 190 y(op)q(eration)16 b(returns)j(only)c(when)i(the)h(comm)o (unicati)o(on)c(completes)i(and)g(no)h(comm)o(unicatio)o(n)d(ob)r(ject)j(p)q (ersists)75 239 y(after)d(the)h(call)e(completed.)k(Ho)o(w)o(ev)o(er,)d(the)h (bu\013er)f(descriptor)i(ob)r(ject)e(needs)i(b)q(e)e(created)h(ahead)f(of)g (the)g(call.)158 291 y(W)m(e)g(use)g(the)h(follo)o(wing)c(naming)h(con)o(v)o (en)o(tion)h(for)h(suc)o(h)g(op)q(erations:)793 352 y Fa(2)793 427 y(4)844 386 y Fe(\000)842 435 y Fh(R)846 485 y(S)898 352 y Fa(3)898 427 y(5)933 377 y(\024)979 411 y Fh(SEND)976 460 y(RECV)1135 377 y Fa(\025)158 583 y Fo(The)h(\014rst)g(letter)g(\(v)o(oid,)e Fh(R)h Fo(or)g Fh(S)p Fo(\))g(indicates)g(the)h(start)g(mo)q(de)e (\(standard,)h(ready)m(,)g(or)g(secure\).)21 b(Only)14 b(t)o(w)o(o)75 633 y(of)f(the)i(com)o(binations)d(\(standard)i(and)g(secure\))i(are)e (meaningful)d(for)j(receiv)o(es.)158 720 y Fh(MPI)p 257 720 15 2 v 17 w(SEND)i(\(bu\013er)p 565 720 V 15 w(handle,)e(dest,)h(tag,)h(con)o (text\))c Fo(is)75 846 y Fi(MPI_INIT_SEND\(han)o(dle,)18 b(buffer_handle,)h (dest,)i(tag,)g(context,)f(MPI_STANDARD,)75 896 y(MPI_EPHEMERAL\))75 946 y(MPI_START\(handle\))75 996 y(MPI_WAIT\(handle,)e(null\))158 1120 y Fh(MPI)p 257 1120 V 17 w(RSEND)e(\(bu\013er)p 601 1120 V 15 w(handle,)e(dest,)h(tag,)h(con)o(text\))c Fo(is)75 1246 y Fi(MPI_INIT_SEND\(han)o(dle,)18 b(buffer_handle,)h(dest,)i(tag,)g(context,) f(MPI_READY,)g(MPI_EPHEMERAL\))75 1296 y(MPI_START\(handle\))75 1346 y(MPI_WAIT\(handle,)e(null\))158 1470 y Fh(MPI)p 257 1470 V 17 w(SSEND)e(\(bu\013er)p 592 1470 V 15 w(handle,)e(dest,)h(tag,)h(con)o (text\))11 b Fo(is)75 1597 y Fi(MPI_INIT_SEND\(han)o(dle,)18 b(buffer_handle,)h(dest,)i(tag,)g(context,)f(MPI_SECURE,)g(MPI_EPHEMERAL\))75 1647 y(MPI_START\(handle\))75 1696 y(MPI_WAIT\(handle,)e(null\))158 1821 y Fh(MPI)p 257 1821 V 17 w(RECV\(bu\013er)p 554 1821 V 16 w(handle,)c(source,)h(tag,)h(con)o(text,)f(return)p 1312 1821 V 15 w(handle\))c Fo(is)75 1947 y Fi(MPI_INIT_RECV\(han)o(dle,)18 b(buffer_handle,)h(source,)i(tag,)g(context,)f(MPI_STANDARD,)f (MPI_EPHEMERAL\))75 1997 y(MPI_START\(handle\))75 2047 y(MPI_WAIT\(handle,r)o (eturn)o(_hand)o(le\))158 2171 y Fh(MPI)p 257 2171 V 17 w(SRECV\(bu\013er)p 581 2171 V 16 w(handle,)14 b(source,)h(tag,)h(con)o(text,)e(return)p 1338 2171 V 15 w(handle\))e Fo(is)75 2298 y Fi(MPI_INIT_RECV\(han)o(dle,)18 b(buffer_handle,)h(source,)i(tag,)g(context,)f(MPI_SECURE,)f(MPI_EPHEMERAL\)) 75 2347 y(MPI_START\(handle\))75 2397 y(MPI_WAIT\(handle,r)o(eturn)o(_hand)o (le\))158 2565 y Fm(Implemen)o(tation)13 b(note:)158 2612 y Fl(While)20 b(these)f(functions)g(can)g(b)q(e)f(implemen)o(ted)i(via)f(calls) h(to)e(functions)i(that)e(implemen)o(t)i(sub)q(op)q(erations,)i(as)75 2658 y(describ)q(ed)16 b(in)g(this)f(subsection,)h(an)f(e\016cien)o(t)g (implemen)o(tation)j(ma)o(y)d(optimize)h(a)o(w)o(a)o(y)e(these)h(m)o(ultiple) i(calls,)f(pro)o(vided)75 2704 y(it)d(do)q(es)h(not)f(c)o(hange)h(the)f(b)q (eha)o(vior)i(of)e(correct)g(programs.)p eop %%Page: 22 23 bop 75 -100 a Fo(22)716 b Fg(CHAPTER)14 b(1.)27 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m(TION)158 45 y Fm(Discussion:)158 91 y Fl(W)m(e)h(use)g(a)g(di\013eren)o(t)h(function)h(name,)e(rather)g(than)g (an)h(additional)i(mo)q(de)d(parameter,)h(in)f(order)g(to)g(sa)o(v)o(e)g(on)g (an)75 136 y(additional)g(parameter)d(\(p)q(erformance)g(and)g(user)g(con)o (v)o(enience\).)18 b(Lyndon)13 b(and)f(p)q(erhaps)g(other)g(prefer)g(few)o (er)e(functions)75 182 y(and)k(an)f(additional)j(parameter.)75 401 y Fn(1.10)70 b(Non)n(blo)r(c)n(king)22 b(Comm)n(unication)75 492 y Fo(Non)o(blo)q(c)o(king)11 b(send)i(and)f(receiv)o(e)h(op)q(erations)f (com)o(bine)f(the)i(\014rst)g(t)o(w)o(o)e(sub)q(op)q(erations)i(\()p Fi(INIT)f Fo(and)g Fi(START)p Fo(\))f(in)o(to)75 542 y(one)16 b(call.)21 b(They)16 b(use)g(ephemeral)f(comm)o(uni)o(cation)d(ob)r(jects,)17 b(so)e(that)g(the)h(op)q(eration)f(is)h(completed,)e(and)h(the)75 592 y(asso)q(ciated)d(resources)i(are)e(freed,)h(b)o(y)e(using)h(one)f(of)g (the)i(functions)e Fi(MPI)p 1221 592 14 2 v 15 w(WAIT,)21 b(MPI)p 1433 592 V 15 w(STATUS,)g(MPI)p 1689 592 V 15 w(WAITANY,)75 641 y(MPI)p 144 641 V 15 w(STATUSANY)p Fo(,)12 b(or)i Fi(MPI)p 498 641 V 15 w(WAITALL)p Fo(.)e(Here,)j(to)q(o,)f(a)g(bu\013er)h(ob)r(ject)g (has)g(to)f(b)q(e)g(created)i(ahead)e(of)g(the)h(comm)o(u-)75 691 y(nication)e(initiation)f(op)q(eration.)158 741 y(W)m(e)h(use)h(the)h (same)d(naming)f(con)o(v)o(en)o(tion)j(as)f(for)g(blo)q(c)o(king)g(op)q (erations:)18 b(a)13 b(pre\014x)h(of)f Fh(R)g Fo(\()p Fh(S)p Fo(\))h(indicates)f(the)75 791 y Fi(READY)e Fo(\()p Fh(SECURE)p Fo(\))i(mo)q(de.)k(In)12 b(addition,)f(a)h(pre\014x)h(of)f Fh(I)g Fo(is)g(used)h(to)g(indicate)f Fj(imme)n(diate)g Fo(\(i.e.,)f(non)o (blo)q(c)o(king\))75 841 y(execution.)158 926 y Fh(MPI)p 257 926 15 2 v 17 w(ISEND)16 b(\(handle,)e(bu\013er)p 750 926 V 15 w(handle,)g(dest,)h(tag,)h(con)o(text\))c Fo(is)75 1039 y Fi(MPI_INIT_SEND\(han)o(dle,)18 b(buffer_handle,)h(dest,)i(tag,)g(context,) f(MPI_STANDARD,)f(MPI_EPHEMERAL\))75 1089 y(MPI_START\(handle\))158 1203 y Fh(MPI)p 257 1203 V 17 w(IRSEND)d(\(handle,)e(bu\013er)p 786 1203 V 15 w(handle,)g(dest,)h(tag,)h(con)o(text\))c Fo(is)75 1316 y Fi(MPI_INIT_SEND\(han)o(dle,)18 b(buffer_handle,)h(dest,)i(tag,)g (context,)f(MPI_READY,)g(MPI_EPHEMERAL\))75 1366 y(MPI_START\(handle\))158 1479 y Fh(MPI)p 257 1479 V 17 w(ISSEND)c(\(handle,)d(bu\013er)p 776 1479 V 16 w(handle,)h(dest,)h(tag,)h(con)o(text\))c Fo(is)75 1593 y Fi(MPI_INIT_SEND\(han)o(dle,)18 b(buffer_handle,)h(dest,)i(tag,)g (context,)f(MPI_SECURE,)g(MPI_EPHEMERAL\))75 1643 y(MPI_START\(handle\))158 1756 y Fh(MPI)p 257 1756 V 17 w(IRECV\(handle,)15 b(bu\013er)p 740 1756 V 15 w(handle,)f(source,)i(tag,)f(con)o(text,)g(return)p 1497 1756 V 15 w(status)p 1638 1756 V 16 w(handle\))c Fo(is)75 1869 y Fi(MPI_INIT_RECV\(han)o(dle,)18 b(buffer_handle,)h(source,)i(tag,)g (context,)f(MPI_STANDARD,)f(MPI_EPHEMERAL\))75 1919 y(MPI_START\(handle\))158 2033 y Fh(MPI)p 257 2033 V 17 w(ISRECV\(handle,)14 b(bu\013er)p 766 2033 V 16 w(handle,)g(source,)h(tag,)h(con)o(text,)e(return)p 1523 2033 V 15 w(status)p 1664 2033 V 16 w(handle\))d Fo(is)75 2146 y Fi(MPI_INIT_RECV\(han)o(dle,)18 b(buffer_handle,)h(source,)i(tag,)g (context,)f(MPI_SECURE,)f(MPI_EPHEMERAL\))75 2196 y(MPI_START\(handle\))75 2332 y Fn(1.11)70 b(Con)n(tiguous)23 b(Blo)r(c)n(k)f(Comm)n(unication)f(Op)r (erations)75 2423 y Fo(The)c(most)f(frequen)o(t)i(t)o(yp)q(e)g(of)e(bu\013er) i(used)g(is)f(a)g(bu\013er)h(with)f(one)g(con)o(tiguous)g(comp)q(onen)o(t.)27 b(W)m(e)16 b(sp)q(ecialize)75 2473 y(the)e(functions)f(in)f(the)i(t)o(w)o(o)e (previous)i(subsections)g(to)f(this)g(case,)h(th)o(us)f(a)o(v)o(oiding)e(the) j(need)g(for)f(the)g(creation)h(of)75 2523 y(a)h(bu\013er)i(descriptor)f(ob)r (ject.)24 b(W)m(e)15 b(use)h(the)g(same)e(naming)g(sc)o(heme)h(used)h(in)f (the)h(previous)g(subsections,)h(and)75 2573 y(app)q(end)d(a)g Fh(C)g Fo(in)g(the)g(function)g(name,)e(for)i Fi(CONTIGUOUS)p Fo(.)158 2658 y Fh(MPI)p 257 2658 V 17 w(SENDC)j(\(start,)d(coun)o(t,)h (datat)o(yp)q(e,)f(dest,)h(tag,)h(con)o(text\))c Fo(is)p eop %%Page: 23 24 bop 75 -100 a Fg(1.11.)26 b(CONTIGUOUS)14 b(BLOCK)h(COMMUNICA)m(TION)g(OPERA) m(TIONS)505 b Fo(23)75 45 y Fi(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g (MPI_EPHEMERAL\))75 95 y(MPI_APPEND\()h(buffer_handle,)e(start,)j(count,)g (datatype\))75 145 y(MPI_SEND)f(\(buffer_handle,)f(count,)h(dest,)h(tag,)g (context\))158 257 y Fh(MPI)p 257 257 15 2 v 17 w(RSENDC)16 b(\(handle,)e(start,)h(coun)o(t,)f(datat)o(yp)q(e,)h(dest,)g(tag,)h(con)o (text\))c Fo(is)75 368 y Fi(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g (MPI_EPHEMERAL\))75 418 y(MPI_APPEND\()h(buffer_handle,)e(start,)j(count,)g (datatype\))75 468 y(MPI_RSEND\()f(buffer_handle,)f(count,)h(dest,)h(tag,)g (context\))158 580 y Fh(MPI)p 257 580 V 17 w(SSENDC)16 b(\(handle,)e(start,)h (coun)o(t,)f(datat)o(yp)q(e,)h(dest,)g(tag,)h(con)o(text\))c Fo(is)75 691 y Fi(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g (MPI_EPHEMERAL\))75 741 y(MPI_APPEND\()h(buffer_handle,)e(start,)j(count,)g (datatype\))75 791 y(MPI_SSEND\()f(buffer_handle,)f(count,)h(dest,)h(tag,)g (context\))158 903 y Fh(MPI)p 257 903 V 17 w(RECV)o(C)f(\(start,)e(coun)o(t,) h(datat)o(yp)q(e,)f(source,)h(tag,)g(con)o(text,)g(return)p 1565 903 V 15 w(status)p 1706 903 V 16 w(handle\))75 953 y Fo(is)75 1064 y Fi(MPI_CREATE_BUFFER)o(\()g(buffer_handle,)g(MPI_EPHEMERAL\)) 75 1114 y(MPI_APPEND\()h(buffer_handle,)e(start,)j(count,)g(datatype\))75 1164 y(MPI_RECV\()f(buffer_handle,)f(source,)h(tag,)h(context,)f (return_status_handl)o(e\))158 1276 y Fh(MPI)p 257 1276 V 17 w(SRECV)o(C)c(\(start,)f(coun)o(t,)f(datat)o(yp)q(e,)h(source,)g(tag,)g(con)o (text,)g(return)p 1565 1276 V 15 w(status)p 1706 1276 V 16 w(handle\))75 1326 y Fo(is)75 1437 y Fi(MPI_CREATE_BUFFER)o(\()k (buffer_handle,)g(MPI_EPHEMERAL\))75 1487 y(MPI_APPEND\()h(buffer_handle,)e (start,)j(count,)g(datatype\))75 1537 y(MPI_SRECV\()f(buffer_handle,)f (source,)h(tag,)h(context,)f(return_status_hand)o(le\))158 1649 y Fh(MPI)p 257 1649 V 17 w(ISENDC)d(\(handle,)c(start,)i(coun)o(t,)g (datat)o(yp)q(e,)g(dest,)g(tag,)g(con)o(text\))d Fo(is)75 1761 y Fi(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g(MPI_EPHEMERAL\))75 1810 y(MPI_APPEND\()h(buffer_handle,)e(start,)j(count,)g(datatype\))75 1860 y(MPI_ISEND\()f(handle,)g(buffer_handle,)f(dest,)i(tag,)g(context\))158 1972 y Fh(MPI)p 257 1972 V 17 w(IRSENDC)16 b(\(handle,)e(start,)h(coun)o(t,)g (datat)o(yp)q(e,)f(dest,)h(tag,)h(con)o(text\))c Fo(is)75 2084 y Fi(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g(MPI_EPHEMERAL\))75 2134 y(MPI_APPEND\()h(buffer_handle,)e(start,)j(count,)g(datatype\))75 2183 y(MPI_IRSEND\()f(handle,)g(buffer_handle,)f(dest,)i(tag,)f(context\))158 2295 y Fh(MPI)p 257 2295 V 17 w(ISSENDC)c(\(handle,)e(start,)h(coun)o(t,)f (datat)o(yp)q(e,)h(dest,)g(tag,)h(con)o(text\))c Fo(is)75 2407 y Fi(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g(MPI_EPHEMERAL\))75 2457 y(MPI_APPEND\()h(buffer_handle,)e(start,)j(count,)g(datatype\))75 2507 y(MPI_ISSEND\()f(handle,)g(buffer_handle,)f(dest,)i(tag,)f(context\))158 2618 y Fh(MPI)p 257 2618 V 17 w(IRECV)o(C\(handle,)11 b(start,)f(coun)o(t,)h (datat)o(yp)q(e,)f(source,)h(tag,)h(con)o(text,)e(return)p 1679 2618 V 15 w(status)p 1820 2618 V 16 w(handle\))75 2668 y Fo(is)p eop %%Page: 24 25 bop 75 -100 a Fo(24)712 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m(TION)75 45 y Fi(MPI_CREATE_BUFFER)k (\(buffer_handle,)h(MPI_EPHEMERAL\))75 95 y(MPI_APPEND\()h(buffer_handle,)e (start,)j(count,)g(datatype\))75 145 y(MPI_IRECV\()f(handle,)g (buffer_handle,)f(source,)h(tag,)h(context\))158 266 y Fh(MPI)p 257 266 15 2 v 17 w(ISRECV)o(C\(handle,)10 b(start,)h(coun)o(t,)f(datat)o(yp) q(e,)h(source,)g(tag,)g(con)o(text,)g(return)p 1706 266 V 15 w(status)p 1847 266 V 15 w(handle\))75 316 y Fo(is)75 439 y Fi(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g(MPI_EPHEMERAL\))75 489 y(MPI_APPEND\()h(buffer_handle,)e(start,)j(count,)g(datatype\))75 539 y(MPI_ISRECV\()f(handle,)g(buffer_handle,)f(source,)h(tag,)h(context\))75 682 y Fn(1.12)70 b(Prob)r(e)23 b(and)h(Cancel)75 775 y Fo(The)18 b Fi(MPI)p 233 775 14 2 v 15 w(PROBE)e Fo(op)q(eration)h(allo)o(ws)f (incoming)f(messages)i(to)g(b)q(e)h(c)o(hec)o(k)o(ed)g(for,)g(without)e (actually)h(receiving)75 825 y(them.)22 b(The)16 b(user)g(can)g(then)g (decide)h(where)f(to)g(receiv)o(e)g(them,)f(based)h(on)f(the)h(information)d (returned)k(b)o(y)e(the)75 874 y(prob)q(e)i(\(basically)m(,)f(the)h (information)d(on)i(the)i(message)e(en)o(v)o(elop)q(e\).)27 b(An)17 b(additional)e(function,)h Fi(MPI)p 1716 874 V 16 w(GET)p 1798 874 V 15 w(LEN)75 924 y Fo(allo)o(ws)11 b(the)j(amoun)o(t)c(of)i (storage)h(needed)i(to)d(receiv)o(e)i(the)f(message)g(to)f(b)q(e)h(computed,) f(when)h(this)g(length)g(is)f(not)75 974 y(readily)h(computed)h(from)e(the)i (information)d(returned)16 b(b)o(y)e Fi(MPI)p 1094 974 V 15 w(PROBE)p Fo(.)158 1025 y(The)f Fi(MPI)p 311 1025 V 15 w(CANCEL)e Fo(op)q(eration)h(allo)o(ws)f(p)q(ending)i(comm)o(unicatio)o(ns)d(to)i(b)q(e) h(cancelled.)19 b(This)12 b(is)g(required)h(for)75 1075 y(clean)o(up.)20 b(P)o(osting)14 b(a)h(send)g(or)g(a)f(receiv)o(e)i(ties)f(user)g(resources)i (\(send)f(or)e(receiv)o(e)i(bu\013ers\),)g(and)e(a)h(cancel)g(ma)o(y)75 1125 y(b)q(e)g(needed)g(to)f(free)g(these)i(resources)g(gracefully)m(.)158 1211 y Fh(MPI)p 257 1211 15 2 v 17 w(PR)o(OBE\()f(source,)g(tag,)h(con)o (text,)f(\015ag,)g(datat)o(yp)q(e,)g(return)p 1367 1211 V 15 w(status\))75 1342 y(IN)h(source)k Fo(rank)14 b(in)f(con)o(text)i(of)e (source,)i(or)f Fi(MPI)p 897 1342 14 2 v 15 w(ANY)p 978 1342 V 15 w(SOURCE)f Fo(\(in)o(teger\).)75 1429 y Fh(IN)j(tag)21 b Fo(user)15 b(tag)e(for)h(messages)g(receiv)o(ed)h(with)f(this)f(handle,)h (or)g Fi(MPI)p 1232 1429 V 15 w(ANY)p 1313 1429 V 15 w(TAG)f Fo(\(in)o(teger\).)75 1517 y Fh(IN)j(con)o(text)k Fo(con)o(text)14 b(of)f(messages)h(receiv)o(ed)i(with)d(this)h(handle.)75 1604 y Fh(OUT)i(\015ag)k Fo(\(logical\))75 1692 y Fh(IN)c(t)o(yp)q(e)k Fo(assumed)14 b(t)o(yp)q(e)g(of)f(data)h(in)f(message)h(\(status)h(v)n (ariable\).)75 1779 y Fh(OUT)h(return)p 335 1779 15 2 v 15 w(handle)i Fo(handle)c(that)g(is)g(asso)q(ciated)g(with)g(return)h(status)g (ob)r(ject.)158 1875 y Fi(MPI)p 227 1875 14 2 v 15 w(PROBE)h Fo(returns)i Fi(flag)j(=)g(true)16 b Fo(if)f(there)j(is)f(a)f(message)g(that) h(can)f(b)q(e)i(receiv)o(ed)f(and)g(that)f(matc)o(hes)75 1924 y(the)21 b(pattern)h(sp)q(eci\014ed)g(b)o(y)e(the)h(parameters)g Fi(source)p Fo(,)g Fi(tag)p Fo(,)g(and)f Fi(context)p Fo(.)37 b(It)21 b(returns)h Fi(flag)f(=)h(false)p Fo(,)75 1974 y(otherwise.)38 b(If)19 b Fi(MPI)p 411 1974 V 16 w(PROBE)g Fo(returns)i Fi(flag)g(=)h(true)p Fo(,)e(then)h(the)g(length,)g(source)g(and)f(tag)g(of)g(the)g(message)75 2024 y(matc)o(hed)13 b(are)i(returned)h(in)d(the)i(return)g(status)g(ob)r (ject.)20 b(These)15 b(are)g(the)f(same)g(v)n(alues)f(that)i(w)o(ould)e(ha)o (v)o(e)h(b)q(een)75 2074 y(returned)k(b)o(y)d(a)h(call)f(to)h Fi(MPI)p 542 2074 V 15 w(RECV)f Fo(or)h(to)g Fi(MPI)p 832 2074 V 15 w(SRECV)f Fo(executed)j(at)e(the)g(same)f(p)q(oin)o(t)h(in)f(the)i (program)d(\(with)75 2124 y(a)g(ca)o(v)o(eat)g(concerning)h(length;)f(see)i (b)q(elo)o(w\).)j(These)c(v)n(alues)f(can)h(b)q(e)g(deco)q(ded)g(from)e(the)i (return)g(status)g(ob)r(ject)75 2173 y(using)f(the)g Fi(MPI)p 324 2173 V 15 w(RETURN)p 471 2173 V 15 w(STAT)f Fo(function.)18 b(The)c(v)n(alue)f(returned)j(in)e(the)g(return)h(status)g(ob)r(ject)g(is)f (unde\014ned)h(if)75 2223 y Fi(flag=false)p Fo(.)158 2274 y(The)j(length)g(v) n(alue)f(returned)i(b)o(y)e(the)i(return)f(status)h(ob)r(ject)f(is)g (\(correctly\))h(the)f(n)o(um)o(b)q(er)f(of)g(elemen)o(ts)75 2324 y(in)h(the)h(message,)f(pro)o(vided)h(that)f(all)f(elemen)o(ts)h(in)g (the)h(message)f(are)h(of)e(the)i(t)o(yp)q(e)g(sp)q(eci\014ed)h(b)o(y)e(the)h Fi(type)75 2374 y Fo(parameter;)13 b(otherwise)i(the)f(length)g(v)n(alue)f (returned)j(is)e(unde\014ned.)158 2425 y(A)g(subsequen)o(t)j(receiv)o(e)e (executed)h(with)e(the)h(same)f(con)o(text,)g(and)h(the)g(source)g(and)f(tag) g(returned)i(b)o(y)e(the)75 2475 y(call)h(to)g Fi(MPI)p 274 2475 V 15 w(PROBE)f Fo(will)g(receiv)o(e)i(the)g(message)f(that)h(w)o(as)f (matc)o(hed)g(b)o(y)g(the)h(prob)q(e,)g(if)e(no)h(other)h(in)o(terv)o(ening) 75 2524 y(receiv)o(e)c(o)q(ccurred)g(after)f(the)g(prob)q(e.)18 b(If)10 b(the)h(receiving)g(pro)q(cess)i(is)d(m)o(ultithreaded,)g(it)g(is)h (the)g(user)g(resp)q(onsibilit)o(y)75 2574 y(to)j(ensure)h(that)f(the)h(last) e(condition)g(holds.)158 2704 y Fm(Discussion:)p eop %%Page: 25 26 bop 75 -100 a Fg(1.12.)31 b(PR)o(OBE)14 b(AND)g(CANCEL)1195 b Fo(25)158 45 y Fl(Do)12 b(w)o(e)f(w)o(an)o(t)g Ff(MPI)p 427 45 12 2 v 13 w(PROBE)e Fl(to)i(return)h(mo)q(de)g(\(standard)h(or)e (secure\)?)18 b(If)10 b(y)o(es,)i(w)o(e)f(need)h(to)f(carry)h(mo)q(de)g (information)75 91 y(with)j(messages)g(\(it's)f(deja)h(vu)g(all)g(o)o(v)o(er) g(again\).)22 b(If)14 b(not,)g(then)h(it)g(is)g(up)g(to)f(the)g(user)h(to)f (enco)q(de)h(this)h(information)g(\(in)75 136 y(the)d(tag\))g(so)g(that)h (receiv)o(er)g(can)f(decide)h(whether)g(to)f(use)g(a)g(secure)g(or)g (standard)i(send.)158 182 y(MPI)10 b(guaran)o(tees)h(that)e(successiv)o(e)j (messages)e(sen)o(t)g(from)f(a)h(source)g(to)g(a)g(destination)i(within)f (the)f(same)g(con)o(text)g(are)75 228 y(receiv)o(ed)i(in)f(the)g(order)g (they)g(are)f(sen)o(t.)17 b(Th)o(us,)11 b(MPI)f(m)o(ust)h(supp)q(ort,)h (either)f(explicitly)j(or)c(implicitl)q(y)m(,)k(a)c(FIF)o(O)g(structure)75 273 y(to)i(manage)h(messages)h(b)q(et)o(w)o(een)e(eac)o(h)h(pair)g(of)f (messages,)h(for)f(eac)o(h)h(con)o(text.)k Ff(MPI)p 1308 273 V 13 w(PROBE)11 b Fl(returns)i(information)h(on)f(the)75 319 y(\014rst)i(matc)o(hing)g(message)g(in)h(this)f(FIF)o(O;)e(this)j(will)g (also)f(b)q(e)g(the)f(message)h(receiv)o(ed)h(b)o(y)f(the)f(\014rst)h (subsequen)o(t)h(receiv)o(e)75 365 y(with)d(the)h(same)f(source,)g(tag)g(and) h(con)o(text)f(as)h(the)f(message)g(matc)o(hed)h(b)o(y)f Ff(MPI)p 1261 365 V 13 w(PROBE)p Fl(.)158 410 y(Message)19 b(passing)g(in)g(MPI)e(can) i(b)q(e)f(implemen)o(ted)i(without)e(app)q(ending)j(t)o(yp)q(e)d(information) i(to)d(messages.)32 b(A)75 456 y(message)14 b(is)g(merely)h(a)f(string)g(of)g (b)o(ytes)g(and)g(the)g(in)o(terpretation)i(of)d(these)h(b)o(ytes)g(in)o(to)h (a)e(sequence)i(of)e(t)o(yp)q(ed)i(elemen)o(ts)75 502 y(is)f(done)f(using)i (the)e(information)i(in)e(the)g(bu\013er)h(descriptors)h(at)e(eac)o(h)g(end.) 18 b(The)13 b(abilit)o(y)i(to)e(use)g(suc)o(h)h(implemen)o(tation)75 547 y(strategy)k(is)f(deemed)h(to)f(b)q(e)g(an)g(imp)q(ortan)o(t)i(goal.)30 b(In)17 b(suc)o(h)g(implemen)o(tation)q(,)j(when)d(a)g(message)h(arriv)o(es,) h(it)e(is)h(not)75 593 y(b)q(e)g(kno)o(wn)g(ho)o(w)f(man)o(y)h(elemen)o(ts)h (it)f(con)o(tains,)i(or)d(ev)o(en)h(ho)o(w)g(m)o(uc)o(h)g(storage)g(is)g (needed)h(to)e(receiv)o(e)h(that)g(message)75 639 y(\(b)q(ecause)13 b(of)f(p)q(ossible)j(represen)o(tation)f(con)o(v)o(ersion)g(in)f(a)g (heterogeneous)h(en)o(vironmen)o(t\).)k(The)12 b(prob)q(e)h(function)h (cannot)75 684 y(use)h(a)g(bu\013er)g(descriptor;)i(this)f(defeats)f(the)g (purp)q(ose)h(of)e(probing)j(in)e(order)g(to)g(decide)h(where)f(to)f(receiv)o (e)i(a)e(message.)75 730 y(Therefore,)d(prob)q(e)h(cannot,)g(in)f(general,)i (return)e(correct)g(length)h(information.)19 b(Still,)13 b(it)e(is)g(often)g (the)g(case)g(that)g(prob)q(e)h(is)75 776 y(used)g(to)f(decide)i(ho)o(w)e(m)o (uc)o(h)g(storage)h(to)f(allo)q(cate)i(in)f(order)g(to)f(receiv)o(e)h(a)f (message.)17 b(Enco)q(ding)d(suc)o(h)d(information)j(in)e(the)75 821 y(message)i(tag)f(is)g(deemed)h(to)f(b)q(e)g(to)q(o)g(a)o(wkw)o(ard,)g (and)g(it)g(is)h(deemed)g(imp)q(ortan)o(t)g(for)f Ff(MPI)p 1396 821 V 13 w(PROBE)d Fl(to)j(return)h(some)f(useful)75 867 y(size)k(information.)28 b(The)16 b(curren)o(t)h(de\014nition)i(of)c Ff(MPI)p 884 867 V 13 w(PROBE)f Fl(is)j(a)f(compromise)i(b)q(et)o(w)o(een)e (these)h(t)o(w)o(o)e(goals.)28 b(F)m(or)16 b(the)75 913 y(most)f(common)g (case)g(of)f(messages)h(where)g(all)h(en)o(tries)f(ha)o(v)o(e)g(the)g(same)g (t)o(yp)q(e)g Ff(MPI)p 1316 913 V 12 w(PROBE)e Fl(returns)i(the)g(correct)f (length)75 958 y(information;)20 b(the)d(more)g(esoteric)g(case)g(is)g (handled)i(b)o(y)e(the)g Ff(MPI)p 1076 958 V 13 w(GET)p 1149 958 V 12 w(LEN)f Fl(that)h(is)g(describ)q(ed)h(b)q(elo)o(w.)29 b(The)16 b(curren)o(t)75 1004 y(solution)g(sa)o(v)o(es)e(us)g(the)f(need)h (for)g(one)g(additional)i(w)o(ord)e(p)q(er)g(message)g(that)g(w)o(ould)g (otherwise)h(b)q(e)f(needed)g(to)g(transfer)75 1050 y(the)f(message)h(length) g(\(in)g(elemen)o(ts\))g(with)f(the)g(message.)158 1095 y(There)g(w)o(as)g(a) g(request)h(for)e(a)h(blo)q(c)o(king)j(prob)q(e)e(function.)158 1263 y Fh(MPI)p 257 1263 15 2 v 17 w(GET)p 376 1263 V 18 w(LEN\()h(coun)o(t,) g(return)p 805 1263 V 15 w(status,)g(bu\013er)p 1098 1263 V 15 w(descriptor\))158 1349 y Fo(Computes)g(the)i(n)o(um)o(b)q(er)e(of)h (elemen)o(ts)g(that)g(w)o(ere)h(to)f(b)q(e)g(receiv)o(ed,)i(if)d(the)i (message)e(that)h(is)g(asso)q(ciated)75 1398 y(with)e(the)i(return)f(status)h (handle)f(w)o(ould)e(b)q(e)j(receiv)o(ed)g(in)e(the)h(bu\013er)h(asso)q (ciated)f(with)g(the)g(bu\013er)h(descriptor)75 1448 y(handle.)75 1529 y Fh(OUT)g(coun)o(t)j Fo(n)o(um)o(b)q(er)13 b(of)g(elemen)o(ts)h(that)g (w)o(ere)h(to)f(b)q(e)g(receiv)o(ed)h(\(in)o(teger\))75 1612 y Fh(IN)h(return)p 284 1612 V 15 w(status)k Fo(handle)13 b(to)h(return)h (status)g(descriptor)75 1694 y Fh(IN)h(bu\013er)p 273 1694 V 16 w(descriptor)i Fo(handle)13 b(to)h(bu\013er)h(descriptor)158 1854 y Fm(Discussion:)158 1899 y Fl(In)d(order)h(to)g(supp)q(ort)g(this)h (function,)f(an)g(additional)j(\014eld)d(is)g(needed)h(in)f(the)g(return)g (status)f(ob)r(ject,)h(i.e.,)f(n)o(um)o(b)q(er)75 1945 y(of)17 b(b)o(ytes)g(in)h(the)f(incoming)j(message)d(matc)o(hed)h(b)o(y)f(prob)q(e,)i (or)e(v)n(alue)h(of)f(datat)o(yp)q(e)h(parameter)g(pro)o(vided)h(b)o(y)e (prob)q(e)75 1991 y(\(unless)d(w)o(e)f(add)h(this)g(parameter)f(to)g(the)g Ff(MPI)p 768 1991 12 2 v 13 w(GET)p 841 1991 V 13 w(LEN)f Fl(function,)i (whic)o(h)g(is)f(to)q(o)h(ugly)g(to)f(b)q(ear\).)158 2159 y Fh(MPI)p 257 2159 15 2 v 17 w(CANCEL\()k(handle,)d(\015ag\))75 2283 y(IN)i(handle)j Fo(handle)14 b(to)g(comm)o(uni)o(cation)d(ob)r(ject)75 2365 y Fh(OUT)16 b(\015ag)k Fo(\(logical\))158 2455 y(A)c(call)f(to)h Fi(MPI)p 405 2455 14 2 v 16 w(CANCEL)e Fo(cancels)j(a)f(p)q(ending)g(comm)o (unication)d(op)q(eration)j(\(send)h(or)f(receiv)o(e\).)26 b(The)17 b(call)75 2504 y(returns)12 b Fi(flag)21 b(=)g(true)9 b Fo(if)h(the)g(cancel)h(op)q(eration)f(succeeded,)j Fi(flag)21 b(=)h(false)9 b Fo(otherwise.)17 b(It)11 b(m)o(ust)e(b)q(e)h(the)h(case)75 2554 y(that)16 b(either)g(the)g(cancel)h(op)q(eration)e(succeeds)j(or)e(that) g(the)g(p)q(ending)g(comm)o(uni)o(cation)d(op)q(eration)i(completes)75 2604 y(\(but)k(not)g(b)q(oth\).)33 b(If)19 b(a)f(comm)o(unication)e(op)q (eration)i(w)o(as)h(cancelled)h(successfully)m(,)g(the)g(e\013ect)g(is)f(as)g (if)f(this)75 2654 y(op)q(eration)12 b(w)o(as)h(nev)o(er)g(executed.)20 b(If)12 b(a)g(send)h(w)o(as)g(cancelled)g(successfully)g(then)g(the)h (message)e(sen)o(t)h(will)e(not)h(b)q(e)75 2704 y(receiv)o(ed,)j(the)f (receiv)o(e)h(bu\013er)g(of)e(an)o(y)g(p)q(osted)i(receiv)o(e)g(for)e(that)h (message)f(will)g(not)g(b)q(e)i(altered,)e(and)h(an)o(y)f(suc)o(h)p eop %%Page: 26 27 bop 75 -100 a Fo(26)712 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m(TION)75 45 y Fo(receiv)o(e)i(has)g(to)f(b)q(e)h (satis\014ed)g(b)o(y)f(another)g(send;)h(if)f(a)g(receiv)o(e)h(w)o(as)f (cancelled)h(successfully)m(,)g(then)g(the)g(receiv)o(e)75 95 y(bu\013er)f(p)q(osted)g(will)d(not)i(b)q(e)h(altered,)f(and)f(an)o(y)h (send)h(matc)o(hing)d(this)i(receiv)o(e)h(has)f(to)g(b)q(e)g(satis\014ed)h(b) o(y)f(another)75 145 y(receiv)o(e.)158 276 y Fm(Implemen)o(tation)f(note:)158 324 y Fl(There)g(is)h(not)f(exp)q(ectation)i(that)e(a)g(cancel)h(op)q (eration)h(will)g(b)q(e)e(fast.)75 559 y Fn(1.13)70 b(Correctness)75 737 y Fm(Discussion:)18 b Fl(The)13 b(material)h(in)g(this)f(section)h(has)g (not)f(y)o(et)f(b)q(een)i(discussed)h(b)o(y)e(MPIF.)f(Some)i(or)e(all)j(of)d (it)h(is)h(lik)o(ely)h(to)75 787 y(mo)o(v)o(e)e(to)g(Section)i Fm(??)p Fl(.)h(It)d(is)g(incorp)q(orated)i(here)f(for)f(completeness.)75 1001 y Fk(1.13.1)55 b(Order)75 1083 y Fo(MPI)15 b(preserv)o(es)i(the)e(order) g(of)f(messages)h(within)f(the)h(same)f(con)o(text,)h(b)q(et)o(w)o(een)h(an)o (y)e(\014xed)h(pair)f(of)g(pro)q(cesses.)75 1132 y(In)19 b(other)g(w)o(ords,) h(if)e(pro)q(cess)j(A)e(executes)i(t)o(w)o(o)e(successiv)o(e)i(send)f Fi(start)d Fo(sub)q(op)q(erations)j(within)e(the)i(same)75 1182 y(con)o(text,)13 b(pro)q(cess)i(B)d(executes)j(t)o(w)o(o)d(successiv)o (e)j(receiv)o(e)f Fi(start)e Fo(op)q(erations,)g(and)h(b)q(oth)f(receiv)o(es) j(matc)o(h)c(either)75 1232 y(sends,)17 b(then)f(the)g(\014rst)g(receiv)o(e)h (will)d(receiv)o(e)j(the)f(message)f(sen)o(t)h(b)o(y)f(the)h(\014rst)h(send,) f(and)f(the)h(second)h(receiv)o(e)75 1282 y(will)12 b(receiv)o(e)i(the)g (message)f(sen)o(t)h(b)o(y)e(the)i(second)g(send.)19 b(Th)o(us,)13 b(if)f(a)h(t)o(w)o(o)g(messages)g(from)e(the)j(same)e(source)j(can)75 1332 y(satisfy)h(a)g(p)q(ending)h(receiv)o(e,)h(the)f(\014rst)g(message)f (sen)o(t)h(is)f(accepted;)j(if)d(a)g(message)g(can)h(satisfy)f(t)o(w)o(o)g(p) q(ending)75 1381 y(receiv)o(es,)f(the)g(\014rst)f(receiv)o(e)h(p)q(osted)g (is)f(satis\014ed.)158 1434 y(The)h(last)f(paragraph)f(assumes)h(that)h(the)f (send)i Fi(start)d Fo(op)q(erations)h(are)h(ordered)g(b)o(y)f(the)h(program)d (order)75 1484 y(at)i(pro)q(cess)h(A,)f(and)f(the)i(receiv)o(e)g Fi(start)e Fo(op)q(erations)h(are)g(ordered)h(b)o(y)f(the)g(program)e(order)j (at)e(pro)q(cess)j(B.)e(If)f(a)75 1534 y(pro)q(cess)i(is)f(m)o(ultithreaded)e (and)h(the)h(op)q(erations)g(are)g(executed)h(b)o(y)f(distinct)f(threads,)h (then)h(the)f(seman)o(tics)f(of)75 1583 y(the)e(threaded)h(system)f(ma)o(y)d (not)j(de\014ne)h(an)e(order)h(b)q(et)o(w)o(een)h(the)g(t)o(w)o(o)e(op)q (erations,)h(in)f(whic)o(h)h(case)g(the)h(condition)75 1633 y(is)i(v)o(oid.)75 1764 y Fk(1.13.2)55 b(Progress)19 b(and)g(F)-5 b(airness)75 1846 y Fo(W)m(e)15 b(can)i(mo)q(del)d(the)i(execution)h(of)e (MPI)h(programs)f(as)h(an)g(in)o(teraction)f(b)q(et)o(w)o(een)j(executing)e (pro)q(cesses)j(that)75 1896 y(execute)e(eac)o(h)e(their)h(o)o(wn)e(program,) f(and)i(the)g Fh(comm)o(unication)f(subsystem)p Fo(.)19 b(The)c(comm)o (unication)d(sub-)75 1945 y(system)i(ma)o(y)e(ha)o(v)o(e)h(v)n(arious)h (constrain)o(ts)g(on)g(the)g(amoun)o(t)e(of)i(resources)i(it)d(can)h(use.)19 b(E.g.:)158 1998 y(Bounds)f(on)f(the)g(n)o(um)o(b)q(er)f(and)h(total)g(sizes) h(of)e(activ)o(e)h(comm)o(unication)d(ob)r(jects.)28 b(Suc)o(h)18 b(b)q(ound)f(can)g(b)q(e)75 2048 y(global,)12 b(p)q(er)j(no)q(de,)e(or)h(p)q (er)h(pair)e(of)h(comm)o(unicati)o(ng)d(no)q(des.)158 2100 y(Bounds)18 b(on)e(the)i(n)o(um)o(b)q(er)e(and)h(total)f(sizes)i(of)f (messages)g(bu\013ered)h(in)f(the)h(system.)27 b(Suc)o(h)17 b(b)q(ound)g(can,)75 2150 y(again,)h(b)q(e)h(global,)e(p)q(er)j(no)q(de,)f (or)g(p)q(er)g(pair)f(of)g(comm)o(unicating)d(no)q(de.)32 b(In)19 b(addition,)f(a)g(message)g(ma)o(y)e(b)q(e)75 2200 y(bu\013ered)f(at)f(the)h (sender,)g(at)e(the)i(receiv)o(er,)g(at)f(b)q(oth,)f(or)h(p)q(erhaps)h(at)f (another)g(place)g(altogether.)158 2253 y(Th)o(us,)f(it)g(will)e(b)q(e)j (di\016cult)e(to)h(set)h(rules)f(on)g(resource)i(managemen)o(t)c(of)h(the)i (comm)o(uni)o(cation)c(subsystem.)75 2302 y(Ho)o(w)o(ev)o(er,)i(it)f(is)g (generally)g(exp)q(ected)j(that)d(implemen)o(ters)f(will)g(pro)o(vide)h (information)e(on)i(the)h(mec)o(hanism)d(used)75 2352 y(for)j(resource)i (allo)q(cation,)c(and)i(that)h(query)f(and)g(set)h(functions)g(will)d(allo)o (w)h(to)h(query)h(and)f(p)q(ossibly)f(con)o(trol)h(the)75 2402 y(amoun)o(t)g(of)h(a)o(v)n(ailable)f(resources.)158 2455 y(W)m(e)18 b(pro)o(vide)g(in)g(this)g(section)i(a)e(set)h(of)f(minim)o(al)c(requiremen)o (ts)19 b(on)f(the)h(comm)o(unication)c(subsystem.)75 2504 y(Programs)f(that)h (execute)h(on)f(an)o(y)f(subsystem)h(that)g(ful\014ls)f(these)j(minim)o(al)11 b(requiremen)o(ts)k(are)g Fh(safe)g Fo(and)f(will)75 2554 y(p)q(ort)i(to)g (an)o(y)f(MPI)h(implemen)o(tation.)21 b Fh(Unsafe)15 b Fo(programs)f(ma)o(y)g (execute)k(on)d(some)g(MPI)i(implem)o(en)o(tations,)75 2604 y(dep)q(ending)e(on)f(the)h(amoun)o(t)e(of)g(a)o(v)n(ailable)f(resources)17 b(and)d(the)h(implemen)o(tation)c(used)k(for)f(the)h(MPI)g(comm)o(u-)75 2654 y(nication)f(subsystem.)20 b(Finally)13 b Fh(erroneous)f Fo(programs)h(nev)o(er)i(execute)i(correctly)m(.)j(\(While)14 b(it)g(is)g(desirable)h(to)75 2704 y(detect)i(erroneous)f(programs,)e(it)h (is)g(not)g(p)q(ossible)g(to)g(do)g(so)h(at)f(compile)e(time,)h(and)h(often)g (prohibitiv)o(e)f(to)h(do)p eop %%Page: 27 28 bop 75 -100 a Fg(1.13.)26 b(CORRECTNESS)1328 b Fo(27)75 45 y(so)16 b(a)g(run)g(time.)22 b(Th)o(us,)17 b(the)f(do)q(cumen)o(t)f(do)q(es)i (not)f(sp)q(ecify)g(a)g(b)q(eha)o(vior)f(for)h(erroneous)h(programs,)e (although)75 95 y(the)f(desired)i(b)q(eha)o(vior)d(is)h(to)g(return)h(a)e (useful)h(error)h(message.\))126 186 y(1.)20 b(If)13 b(a)g(pro)q(cess)i (executes)h(an)d Fi(INIT)g Fo(op)q(eration,)g(then)h(the)g(op)q(eration)f(ev) o(en)o(tually)g(succeeds,)j(or)d(a)h Fj(r)n(esour)n(c)n(e)179 236 y(exc)n(eption)20 b Fo(o)q(ccurs.)38 b(The)20 b(standard)h(do)q(es)g(not) e(sp)q(ecify)i(when)f(a)g(resource)i(exception)f(is)e(allo)o(w)o(ed)g(to)179 286 y(o)q(ccur.)33 b(It)18 b(is)h(exp)q(ected)h(that)f(an)f(op)q(erational)f (de\014nition)i(will)d(b)q(e)k(made)d(a)o(v)n(ailable,)g(in)h(the)h(form)e (of)179 336 y(test)e(programs)d(that)i(ha)o(v)o(e)f(to)h(execute)h(with)f(no) f(resource)j(exceptions.)j(It)14 b(is)f(highly)g(desirable)h(to)f(ha)o(v)o(e) 179 386 y(generous)18 b(b)q(ounds)g(on)f(the)h(n)o(um)o(b)q(er)f(of)f (concurren)o(tly)j(activ)o(e)e(comm)o(unication)d(ob)r(jects)k(eac)o(h)g(pro) q(cess)179 435 y(ma)o(y)12 b(ha)o(v)o(e,)h(so)h(that,)f(in)h(practice,)g Fi(INIT)f Fo(op)q(erations)h(will)f(alw)o(a)o(ys)g(b)q(e)h(guaran)o(teed)h (to)e(succeed.)126 519 y(2.)20 b(Eac)o(h)d(pro)q(cess)h(can)f(initiate)f(a)g (comm)o(unicatio)o(n)e(op)q(eration)j(for)f(eac)o(h)h(activ)o(e)f(comm)o (unication)e(ob)r(ject.)179 568 y(I.e.)k(correct)d Fi(START)e Fo(op)q(erations)h(alw)o(a)o(ys)f(succeed)j(\(ev)o(en)o(tually\).)126 652 y(3.)k(A)f(send)i(op)q(eration)e(is)h Fh(enabled)d Fo(if)h(the)j(sending) e(pro)q(cess)j(has)d(issued)i(a)e Fi(COMPLETE)f Fo(op)q(eration)h(and)179 701 y(the)d(receiving)h(pro)q(cess)g(has)f(issued)h(a)f Fi(START)f Fo(op)q(eration)g(for)h(a)g(matc)o(hing)e(receiv)o(e.)25 b(Symmetrically)l(,) 13 b(a)179 751 y(receiv)o(e)h(op)q(eration)f(is)g Fh(enabled)d Fo(if)i(the)i(receiving)f(pro)q(cess)i(has)e(issued)h(a)f Fi(COMPLETE)e Fo(op)q(eration)i(and)f(the)179 801 y(sending)h(pro)q(cess)i(has)e(issued)h (a)f Fi(START)f Fo(op)q(eration)h(for)g(a)f(matc)o(hing)g(send.)18 b(An)c(enabled)f(op)q(eration)g(ma)o(y)179 851 y(b)q(ecome)g Fh(disabled)e Fo(either)k(b)q(ecause)g(it)e(completes)h(successfully)h(or,)e (in)g(the)h(case)h(of)e(a)h(receiv)o(e,)g(b)q(ecause)179 901 y(the)g(matc)o(hing)d(message)i(is)g(successfully)i(receiv)o(ed)f(b)o(y)f (another)h(receiv)o(e)h(op)q(eration)e(or,)g(in)f(the)i(case)h(of)d(a)179 951 y(send,)i(b)q(ecause)i(the)e(matc)o(hing)e(receiv)o(e)j(successfully)g (receiv)o(es)h(another)e(message.)179 1017 y Fh(An)i(enabled)f(op)q(eration)f (either)h(completes)f(successfully)g(or)j(b)q(ecomes)f(p)q(ermanen)o(tl)o(y)e (dis-)179 1067 y(abled.)179 1133 y Fo(I.e.,)g(an)h(enabled)h(op)q(eration)f (either)h(ev)o(en)o(tually)e(succeeds,)k(or)d(b)q(ecomes)g(disabled)g (\(progress\);)i(and)e(an)179 1183 y(op)q(eration)f(cannot)g(b)q(e)g(enabled) g(in\014nitely)f(man)o(y)f(times)h(without)h(ev)o(er)h(succeeding)g (\(fairness\).)126 1266 y(4.)20 b(A)14 b Fi(FREE)f Fo(op)q(eration)h(alw)o(a) o(ys)e(succeeds)17 b(\(ev)o(en)o(tually\).)158 1358 y(The)e(four)g (conditions)f(guaran)o(tee)h(progress)h(in)e(the)i(comm)o(unicati)o(on)c (subsystem.)21 b(The)15 b(third)g(condition)75 1408 y(guaran)o(tee)f(\(w)o (eak\))g(fairness)h(among)d(comp)q(eting)g(comm)o(unication)f(requests.)158 1458 y(Examples)i(\(in)o(v)o(olving)f(t)o(w)o(o)h(pro)q(cessors)j(with)e (ranks)g(0)f(and)h(1\))158 1507 y(The)g(follo)o(wing)e(program)g(is)i(safe,)f (and)h(should)g(alw)o(a)o(ys)f(succeed.)75 1640 y Fi(CALL)21 b(MPI_RANK\(rank,)e(context\))75 1690 y(IF)i(\(rank.EQ.0\))f(THEN)140 1740 y(CALL)h(MPI_SENDC\(sendbuf,)d(count,)j(1,)g(tag,)g(context\))140 1790 y(CALL)g(MPI_RECVC\(recvbuf,)d(count,)j(1,)g(tag,)g(context\))75 1840 y(ELSE)86 b(!)22 b(rank.EQ.1)140 1889 y(CALL)f(MPI_RECVC\(recvbuf,)d (count,)j(0,)g(tag,)g(context\))140 1939 y(CALL)g(MPI_SENDC\(sendbuf,)d (count,)j(0,)g(tag,)g(context\))75 1989 y(END)g(IF)158 2072 y Fo(The)14 b(follo)o(wing)e(program)g(is)i(erroneous,)g(and)g(should)g(alw)o (a)o(ys)f(fail.)75 2214 y Fi(CALL)21 b(MPI_RANK\(rank,)e(context\))75 2263 y(IF)i(\(rank.EQ.0\))f(THEN)140 2313 y(CALL)h(MPI_RECVC\(recvbuf,)d (count,)j(1,)g(tag,)g(context\))140 2363 y(CALL)g(MPI_SENDC\(sendbuf,)d (count,)j(1,)g(tag,)g(context\))75 2413 y(ELSE)86 b(!)22 b(rank.EQ.1)140 2463 y(CALL)f(MPI_RECVC\(recvbuf,)d(count,)j(0,)g(tag,)g(context\))140 2512 y(CALL)g(MPI_SENDC\(sendbuf,)d(count,)j(0,)g(tag,)g(context\))75 2562 y(END)g(IF)158 2654 y Fo(The)13 b(receiv)o(e)h(op)q(eration)f(of)f(the)i (\014rst)f(pro)q(cess)i(m)o(ust)d(complete)g(b)q(efore)i(its)e(send,)i(and)f (can)g(complete)f(only)75 2704 y(if)g(the)h(matc)o(hing)d(send)k(of)e(the)h (second)g(pro)q(cessor)i(is)d(executed;)i(the)f(receiv)o(e)h(op)q(eration)f (of)e(the)j(second)f(pro)q(cess)p eop %%Page: 28 29 bop 75 -100 a Fo(28)712 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m(TION)75 45 y Fo(m)o(ust)i(complete)h(b)q(efore)h (its)g(send)g(and)f(can)h(complete)e(only)h(if)f(the)i(matc)o(hing)e(send)i (of)f(the)h(\014rst)g(pro)q(cess)h(is)75 95 y(executed.)h(This)14 b(program)e(will)g(deadlo)q(c)o(k.)158 145 y(The)i(follo)o(wing)e(program)g (is)i(unsafe,)f(and)h(ma)o(y)e(succeed)k(or)e(fail,)e(dep)q(ending)i(on)g (implemen)o(tatio)o(n.)75 286 y Fi(CALL)21 b(MPI_RANK\(rank,)e(context\))75 336 y(IF)i(\(rank.EQ.0\))f(THEN)140 385 y(CALL)h(MPI_SENDC\(sendbuf,)d (count,)j(1,)g(tag,)g(context\))140 435 y(CALL)g(MPI_RECVC\(recvbuf,)d (count,)j(1,)g(tag,)g(context\))75 485 y(ELSE)86 b(!)22 b(rank.EQ.1)140 535 y(CALL)f(MPI_SENDC\(sendbuf,)d(count,)j(0,)g(tag,)g(context\))140 585 y(CALL)g(MPI_RECVC\(recvbuf,)d(count,)j(0,)g(tag,)g(context\))75 635 y(END)g(IF)158 776 y Fo(The)15 b(message)g(sen)o(t)h(b)o(y)f(eac)o(h)g (pro)q(cess)i(has)e(to)g(b)q(e)h(copied)f(out)g(b)q(efore)h(the)g(send)g(op)q (eration)f(returns)h(and)75 826 y(the)h(receiv)o(e)g(op)q(eration)e(starts.) 26 b(F)m(or)15 b(the)i(program)d(to)i(complete,)f(it)g(is)h(necessary)i(that) e(at)g(least)g(one)g(of)f(the)75 875 y(t)o(w)o(o)d(messages)g(sen)o(t)h(is)f (bu\013ered)i(out)e(of)f(either)i(pro)q(cesses')h(address)g(space.)k(Th)o (us,)13 b(this)f(program)e(can)j(succeed)75 925 y(only)g(if)g(the)i(comm)o (unicati)o(on)c(system)j(has)g(su\016cien)o(t)g(bu\013er)h(space)g(to)f (bu\013er)h Fi(count)d Fo(w)o(ords)i(of)g(data.)158 975 y(If)c(additional)e (requiremen)o(ts)i(will)f(b)q(ecome)h(part)g(of)f(the)i(standard)f(\(e.g.,)g (b)q(ounds)h(on)e(the)i(minim)o(al)c(n)o(um)o(b)q(er)75 1025 y(of)13 b(concurren)o(tly)i(activ)o(e)f(handles)g(that)g(need)h(b)q(e)g(supp) q(orted,)f(then)h(further)g(programs)d(b)q(ecome)i(safe.)75 1162 y Fn(1.14)70 b(Missing)75 1253 y Fh(send-receiv)o(e)39 b Fo(a)13 b(function)h(of)f(the)i(form)d(\(for)i(blo)q(c)o(king)e(comm)o (unication\))158 1303 y Fi(MPI)p 227 1303 14 2 v 15 w(SEND)p 330 1303 V 15 w(RECV\()21 b(send)p 564 1303 V 15 w(buffer,)f(destination,)f (recv)p 1124 1303 V 15 w(buffer,)h(source,)h(tag,)g(context\))158 1353 y Fo(It)13 b(ma)o(y)e(b)q(e)j(sup)q(er\015uous)h(b)q(ecause)f(the)g (collectiv)o(e)f(comm)o(unicatio)o(n)e(shift)h(has)h(similar)e(functionalit)o (y)m(.)16 b(Y)m(et)75 1402 y(it)10 b(allo)o(ws)g(to)g(generate)i(more)e (easily)g(comm)o(unication)d(patterns)12 b(that)f(w)o(ould)f(lead)g(to)h (deadlo)q(c)o(k)f(if)g(implemen)o(ted)75 1452 y(naiv)o(ely)m(,)i(with)h(blo)q (c)o(king)g(comm)o(unication.)158 1502 y(An)20 b(imp)q(ortan)o(t)f(v)n(arian) o(t)g(is)h(the)h(case)g(where)h Fi(send)p 1020 1502 V 14 w(buffer)f(=)h(recv) p 1319 1502 V 14 w(buffer)d Fo(\(this)i(can)f(either)h(b)q(e)g(a)75 1552 y(sub)q(case)c(of)f(the)g(general)g(send)p 578 1552 13 2 v 16 w(recv)h(function,)f(or)f(a)h(separate)h(function,)f(if)f(send)h (bu\013er)h(and)f(receiv)o(e)h(bu\013er)75 1602 y(are)g(required)g(to)f(b)q (e)h(disjoin)o(t)e(in)g(the)i(general)f(function\).)25 b(This)16 b(in)o(tro)q(duces)i(new)e(functionalit)o(y)f(that)h(is)g(not)75 1651 y(a)o(v)n(ailable,)11 b(otherwise,)k(and)e(whic)o(h)h(seems)g(to)g(b)q (e)g(often)g(required.)158 1701 y(An)g Fi(exchange)e Fo(is)i(the)g(sp)q (ecial)h(case)f(where)i(source)f(=)f(destination.)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF >From weeks@mozart.convex.com Wed Jun 9 15:10:59 1993 Return-Path: Received: from ens.ens-lyon.fr by lip.ens-lyon.fr (4.1/SMI-4.1) id AA15096; Wed, 9 Jun 93 15:10:57 +0200 Received: from convex.convex.com by ens.ens-lyon.fr (4.1/SMI-4.1) id AA12736; Wed, 9 Jun 93 15:09:15 +0200 Received: from mozart.convex.com by convex.convex.com (5.64/1.35) id AA01714; Wed, 9 Jun 93 08:08:21 -0500 Received: by mozart.convex.com (5.64/1.28) id AA28959; Wed, 9 Jun 93 08:10:32 -0500 Date: Wed, 9 Jun 93 08:10:32 -0500 From: weeks@mozart.convex.com (Dennis Weeks) Message-Id: <9306091310.AA28959@mozart.convex.com> To: Jack.Dongarra@lip.ens-lyon.fr Subject: May 13 mpi mail #1 of 1 Status: R (received from walker, as I recall; I believe he posted it to mpi but I didn't receive it, so eventually he mailed it directly to me - DW) Minutes of the Message Passing Interface Forum Dallas, Texas March 30 - April 2, 1993 The MPI Forum met March 30 - April 2, 1993, at the Bristol Suites Hotel in North Dallas. This was the fifth meeting of the MPIF and the third of the now regular meetings in Dallas. There were both general meetings of the committee as a whole and meetings of several of the subcommittees. For the first time a number of formal votes were taken at this meeting. All of these are recorded in these minutes (and can be found by searching for VOTE) and have also been published (to the mpi-core mailing list) in a summary of all the formal votes and all of the straw votes for the committee as a whole. The notes for these minutes were taken by Bob Knighten (knighten@ssd.intel.com) and Rusty Lusk (lusk@mcs.anl.gov). These minutes are quite long. If you want to see the important topics you can search for --- and this will quickly the lead to each topic (and a few other things.) Attendees: - --------- Joe Baron IBM Austin jbaron@vnet.ibm.com Eric Barszcz NASA Ames barszcz@nas.nasa.gov Harry Scott Berryman Yale Univ. berryman@cs.yale.edu Rob Bjornson SCA bjornson@sca.com Lyndon Clarke EPCC, U. Edinburgh lyndon@epcc.ed.ac.uk James Cownie Meiko jim@meiko.co.uk Jack Dongarra UT/ORNL dongarra@cs.utk.edu Anne C. Elster Cornell U. elster@cs.cornell.edu Sam Fineberg NASA Ames fineberg@nas.nasa.gov Jon Flower ParaSoft jwf@parasoft.com Ian Glendinning U. of Southampton igl@ecs.soton.ac.uk Adam Greenberg TMC moose@think.com Bill Gropp ANL gropp@mcs.anl.gov Leslie Hart NOAA/FSL hart@fsl.noaa.gov Tom Haupt Syracuse U. haupt@npac.syr.edu Rolf Hempel GMD hempel@gmd.de Tom Henderson NOAA/FSL hender@fsl.noaa.gov C. T. Howard Ho IBM Almaden ho@almaden.ibm.com Steven Huss-Lederman SRC lederman@super.org Rusty Lusk ANL lusk@mcs.anl.gov John Kapenga Western Michigan U. john@cs.wmich.edu Bob Knighten Intel SSD knighten@ssd.intel.com Rik Littlefield PNL rj_littlefield@pnl.gov Peter Madams nCube pmadams@ncube.com Arthur B. Maccabe U. of New Mexico maccabe@cs.unm.edu Oliver McBryan U. Colorado mcbryan@cs.colorado.edu Dan Nessett LLNL nessett@llnl.gov Steve Otto Oregon Graduate Instiute otto@cse.ogi.edu Peter Pacheco U. of San Francisco peter@sun.math.usfca.edu Paul Pierce Intel prp@ssd.intel.com Sanjay Ranka Syracuse U. ranka@top.cis.syr.edu Arch Robison Shell Development robison@shell.com Mark Sears Sandia mpsears@cs.sandia.gov Anthony Skjellum Mississippi State U. tony@cs.msstate.edu Marc Snir IBM, T.J. Watson snir@watson.ibm.com Alan Sussman U. of Maryland als@cs.umd.edu David Walker ORNL walker@msr.epm.ornl.gov Dennis Weeks Convex weeks@convex.com Stephen Wheat Sandia NL srwheat@cs.sandia.gov Wednesday, March 30 - --------- -------- - ------------------------------------------------------------------------------- General Meeting - ------------------------------------------------------------------------------- Jack Dongarra called the meeting to order at 1:30. The first topic for discussion was the agenda. David Walker had mailed out the following Provisional Agenda for MPI Meeting, March 31-April 2, 1993 Wednesday 1:30-6:00 Discussion of Snir, Gropp, Lusk point-to-point proposal (everyone) (Snir) 6:00-7:30 Unofficial dinner break 7:30-10:30 Break up for subcommittee meetings Thursday 9:00-12:00 Discussion of Snir & Geist collection communication proposal (everyone) (Otto?) 12:00-1:30 Lunch (provided) 1:30-3:00 Full group meeting for presentation of alternate approaches to groups and contexts, dynamic vs. static process models, and other issues (Volunteer?) 3:00-6:00 Full group meeting for presentation of process topology subcommittee ideas and proposals. (Hempel) 6:00-8:00 Dinner (attendees pay, but hotel provides transport to area restaurant) 8:00-10:00 Continued informal subcommittee meetings if necessary Friday 9:00-11:00 Full group meeting with the intent of taking binding votes on point-to-point and collective communication proposals, or sending proposals back to subcommittees for revision. (Snir?) 11:00-12:00 Full group meeting for defining timetable for producing MPI (or subset) by deadline in July. (Dongarra) Following the discussion on the mpi-core mailing list the question was moving the discussion of the timetable for producding MPI to the beginning of the meeting. After a brief discussion it was decided to proceed first with reports from the Communication Context and Point-to-Point Communication subcommittees in order to have a basis for discussing the schedule. The schedule was discussed on Thursday afternoon, following the completion of the Point-to-Point subcommittee report. The Context subcommittee was alloted two hours, from 1:30 to 3:00, with the Point-to-Point Communications subcommittee scheduled from 3:00 to 6:00 and on Thursday morning. - ------------------------------------------------------------------------------- Report From the Communication Context Subcommittee - ------------------------------------------------------------------------------- Tony Skjellum presided. There was a large volume of activity on the mpi-context mailing list before this meeting and so there were five proposals available for consideration, labeled: I (Marc Snir) III (Tony Skjellum) VII (Lyndon Clarke/Rik Littlefield) VIII (Mark Sears) X (Tony Skjellum/Lyndon Clarke) 25 minutes was alloted for presentation of each of these in the order I (Marc Snir), VII+X (Lyndon Clarke), VIII (Mark Sears) , III (Tony Skjellum), X followed by general discussion. Tonight there will be a subcommittee meeting to produce a single proposal. Proposal I (Marc Snir) - ---------- ----------- {Marc used overhead projector slides and these notes are largely a transcription of those slides.} Group=Context Proposal Goals: + Keep it simple (and keep MPI small) + Keep it efficient Minimal needs: + Protection mechanism + Local name space Group = Context = Ordered set of processes. - -- Method for protecting communication between e.g. libraries. - -- + All point-to-point communication is WITHIN a group and uses (group,rank) address. + All collective communications by a group (which is a context) OPERATIONS: + Group copy + Group partition + Group creation by list + Group deletion + ALL group prexists Group handle has only local (i.e. within group) use and meaning. - -- There is no reason to pass the handle of a group outside the group - it has no use. - -- 1. Impact on "current practice": Need additional argument ALL in all p-p calls. 2. Overhead for p-p send - no impact when ALL use. One lookup for other groups. receive - context id match Overhead at creation Loosely synch collective communication within group affected Storage Member table (good protection) - -- 3. Compatibility with dynamic process creation and deletion Process creation/deletion requires same for group {What is ALL group after process creation or deletion?} 4. Interaction with topology Group has no topology information (but it can be used as a peg for such information.) 5. Inter group communication (e.g. client-server models) +-------------------------------------+ +---+ ---> (---) | * | | * | | * | +---+ +--------------------------------------+ Do the communication within an encompassing group. - - + Encompassing group needed for protection + May not be convenient for naming (e.g. send(server[5], ...)) + Inconvenience does not warrant change in p-p layer + Can be handled by creating and explicityl passing arrays of ranks: MPI_LIST_RANKS(list, subgroup, group) returns the list of the rank of each subgroup member within group - -- Discussion: How have available both subgroup and group? There must be an encompassing group which has full knowledge, e.g. server that is member of both the group and the subgroup. This proposal is orthogonal to question of attaching additional information (e.g. topology information, caching, etc.) Can't deal with situation of contacting an independent pre-existing server. Marc's approach is that dynamically adding processes requires dynamically creating a group containing all of the processes. Opacity vs. accesibility to mechanism. ============================================================================= Lyndon VII and X VII: Context is a higher level mechanism than a group. It is basically a unique identifier together with a reference to a group. This means that as a group changes, all contents that reference that group change as well. Same ability to hang on facilities (e.g. topology, caching) as others. Relation to p-p: Three forms: "closed form", "null form", "open form" Open form is to allow communication between different groups. Experience is that creating encompassing groups is difficult. Disagree with Snir's claim. Addressing is via (context, rank). Need a "context allocation" mechanism - this implies global communication. Relation to c-c: Works very cleanly for all-all using closed form. Two group communcations - for MPI-2 Discussion: How to establish communication between groups? Can send context. What is opaque? Lyndon - not important. Startup/bootstrap - everyone starts in the ALL group. Can use common ancestor or name registry. Power relative to I? Lyndon claims that this is more convenient once communication is setup. Basic idea is to be able to communicate via (my_context, remote_context, rank) ORTHOGONAL ISSUES - caching, tag selection, transfer of ???? X: Attempt to synthesize III & VII A CONTEX is a space of tags. A GROUP is a set of process references. [What does this mean?] Idea is to give method for combining groups and context for purpose of communication. COMMUNICATORS (see pp. 3-4) Silly names, but a serious proposal. Floopy - arbitrary communication between processes allowing wild card on tag. Bongo - basically like Marc's proposal - communication within a group using rank naming ("closed") Bingo - communication between groups. Question: Why do we want ANY of these proposals? Performance, build large scale software safely We need examples for all of these proposals! Collective -> group; express collective in terms of p-p implies need to discriminate message -> context. But there are reasons to have groups and contexts that have nothing to do with collective communication. Argument that in Marc's proposal the need for a context means have to have an additional group - but is this a problem? Lyndon argues that there are good reasons to separate group and context. Static vs. dynamic groups. Ability to move context. Proposal VIII - Mark Sears {{{ MPI-1 char * MPI_ALLOCATE Group and Context Proposal Number(); }}} - -- Context and Groups are orthogonal + Orthogonal purpose + Orthogonal functionality + Orthongonal implementation - -- Contexts Purpose - promote software modularity by allowing construction of independent tag spaces Definition: A CONTEXT is an integer-valued extension to the tag component of the meggage envelope, and must match exactly between sender and reciver - -- Model: + Contexts are global. + No concept of process belonging to a context. + Contexts are scarce resources (16). + Context allocation is a rare event. + MPI p-p requries no reference to groups. - -- Context allocation/deallocation ALLOCATION: int mpi_getcontext() + called synchronously by all (EVERY SINGLE ONE) processes + signals to mpi that use of context is now allowed DEALLOCATON void MPI_free_context(value) MPI_DEFAULT CONTEXT + Preallocated; can't be freed. + Solves initialization + Free-for-all But believes that allocation/deallocation not truly needed - could have entirely static system. + Contexts are global + No concept of process belonging to a context + Context are scarce resources (e.g. 16) + Context allocation is a rare event + MPI p-p requires no reference to groups - -- GROUPS Purpose - provide tools for organizing subsets of processes in a parallel task (i.e. MPI program.) Definition: A groups is a 1-1 mapping from (0..n-1) to another set of integer. A group is a collection of processes only in so far as the elements are process addresses. Groups have no associated have no associated context or tags, default or otherwise. - -- Group Implementation + local to each process based on information needed to construct the mapping + Group type is local and opaque + Groups can be sent in message only by sending the information needed to construct the group. + Groups are objects in the OOP sense - -- Usage: MPI_SEND(n,buf,process,tag,context) MPI_BROADCAST(n,buf,group,tag,context) Group identifier is a local opaque type, thought of as a pointer to one of many possible group structures. MPI_SEND(..., element(group,rank),tag,context) GROUP FUNCTIONS int order (group) int range(group) int element(group, int rank) int iselement(group, int element) int rank(group, int element) CLASSES identity, permutation, linear, list, bilinear, composition, cartesian CONSTRUCTORS group makelineargroup(order,start,delta) - -- Two kinds of 3rd party code 1. Code that inherits context and tag space from caller. Example: MPI collective communication 2. Code which allocates and manges own context and tag space. MPI should allow both of these. - -- Topology Global topology - mapping of processes to processors. provide inquiry function returning a string describing this mapping: char * MPI_global_topology() Examples of output: "N 564" - random network of 564 processes "H 5" - 5 dimensional hypercube "R 2 16 13" - 2D mesh, 16x13 Local topology - implicity within group; no additional functions needed. - -- ADVANTAGES + Ease of implementation + Close to hardware + Good use of resources + Flexibility in implementation of higher level concepts + MPI p-p requires no reference to groups + MPI c-c can be layered on top of MPI p-p - -- Discussion - Serious problem with global communication. This destroys the software modularity. How to do global operations using groups? Responsibility on code to insure there is disambiguation, synchronization, etc. Tony - III + Tag/context partionting message space for "safe" software + Groups encapsulate scope of operations for 1) notation 2) optimization 3) performance ........... ............... / . GROUPS . . . / ......... . CONTEXTS . / lower,higher? ............. Relation between groups and contexts? Groups can be orthogonal except for group creation. - -- Forms of communiations + collective communication ON groups (compatible with I) + p-p A] (group,rank,tag) - analagous to I B] (context,pid,tag) Models 1) Contexts created/destroyed 2) contexts can be published dynamic server implied or shared address area Contexts & groups interelate when creating new groups not necessarily from LCA ^^^{{{WHAT DOES THIS MEAN?}}} - -- contexts & groups interrelate when creating new groups, not necessarily from LCA (what does this mean?) Dispute/reply regarding optimization. Argument that group can be used to provide to information about special situations (e.g. shared memory) that can be used for optimization. Dynamic groups more feasible using III or VII. Can contexts be sent in III? Yes. VII more complex than III because it offers more layers. Tony believes that dynamic groups are essential for the heterogeneous case and so believes that I is inappropriate. John K. notes that other proposals can be built on top of VIII. Proposal to defer straw vote until tomorrow to give people time to ponder. Is global synchronization an essential part of VIII? Sears - no, there are various possibilities. Why is it important that contexts are global? Because a context is not associated with a group of processes. This looses much of the safety. It also looses the local addressing within a group. Sears argues that this complicates p-p, but Snir says that something like his proposal has been implemented and is not complicated or expensive. What level of protection is needed/desirable? Picture of using context for safety - at startup send a context to each library used which it then uses for internal safety. Problem - libraries using libraries using libraries, etc. Importance of receiving on wild cards and relation. Host/Node model? Must support this and all do. (What about loading program - not part of THIS discussion.) Subcommittee will meet tonight and present a more unified proposal tomorrow morning. Rik asked for example showing how to implement safe barrier using p-p, group, context. Adam - How can we evaluate these proposals if we have not agreed on what we want from the context concept? - ----- break 4-4:20 pm - ----- - ------------------------------------------------------------------------------- Report From the Point-to-Point Communication Subcommittee - ------------------------------------------------------------------------------- First reading of p-p proposal - Marc Snir presiding Presentation with some minimal assumptions about use of context. No language binding included. Use of handles and opaque objects. [Ignore text comments on implementation - issues for language experts.] Discussion of ephemeral/persistant. Greenberg asked about "error to free a handle with pending operation" - change to "handle becomes ephemeral" or some such. How deal with lists - included length, separate length, EOL marker. States [Implementation again - not part of current proposal] If have only one send/receive, what is type of buffer and what to do about this? Marc's proposal is to accept breaking F77 in this case. Skipping 2.4 (Contexts) for now (including error handling.) 2.5.1: What about C/Fortran compatibility for messages? Also skipped for now. 2.6: Greenberg wants at least escape hatch to allow functions (MPI_ADD_?) that add, e.g., other F90 objects. Discussion of len in MPI_ADD_BLOCK as "Number of elements" rather than as bytes. There are language and portability issues. Marc mentioned issues in middle of p. 14 (negative displacements, ...) and notes that these must be settled. Note that delete/commit functions discussed last time are not in this proposal. 2.6.1 (Data Conversion) It just happens - this needs to be stated clearly. - -- back to 2.2 - vote on each section [[NOTE: A proposal, in the form of a proposed chapter is offered. Votes are on amendments and on accepting particular parts of chapter.] How are handles allocated? User or system. There are efficiency advantages to user allocation, e.g. on stack. PRP argues for a create for each data type rather than a generic MPI_CREATE. At the moment there is an admixture in the proposal. ====== Discussion of voting rules. Organizations voting: 24 ===== VOTE: Separate mpi_create for each type - ---- Yes: 20 No: 0 Abstain: 4 Possible meanings of free: free 1.1 can always be done 1.2 can be done only if user will not use handle no pending operation after free 2 if no pending operation, deallocate otherwise free when done 1.2 = current proposal; 2 is Greenberg proposal Greenberg explantion - common thing done in their system is to have handlers with free only taking effect after handler completes. PRP - Has primitive that does essentially 2 (which is not "mark ephemeral") It does not imply cancel. Fire and forget. Cownie: Two arguments - Paul's fire and forget; Adam's handler for messages. Snir: Is buffer available after free? PRP/Adam: No. Why not just have another kind of free? Multiplicity of functions. VOTE: mpi_free is valid even if handle is in use, but effect is to - ---- free the object when the operation completes. Yes: 7 No: 8 Abstain: List of handles - change name to array of handles VOTE: list_length explicit (rather than included in array or EOL)? - ---- Yes: 22 No: 0 Cownie wants a method to provide cheap allocation, e.g. on user's stack rather than in system heap. Need a concrete proposal. RLK wants explict statement of what is erroneous, checked, etc. VOTE: Accept 2.2 - ---- Yes: 23 No: 1 2.3 & 2.4 skipped -- they will be considered elsewhere. 2.5 Cownie - proposes that there be both CHAR and BYTE data types rather than just the basic data types of the host language. Another proposal is to have an MPI_STRING data type. But what length? Null terminated? VOTE: BYTE data type. - ---- Yes: 24 No: 0 Pending action of Context Subcommitte is the content of the envelope. VOTE: Accept 2.5 (minus 2.5.2) - ---- Yes: unanimous Proposals for units of bytes and for consistency. 1. Units are elements 2. bytes in C and elements in F77 (No one favors this. Against: 14) 3. bytes everywhere (and provide some way of getting size of basic types) a) index stays as is, but bytes elsewhere Pro: 9 Con: 13 (VOTE 1) b) truly bytes everywhere, including indices Pro: 10 Con: 8 (VOTE 2) Greenberg will bring forward additional proposal for 2.6 at next meeting. VOTE: In vector, stride may be negative? - ---- Yes: 10 No: 2 VOTE: Allow repetition? - ---- Yes: 13 No: 1 VOTE: Can multiple components overlap? - ---- vote not taken PRP proposes that in vector len is a count of the number of blocks. VOTE: total length is an integer multiple of the block size - ---- Yes: 4 No: 10 Tony proposes adding a COMMIT operation. Note: Current proposal does not clearly state it, but a handle cannot be modified after it is in use. There is nothing that explicitly specifies that the modifications are complete. VOTE: Commit? - ---- Yes: 11 No: 6 VOTE: Accept 2.6 - ---- Yes: 12 No: 4 - ----- Break for dinner - 6:15 p.m. - ----- Subcommittees tonight: Here: Vendor caucus Rm 1: Process topology Rm 2: Collective Communications Subcommittee Rm 3: Context Subcommittee BAR: formal, language binding, profile meet at 8 p.m. ----------------------------------------------------------------------------- Revised agenda Thursday 9 p-p (cont) 12 lunch 1:30 (registration) 1:30 collective communication 3 process topology 6-8 dinner Future: May 12-14; June 23-25; August ? ============================================================================= Thursday, April 1, 1993 9:10 a.m. - - ------------------------------------------------------------------------------- Report From the Point-to-Point Communication Subcommittee (continued) - ------------------------------------------------------------------------------- Point-to-Point Communication - First Reading (continued): - -------------- ------------- ----- ------- Marc Snir presiding 2.7 (Receive Criteria) & 2.8 (Communication Mode) - ------------------------------------------------- Suggestion that we need more than one DONTCARE, a SOURCE_DONTCARE and a TAG_DONTCARE. Lyndon Clarke proposed a "secure" communication mode where the send that will return once the system can guarantee that the receive will actually complete. (During the discussion of this there was a suggestion that the word REGULAR on p. 16 of Marc Snir's draft be changed to STANDARD so first can always be used. Marc remarked that he welcomes all manner of stylistic improvements, but asked that they be sent to him, not voted on here.) Why such a secure communication mode? To provide a portable manner to write programs that are guaraneed to be safe, even without buffering. This is similar to but weaker than the synchronous functions that were rejected in a straw vote last time. The unease with a proliferation of functions was again mentioned. Adam Greenberg suggested that one could be against this and still specify equivalent function by requiring no buffering throughout the system. Rik Littlefield (as a pseudo Tony Skjellum) proposed receive criteria based on an intag and mask. Variations on this have been discussed in the past. VOTE: Typed DONTCARE? - ---- Yes: 19 No: 1 After the vote, a count showed 25 organizations present. Proposal: Receive selection based on (tag & mask) = (intag & mask) Why do this if have context? Discussion of efficiency. PRP proposes sending tag (exact match by system) and extra-info (for recognition of message category by application) in envelope. Cownie unhappy because of effect on latency of small messages. Alternatives being considered: (1) RECEIVE(..., tag, info, ...) no DONTCARE for tag (2) RECEIVE(..., tag, mask, ...) no DONTCARE for tag (3) RECEIVE(..., tag, ...) DONTCARE for tag Does this imply small tags? PRP: reasonable for implementation to limit size of tag, category, and context. Greenberg argues againt PRP propsal because of existing practice and because it could force user to duplicate some system function. Eric argues in favor of wildcarding because it often allows reuse of buffer and so fewer resources. VOTE: (1) fails for lack of second. - ---- (2) Yes: 6 No: 12 so (3) remains. Proposal: Secure communication mode. Rusty - P4 experience argues against this. Cownie - this is useful for reasoning about program correctness. An alternative is to have a "secure mode". But what happens if using secure mode, but some library may have been written without secure mode? VOTE: Do we want a "secure" communication mode? - ---- Yes: 7 No: 6 VOTE: Delete send/ready_receive? - ---- Yes: 10 No: 10 [Amendment tabled] VOTE: Accept amended versions of 2.7 & 2.8 - ---- Yes: 25 No: 1 Discussion of 2.9 (Communication Objects): - ----------------------------------------- Marc Snir began with an overview of the four subsections. He asked if there should be handles for user space objects? This is not in the current proposal. He noted that the use of a STATUS_ALL compared to using STATUS in a loop is one of convenience. Jim Cownie argued: WAIT_ANY should be told where to start scan because of fairness concerns. [Discussion - this is a strong implementation requirement. MPI requires "fairness", though the requirement is so weak as to be untestable. WAIT_ANY is small part of the fairness problem. Easier for user to pass in this information than for the system to maintain it. The user can always guarantee fairness. But at some cost. Postpone until we discuss correctness and fairness in general.] The current draft has MPI_RETURN_STAT providing the free space in the buffer. The current practice appears to return the number of bytes received. Issues: Byte count of data received; Where do handles come from; Partially specified handles; Make explicit that can a message can be sent to oneself. VOTE: The return status for a receive operation should be the number - ---- of bytes received Yes: 17 No: 3 Abstain: 7 The return status handle should be allocated by the user: - -------------------------------------------------------- Typically the user knows exactly the what needs to be done and the lifetime is often short. Thread safety says can't use global storage, so ... Suggestion that there be an overall proposal to deal with handles in user space. But this is not a general handle - this is a special situation. Postpone for general consideration. Partial handles - --------------- return_status_handle + part of communication handle + separate user space object + separate system object VOTE: Accept 2.9 (with amendments) excluding 2.9.1 and WAIT_ANY - ---- Yes: 19 No: 2 Abstain: 3 - ----- break 10:48 - 11:08 - ----- Discussion of sections 2.10-2.12 - -------------------------------- Marc presented the following table for discussion: |general|contig|vector|contig| |buffer |byte |byte |type | - --------------+-------+------+------+------+-------------------------------- blocking | | | | | send | * | | | | receive | | | | | - --------------+-------+------+------+------+ blocking | | | | | ready-receive | * | | | | send | | | | | - --------------+-------+------+------+------+ immediate | | | | | send | * | | | | receive | | | | | - --------------+-------+------+------+------+ immediate | | | | | ready-receive | * | | | | send | | | | | - --------------+-------+------+------+------+ secure | | | | | send | | | | | receive | | | | | - --------------+-------+------+------+------+ Same buffer types are used in Collective Communication. Discuss probe Issues: Counting units? Vector type? Contig type includes contig byte because have byte type? Lyndon - There should be a secure-receive (for optimization of the protocol.) Marc - Is system or user responsible for insuring this works? Lyndon - User. Various - What is value of secure-receive. Gropp - Because these are different protocols loose performance if don't have different functions as general must always deal with worst case. Marc: error? ssend(2) ---------------------------- recv(1) Proposal is that it is erroneous to attempt to receive a message sent by a secure-send by other than a secure-receive. Rusty argues that this should not be erroneous because ??? Possibilities: 1) secure-send {can, cannot} be received by receive 2) Enforced by MPI or user responsibility Adam argues that this is confusing the secure-send features with pre-acknowledgement and these are independent issues. The purpose of secure-receive is entirely performance - it does not have any semantic content. secure-send/secure-receive can be implemented on top of regular p-p. Note that the proposal is to add 2 secure-sends, 2 secure-receive VOTE: Add both blocking and immediate secure-send/secure-receive - ---- with failure of program to match being erroneous. Yes: 10 No: 8 Abstain: 9 VOTE: 2.10 as amended - ---- Yes: 20 No: 4 Abstain: 3 VOTE: 2.11 - ---- Yes: 26 No: 0 Abstain: 1 2.12 - ---- Are blocks just blocks of bytes or are they typed? How do count the size of blocks? Lusk - Want to have blocks of typed data for use in a heterogeneous environment. PRP - Offers precise propsal that use the same parameters as in MPI_ADD_BLOCK. VOTE: Use exactly the same parameters in MPI_ADD_BLOCK as amended - ---- Yes: 26 No: 0 Abstain: 1 Adam - Proposal to have functions to have strided messages using blocks. Rusty - Argues against becasue of problem of proliferation at this level. [Arguement was clearly on matter of taste. This is syntactic sugar as the most general low level routines certainly can be used for this.] VOTE: Have strided block message functions. - ---- Yes: 5 No: 9 Abstain: 12 - ----- lunch 12:00 - 1:35 - ----- Continuation of discussion of Chapter 2. Proposal that more time discussing 2.10-2.14 needed, so p-p subcommittee will meet after dinner tonight. Discussion of Schedule - ---------------------- Future meetings 4. May 12-14 set 5. June 23-25 set 6. August 11-13 tent. 7. September 22-24 tent. Draft to be available: November 15-19 SC '93 Portland Reading Schedule - ---------------- P-P Snir April & May Collective Otto/Guist April & May Profiling Cownie April & May Process Topolgy Hempel May & June Environmental Mgmt Gropp May & June Lang. Bind. gen. Berryman May & June Context Skjellum May & June Formal Spec Zenith June & August ??? Specific language bindings will follow general material by one meeting. Where is the language binding material? There will be a general principles chapter and also the actual bindings. These will be separate votes. General language Berryman June Specific lang. bind. Berryman Is anything coming in the Formal Spec? Does anyone care. Rusty - one participant and two observers. Zenith told Rusty that he was working on something, but nothing has appeared and he is not here. What about public comment? Discussion of the HPFF model of two opportunities for comment. Proposal to have only one round of public comment with draft released to public at Supercomputing '93. Reference implementation - Gropp/Lusk effort Test suite - Greenberg and Haup will lead an effort Subset - "Implementation Order Recommendation" Huss Goals + Define a reasonable subset of MPI that is recommended for initial implementation + Only a minimum - welcome to implement more + Allow MPI to begin to show up in a timely fashion while still consistent across vendors + Consistent with complete standard - -- Method + Create new "subset committee" + Write an Annex (like HPF) + Present flushed out proposal/annex at next meeting + Hope "other" committee will create initial test suite for minimal implementation :-) Might motivate implementors - -- First shot list + want subcommitte members + Begin discussion via e-mail - ----- NOT IN SUBSET 1 No persistent handles 2 No multicomponent buffer descriptors {only one item described} 3 No indexed component 4 No waitany or waitall 5 No name server model If interested send mail to Huss - ------------------------------------------------------------------------------- Report From the Collective Communication Subcommittee - ------------------------------------------------------------------------------- Collective Communications - First Reading - ----------------------------------------- Steve Otto Subcommitte of three met last night (Otto, Ho, El...) Propose a discussion about safety, semantics and function of collect c NOT: Contexts/groups NOT: detailed questions of data types, lan. bind, p-p Semantic Warm-up - ---------------- Barriere implies a time synchronization of processes, but NOT an emptying of all message buffers. I.e. p-p messages may span (in time) a barrier: 1 2 post-send(2) post-receive(1) barrier barrier complete-send(2) barrier does not imply that message queues were emptied. Object to statement on p. 8. {{{QUOTE}}} - -- If we want collective function that does ensure all messages queues were emptied, let's invent it. wait-for-all-global Example 3. p. 46 Is this safe? 0 1 bcast /--------------- receive(0) send(1) ------------/ bcast Does this deadlock? Depends on implementation of bcast If bcast implemented with buffered contenxt-unique messages, probably won't deadlock If bcast synchronizes strongly, probably will deadlock Otto unhappy about defining sematics of collective communication routines in terms of operational p-p routines because this depends on side effects such as emptying message queues. - -- Cownie/Snir argue that this is not true because ... {{{MISSED}}} - -- Conservative Proposal Instead of mandating that example above is "safe", we propose that no messages are allowed to be "in the air" upon entry to collective communication call. So: We require that c-c routines be used AS IF they implied barrier synch but === the user cannot assume that they actually provide barrier synch - -- Confusion about what is actually intended. Unhappiness about phrase "in the air". The example above is unsafe, but is it erroneous? Claim is that this is unsafe, but an implementation that allows it is compliant. Now what is the situation of the example on p. 46? It is there because Marc want people to be aware that the behavior of a valid (if unsafe) program may be surprising. - -- Related point: May want to have a "barrier mode" for c-c so that they do behave as barriers when this is on. Useful - can detect many erroneous programs. - -- Keeping multiple c-c's separated - -------------------------------- /group 1 /collective operation 2 [ ( ] ) \collective operation 1 ^ \group 2 | can messages in here to go to wrong destination? Intersecting spanning trees ... can they catch each other's messages It seems that good implementations can be constructed so that they don't. ==> NOT dependent on user-tags TAGS ARE SUPERFLUOUS FOR C-C + keep for consistency with p-p? + but, what is the implemntor supposed to do with them? + ignore them? + what do we tell users about tags in c-c? We propose - NO TAGS in c-c! - -- Why have them - for debugging. Anything else? Does lack of tags affect non-blocking c-c? Yes. Steve Huss: Want to make sure that system must insure that successive broadcasts will never result in confusion of messages. Marc: What is the exact semantics of c-c, in particular when is the burden on the user of MPI to insure no ambiguity vs.l when this is guaranteed by the system? This is particularly important if generalize to allow non-blocking c-c. PRP: Might want to guarantee that ordering is guaranteed between c-c. Snir: Certainly want to guarantee that if parameters in c-c are different, then there is no ambiguity. Further may guarantee even if have same parameters. Proposal - no tags and no ambiguity (so matching is via parameters and sequencing.) [System responsiblity vs. user responsiblity] Steve: Wants MPI to guarantee preservation of order Kapenga/Pierce: Order may not be meaningful Gropp: Already have similar issue for p-p. Snir: So what is behavior when order is not meaningful? Cownie: Kapenga made important point that multithreaded libraries cannot use c-c without context information. Various: MIght be able to do this by using duplicated groups. Again, issues is what responsibility falls on system and what on user. One possible solution is to require user to provide ordering of c-c. - -- Functions - --------- bcast barrier gather global reductions cshift/eoshift scans all-to-all bcast index - -- 3.1 Introduction - ---------------- Note that non-blocking c-c is not included VOTE: Include non-blocking c-c? - ---- Yes: 0 No: uncounted Abstain: 9 Note that groups carry no topology information. Weeks proposes adding a perfect shuffle c-c function. Ranka proposes adding a permutation c-c function. Mention of variations on this. Rik observes that the arbitrary buffer descriptor versions will be extremely complex in implementation. He also objected that this cannot be implemented using p-p. Marc responed that it is possible using p-p because can send message to oneself. Discuss this in detail when we get to the reduction section. 3.2 (Group Functions) & 3.3 ( - ----------------------------- LATER 3.4 Synchronization - ------------------- Tag is removed, so examples are now wrong and must be replaced. Jon wants the examples removed as they contain semantics that are not guaranteed. Marc suggests moving to appendix with the hope that this would eventually contain a full specification of c-c in terms of p-p. Agreed. VOTE: Remove tag in all c-c? - ---- Yes: 15 No: 5 Abstain: 5 VOTE: Accept just semantics of 3.4 - ---- Yes: 23 No: 0 Abstain: 2 - --- break 3:15 p.m. - 3:35 p.m. - --- Ho has paper on collective-communication library. He will make it generally available. 3.5 (Data move functions) - ------------------------- Otto: max size to shift? NO Issues: + What if inbuf is outbuf? + Periodicity in topology? [Hempel] + Tie topology to chsift [Hemple] + cshift as sendreceive(source, destination, ...) [Flower] Allowing inbuf and outbuf to be the same violates F77. Rik - user-level double buffering a pain. various: back and forth on responsibility of user vs. system. Steve Huss: Proposes a cshift with only 1 buffer. Disallowing partially overlapping buffers. Note that this new cshift will have an INOUT argument. Does this same argument apply to other reduction functions? Have to check one by one. VOTE: Allow 2nd cshift with only one buffer. No partial overlap on - ---- orig cshift Yes: 21 No: 0 Abstain: 6 Note that earlier encompassing vote imples that we will have types in cshift. Marc remarks that changes to measuring size in bytes affects statement in text that buffers have same number of units. eoshift - ------- Second single buffer form? zero filling - another argument? VOTE: Accept cshift, eoshift proposals with amendment of 2nd form of each. - ---- Yes: 20 No: 2 Abstain: 4 bcast - ----- VOTE: Accept bcast - ---- Yes: 22 No: 0 Abstain: 2 gather - ------ Note - len is number of bytes, not what is written. Long general, rambling discussion of gather. Proposal - separate IN argument to gather of sizes; separate functions to find sizes. Proposal - version of gather with list of outbufs Proposal [Flower] - all-to-all gather(???) Cownie moved to direct gather and scatter back to subcommittee for further consideration. Accepted. 3.6 (Global Compute Operations) - ------------------------------- Issues: + inbuf=outbuf problem? 2nd version + have types, so don't need (R,I)MAX, etc. + return value to all? 3rd version + maxloc (etc.) return location and value + restriction on user defined functions + vectorized user function Returned to committee for further work. scan - nothing to say for today correctness - nothing to say for today Finished with collective communication for today. - ------------------------------------------------------------------------------- Report From the Process Topology Subcommittee - ------------------------------------------------------------------------------- Rolf Hempel presided. No presentation, but need to discuss the direction to go. First question - is topology going to be part of group management at all? Rolf remarks that vast majority of applications have a natural topology. There are implementation efficiencies - e.g. avoiding tables for mapping to processors. Marc: User visible mapping of processes to processors is not likely to be valuable. Trend in hardware is to hide hardware topology. What is the advantage of topology? Convenient in writing programs when topology is natural to problem. This information may be useful for implementation on particular systems. What do vendors have to say about the utility of such information. Cownie: The point is to be as flat as possible. Snir: Topology can certainly be built as a superstructure. What is value of integration? Convenience, safety. Safety does not appear important and he does not see sufficient value to convenience. Major issue is relation of topologies and groups. Can store topology as part of group (current proposal) or as a superstructure (Marc's preference.) Certainly convenient to have standardized method to build e.g. row group, column group, etc. This is what is in current draft, but the issue is one of integration. But is it more efficient or even substantially more convenient. Three possibilies: Not in MPI, in MPI but not integrated, integrated into group mechanism in MPI. Various repetitive arguments for each of these positions. Straw vote: Topologies in MPI? - ---------- If yes integrated with groups (e.g. eoshift), OR separate library, environmental inquiry In: 25 Out: 4 Abstain: 5 integrated: 4 Separate: 23 Abstain: 7 Back to discussion in the subcommittee. Meeting tonight: p-p room 1 c-c room 2 ============================================================================= Friday, April 2, 1993 - ------------------------------------------------------------------------------- Report From the Profiling Subcommittee - ------------------------------------------------------------------------------- Profiling - Jim Cownie - ---------------------- Based on draft section ?? of document. This is a single level approach and is basically static, i.e. based on selecting actual functions at link time. Questions: 1) Is one level OK? - other option is chain of function pointers 2) Debugging support? - Dump all message envelopes - Status of active handles Single level has no extra cost, but limits as might want to use the single level for another purpose e.g. as a network intermediary. Yes, that limit could be significant. Provide multi-level interface in the same manner (i.e. publishing alternative names for each level)? [Problem is setting the multiple interfaces.] Provide environmental facilities for exporting a multi-level dynamic approach? General agreement that single level is better than extra cost for everyone. Note that single level actually can support full dynamic approach as the MPI routines can be replace by functions that do function pointer swizzling. Debugging support? - ------------------ A small amount of discussion last night. What is needed and what can be provided? Besides items mentioned above, need to be able to decode the opaque objects in MPI. Useful to have a recording mechanism. - ------------------------------------------------------------------------------- Report From the Language Binding Subcommittee - ------------------------------------------------------------------------------- Scott Berryman Gross assumptions: No Fortran 90 binding C++ binding = C binding Specification says nothing about language interoperability {The confusion bomb went off!} [sender deals with native message; XDR in general; underspecified buffer descriptors - incl. lengths or incl. lang. spec.; general vs. limited translation; know transl in hetero - don't know in homo; this is not just a language issue - it is a language implementation issue][Need concrete proposals] - ------------------------------------------------------------------------------- Report From the Point-to-Point Communication Subcommittee (continued) - ------------------------------------------------------------------------------- Marc Snir Byte vs. Element Count + Need ADD_VEC, ADD_INDEX with byte displacement + Most usage will be element displacement - -- [R] [III] [R] [III] [R] [III] [R] [III] (array of records) so odd displacement - -- Have two different components + Block start len - number of elements data type + VEC start len - total number of elements stride - number of elements between blocks lenblk - number of elements per block data type + HVEC start len - total number of elements stride - number of BYTES between blocks lenblk - number of elements per block data type [///] [////] [ ][///] [////] [ ][///] [////] [ ] VEC(..., 5, 3, 2, REAL) HVEC(..., 5, 3*size_of_real, 2, REAL) INDEX start array_of_indices - element index (start has index zero) type HINDEX start array_of_indices - element displacement in bytes type HVEC, HINDEX more general but less convenient and more error prone Returned length of received messages - ------------------------------------ Number of elements IS always meaningful (even for message containing multiple type.) And number of elements sent is always the same as the number of elements received, while the number of bytes sent/received need not be the same. Moreover the number of elements is not too hard to compute. Proposal: Use the element count except for displacements in Hxx buffer components. Organizations present: 22 VOTE: proposal as above - ---- Yes: 19 No: 0 Abstain: 3 Probe - ----- 1) Use to decide where to receive message (allocate memory) 2) Use to debug. Propose to support 1. Do we need probe? Why not receive in a system buffer and return a pointer? Problems - system buffer is untyped; memory management Proposal: MPI_PROBE(source, tag, context, flag, return_status_handle) MPI_RETURN_STATUS(handle, len, source, tag) Assuming no other concurrent receive (single thread) MPI_RECV executed with source/tag returned by PROBE and same context will return message found by probe. Multithread programs need suitable critical region. What is returned in the len field? Should be the number of elements, but unfortunately don't know this without buffer descriptor. So return number of bytes? This may not be useful for deciding size of buffer in a hetero env. So possibilities: 1) len = -1 and provide DECODE(buff_desc, msg_status_handle) 2) Number of bytes ("on the wire") 3) Number of elements (cost of including this info in envelope) Could have 2) and also provide DECODE. The only inconvenience is the difference in units between DECODE and RECEIVE. Another alternative is to provide a data type as part of the PROBE rather than a buffer_descriptor - this would cover the most common case of uniform buffer_descriptors. Cost of rebuilding buffer_descriptor for receive because have to add in pointer for actual storage space. {{???Is this right}} Oliver object proposal again 4) PROBE(source, tag, context, type, return_status_handle) returning number of elements. Straw Vote: Probe? - ---------- Yes: 23 No: 2 Straw Vote: 1) probe - no length 8 - ---------- 2) probe - simple type length 17 3) probe - byte count 11 4) probe - element count 6 5) decode function 22 Cancel - ------ MPI_CANCEL(handle) Either communication succeeds or CANCEL succeeds, but not both nor neither. (This is to cancel a non-blocking send.) + Need CANCEL to recover committed resource + Implementation is not trivial Is it valid for CANCEL to always fail? No, if there is a send with no corresponding posted receive, then CANCEL must succeed. Recognition that CANCEL may be a very expensive operation. For example it may require an interrupt drive mechanism. What is the effect (cost) on normal communication? Straw Vote: CANCEL? - ---------- Yes: 14 No: 6 Type Mismatch - ------------- Suppose send 4 bytes and receive integer or conversely. Is this unsafe or erroneous? Various proposals: type mismatch is always erroneous; type mismatch is never erroneous; BYTE type is an escape hatch. Another proposal is to allow type conversion as well. Straw Vote: - ---------- 1. type mismatch always erroneous 2 2. type mismatch erroneous except BYTE 12 3. never erroneous 9 - ------------------------------------------------------------------------------- Report on MPI -1.2 - ------------------------------------------------------------------------------- Jim Cownie presented. Procedure clarification: All official votes require majority of ABSTENTIONS. Added Features Insecure send for users who lack confidence Tags are opaque objects Message data is opaque All lengths are in bits, measure as floats (for sufficient precision) Messages are not passed in envelopes (they are too small), but packing crates. Context proposal number server - returns 64 bit integer (expected to last at least 1 week) Group simplification: maximum number of elements in a group is 1 All communications occur in a group From owner-mpi-comm@CS.UTK.EDU Mon Aug 9 22:20:29 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA16990; Mon, 9 Aug 93 22:20:29 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21096; Mon, 9 Aug 93 22:18:03 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 9 Aug 1993 22:18:02 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from noc.usfca.edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21088; Mon, 9 Aug 93 22:17:59 -0400 Received: by noc.usfca.edu (5.65/DEC-Ultrix/4.3) id AA16320; Mon, 9 Aug 1993 19:20:01 -0700 Received: by mobydick.usfca.edu.usfca.edu (4.1/SMI-4.1) id AA01133; Mon, 9 Aug 93 19:17:23 PDT Date: Mon, 9 Aug 93 19:17:23 PDT From: peter@mobydick.usfca.edu (Peter Pacheco) Message-Id: <9308100217.AA01133@mobydick.usfca.edu.usfca.edu> To: mpi-comm@cs.utk.edu Subject: user's guide Cc: peter@mobydick.usfca.edu Salutations! If we're to have a successful standard, it seems very important that we have a User's Guide. So I thought I would try to get the ball rolling on this. I've appended a draft table of contents with a few notes. We definitely need suggestions for sample programs that we can include in the Guide. At the beginning of the Guide, it would be good to have some short, easy programs to get the user started. Later on, of course, it would be good to have some longer, more sophisticated programs. Throughout the Guide, it would be useful to have short examples illustrating various features of MPI. So send your questions, comments, criticisms . . . Do your worst, Peter Pacheco Department of Mathematics University of San Francisco San Francisco, CA 94117 (415) 666-6630 peter@mobydick.usfca.edu Introduction. [Brief discussion of history and purpose of MPI.] [Quick overview of MPI.] [Pointers to literature on parallel processing for those users who need more background before reading the Guide.] Chapter I. Basic message passing using MPI. 1. MPI_SEND and MPI_RECV. 2. Basic program. [How about calculating a definite integral using the trapezoidal rule. Include source code with mpi_init, or whatever is decided on. Will we also need to call mpi_iomode?] 3. More short examples. 4. Some short exercises. Chapter 2. Collective communication using MPI. 1. Making the basic program run faster. [Assuming trapezoidal rule:] (a) Roll-your-own global sum. (b) MPI_REDUCE. 2. Other MPI collective communication routines and examples. [Defer detailed discussion of some routines to appendix?] (a) Barrier. (b) Broadcast. (c) Gather. (d) Scatter. (e) Scatter_gather. (f) All_cast. (h) Scan. 3. Short exercises. Chapter 3. Writing larger programs with MPI. 1. Communicators (using naive user functions). 2. Topologies. 3. Writing programs that partition data. [A possible example: Solve a pde using finite differences on a mesh.] 4. Writing programs that partition control. [A possible example: Supervisor-worker type problem, e.g., find the chromatic number of a graph using the chromatic polynomial.] 5. Other Examples. 6. Estimating Performance. 7. Short Exercises [on use of communicators, etc.] Chapter 4. Debugging and Profiling. [Do we need this chapter? MPI, of course, mandates nothing about debuggers or profilers. However, if the Guide is to be useful to the infamous "naive" user, it might be a good idea to include something here.] Chapter 5. Making your program run faster. 1. A quick overview of what happens when a message is sent. 2. Non-blocking communication in MPI. 3. User-defined buffering in MPI. 4. Parallel libraries and contexts. 5. Exercises. Chapter 6. Everything you always wanted to know about MPI. 1. Environmental enquiry functions. 2. More on communicators, groups, and contexts. 3. Exercises. [Do we want a section on what's NOT in MPI, e.g., parallel i/o? It also might be appropriate to include a section with pointers to the literature on parallel algorithms.] Appendix A. A summary of MPI. Bibliography. From owner-mpi-comm@CS.UTK.EDU Wed Aug 11 11:04:58 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA03724; Wed, 11 Aug 93 11:04:58 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24741; Wed, 11 Aug 93 11:01:42 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 11 Aug 1993 11:01:40 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24726; Wed, 11 Aug 93 11:01:39 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA01830; Wed, 11 Aug 93 10:01:37 CDT Date: Wed, 11 Aug 93 10:01:37 CDT From: Tony Skjellum Message-Id: <9308111501.AA01830@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu Subject: Netlib MPI Way out of Date Everyone... I have just seen the context chapter from netlib, which someone has brought to the meeting. It is not even the June 24 version of our document, it is something from May. Netlib needs to be up-to-date, or we need to rely solely on anonymous ftp. People are asking me questions about versions of MPI drafts long dead. No wonder there is added confusion. Can this be fixed? Tony From owner-mpi-comm@CS.UTK.EDU Mon Aug 16 09:11:38 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA23739; Mon, 16 Aug 93 09:11:38 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04143; Mon, 16 Aug 93 09:09:37 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 16 Aug 1993 09:09:36 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04135; Mon, 16 Aug 93 09:09:35 -0400 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA23530; Mon, 16 Aug 1993 09:09:34 -0400 Message-Id: <9308161309.AA23530@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Fortran 90 bindings Date: Mon, 16 Aug 93 09:09:33 -0500 From: David W. Walker ------- Forwarded Message Received: from sun2.nsfnet-relay.ac.uk by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA14642; Wed, 11 Aug 1993 06:29:54 -0400 Via: uk.ac.edinburgh.castle; Wed, 11 Aug 1993 11:29:10 +0100 Date: 11 Aug 93 11:27:35 BST From: D Muxworthy Subject: Guidelines for Bindings to Fortran 90 To: mjhanna@somnet.sandia.gov, benson@buffer.enet.dec.com, als@cs.umd.edu, walker@rios2.epm.ornl.gov, donn@bix.com Message-Id: <9308111127.aa25840@uk.ac.ed.castle> Status: O You might be interested to know, or you might not, that the SC22/WG5 bindings guidelines document is available as from today in compressed postscript form from the NCSA anonymous ftp server. server: ftp.ncsa.uiuc.edu file: sc22wg5/N938.ps.Z (use upper and lower case letters exactly as shown) Best wishes, David Muxworthy ------- End of Forwarded Message From owner-mpi-comm@CS.UTK.EDU Tue Aug 17 00:59:20 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA00684; Tue, 17 Aug 93 00:59:20 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02644; Tue, 17 Aug 93 00:57:02 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 17 Aug 1993 00:56:56 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02631; Tue, 17 Aug 93 00:56:53 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA19016; Mon, 16 Aug 93 23:56:51 CDT Date: Mon, 16 Aug 93 23:56:51 CDT From: Tony Skjellum Message-Id: <9308170456.AA19016@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu Subject: proposal: comp.parallel.mpi Colleagues: I would like to have an unmoderated group, like Jack's comp.parallel.pvm, so that post-MPI-meetings, we can promulgate info in a timely and efficient manner. Comments? Anyone willing to give me the text that was used to propose comp.parallel.pvm? Thanks, Tony From owner-mpi-comm@CS.UTK.EDU Tue Aug 17 08:01:34 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA01915; Tue, 17 Aug 93 08:01:34 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA29925; Tue, 17 Aug 93 07:59:28 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 17 Aug 1993 07:59:27 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA29917; Tue, 17 Aug 93 07:59:26 -0400 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA13869; Tue, 17 Aug 1993 07:59:25 -0400 Message-Id: <9308171159.AA13869@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: comp.parallel.mpi Date: Tue, 17 Aug 93 07:59:24 -0500 From: David W. Walker I think Tony's suggestion of a comp.parallel.mpi group is a good idea, but only after the MPI draft has been presented at Supercomputing 93. If it's established before this I think it will just increase the noise level. There's little point, IMHO, of involving "outsiders" at this stage. Regards, David From owner-mpi-comm@CS.UTK.EDU Tue Aug 17 09:54:02 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA02883; Tue, 17 Aug 93 09:54:02 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06781; Tue, 17 Aug 93 09:51:01 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 17 Aug 1993 09:51:00 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06768; Tue, 17 Aug 93 09:50:58 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA19531; Tue, 17 Aug 93 08:50:57 CDT Date: Tue, 17 Aug 93 08:50:57 CDT From: Tony Skjellum Message-Id: <9308171350.AA19531@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, walker@rios2.epm.ornl.gov Subject: Re: comp.parallel.mpi Given the latency of setting up a newsgroup, I will try to time it so it comes into existence after SC93. - Tony From owner-mpi-comm@CS.UTK.EDU Fri Aug 27 08:27:44 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA07742; Fri, 27 Aug 93 08:27:44 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12502; Fri, 27 Aug 93 08:24:43 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 27 Aug 1993 08:24:41 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from hub.meiko.co.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12494; Fri, 27 Aug 93 08:24:35 -0400 Received: from tycho.co.uk (tycho.meiko.co.uk) by hub.meiko.co.uk with SMTP id AA17512 (5.65c/IDA-1.4.4 for mpi-comm@cs.utk.edu); Fri, 27 Aug 1993 13:24:32 +0100 Date: Fri, 27 Aug 1993 13:24:32 +0100 From: James Cownie Message-Id: <199308271224.AA17512@hub.meiko.co.uk> Received: by tycho.co.uk (5.0/SMI-SVR4) id AA00275; Fri, 27 Aug 93 13:23:06 BST To: mpi-comm@cs.utk.edu Subject: T-Shirts ! Content-Length: 713 If we're serious about a T shirt we probably need to get our act together NOW, for the next meeting. The reasons are 1) It would be good to have them before SC93 2) If the final meeting is in Europe, then there may be hassle over importing large numbers of T shirts. 3) U.S T shirts are better than European ones anyway ! I'm on vacation now for two weeks, and have no strong opinion on design, however I'm sure lots of you do. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Fri Aug 27 12:41:16 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA09889; Fri, 27 Aug 93 12:41:16 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA16516; Fri, 27 Aug 93 12:37:51 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 27 Aug 1993 12:37:49 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA16508; Fri, 27 Aug 93 12:37:46 -0400 Received: from macaw.fsl.noaa.gov by fslg8.fsl.noaa.gov (5.65/Ultrix3.0-C) id AA02850; Fri, 27 Aug 1993 15:37:34 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA00580; Thu, 26 Aug 93 17:10:23 MDT Date: Thu, 26 Aug 93 17:10:23 MDT From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9308262310.AA00580@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu, jim@meiko.co.uk Subject: Re: T-Shirts ! > If we're serious about a T shirt we probably need to get our act > together NOW, for the next meeting. > > -- Jim Sounds like a good idea to me. Who is in charge of the T-shirt design subcommittee (he asks, not volunteering... :-)? Al? Tom From owner-mpi-comm@CS.UTK.EDU Fri Aug 27 13:10:34 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA10026; Fri, 27 Aug 93 13:10:34 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19089; Fri, 27 Aug 93 13:08:09 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 27 Aug 1993 13:08:07 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from convex.convex.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19073; Fri, 27 Aug 93 13:08:04 -0400 Received: from mozart.convex.com by convex.convex.com (5.64/1.35) id AA29076; Fri, 27 Aug 93 12:07:48 -0500 Received: by mozart.convex.com (5.64/1.28) id AA17459; Fri, 27 Aug 93 12:07:07 -0500 From: joelw@mozart.convex.com (Joel Williamson) Message-Id: <9308271707.AA17459@mozart.convex.com> Subject: Re: T-Shirts ! To: hender@macaw.fsl.noaa.gov (Tom Henderson) Date: Fri, 27 Aug 93 12:07:06 CDT Cc: mpi-comm@cs.utk.edu, jim@meiko.co.uk In-Reply-To: <9308262310.AA00580@macaw.fsl.noaa.gov>; from "Tom Henderson" at Aug 26, 93 5:10 pm X-Mailer: ELM [version 2.3 PL11] At HPFF we simply passed around the enclosed Tshirt design file, finding and replacing text and submitting it back to the group for praise/derision. Eventually the group chairman Chuck Keolbel called for a vote. So just strip of this header, and lpr the remaining to a postscript printer, modify and ship it back out. Perhaps we can come up with something more bold than "Don't blame me, I didn't vote for that feature." Best regards, Joel Williamson --------------cut here-------------------------- %!PS-Adobe-2.0 EPSF-1.2 %%DocumentFonts: Helvetica-Bold %%Pages: 1 %%BoundingBox: 5 59 540 782 %%EndComments 50 dict begin /arrowHeight 8 def /arrowWidth 4 def /none null def /numGraphicParameters 17 def /stringLimit 65535 def /Begin { save numGraphicParameters dict begin } def /End { end restore } def /SetB { dup type /nulltype eq { pop false /brushRightArrow idef false /brushLeftArrow idef true /brushNone idef } { /brushDashOffset idef /brushDashArray idef 0 ne /brushRightArrow idef 0 ne /brushLeftArrow idef /brushWidth idef false /brushNone idef } ifelse } def /SetCFg { /fgblue idef /fggreen idef /fgred idef } def /SetCBg { /bgblue idef /bggreen idef /bgred idef } def /SetF { /printSize idef /printFont idef } def /SetP { dup type /nulltype eq { pop true /patternNone idef } { /patternGrayLevel idef patternGrayLevel -1 eq { /patternString idef } if false /patternNone idef } ifelse } def /BSpl { 0 begin storexyn newpath n 1 gt { 0 0 0 0 0 0 1 1 true subspline n 2 gt { 0 0 0 0 1 1 2 2 false subspline 1 1 n 3 sub { /i exch def i 1 sub dup i dup i 1 add dup i 2 add dup false subspline } for n 3 sub dup n 2 sub dup n 1 sub dup 2 copy false subspline } if n 2 sub dup n 1 sub dup 2 copy 2 copy false subspline patternNone not brushLeftArrow not brushRightArrow not and and { ifill } if brushNone not { istroke } if 0 0 1 1 leftarrow n 2 sub dup n 1 sub dup rightarrow } if end } dup 0 4 dict put def /Circ { newpath 0 360 arc patternNone not { ifill } if brushNone not { istroke } if } def /CBSpl { 0 begin dup 2 gt { storexyn newpath n 1 sub dup 0 0 1 1 2 2 true subspline 1 1 n 3 sub { /i exch def i 1 sub dup i dup i 1 add dup i 2 add dup false subspline } for n 3 sub dup n 2 sub dup n 1 sub dup 0 0 false subspline n 2 sub dup n 1 sub dup 0 0 1 1 false subspline patternNone not { ifill } if brushNone not { istroke } if } { Poly } ifelse end } dup 0 4 dict put def /Elli { 0 begin newpath 4 2 roll translate scale 0 0 1 0 360 arc patternNone not { ifill } if brushNone not { istroke } if end } dup 0 1 dict put def /Line { 0 begin 2 storexyn newpath x 0 get y 0 get moveto x 1 get y 1 get lineto brushNone not { istroke } if 0 0 1 1 leftarrow 0 0 1 1 rightarrow end } dup 0 4 dict put def /MLine { 0 begin storexyn newpath n 1 gt { x 0 get y 0 get moveto 1 1 n 1 sub { /i exch def x i get y i get lineto } for patternNone not brushLeftArrow not brushRightArrow not and and { ifill } if brushNone not { istroke } if 0 0 1 1 leftarrow n 2 sub dup n 1 sub dup rightarrow } if end } dup 0 4 dict put def /Poly { 3 1 roll newpath moveto -1 add { lineto } repeat closepath patternNone not { ifill } if brushNone not { istroke } if } def /Rect { 0 begin /t exch def /r exch def /b exch def /l exch def newpath l b moveto l t lineto r t lineto r b lineto closepath patternNone not { ifill } if brushNone not { istroke } if end } dup 0 4 dict put def /Text { ishow } def /idef { dup where { pop pop pop } { exch def } ifelse } def /ifill { 0 begin gsave patternGrayLevel -1 ne { fgred bgred fgred sub patternGrayLevel mul add fggreen bggreen fggreen sub patternGrayLevel mul add fgblue bgblue fgblue sub patternGrayLevel mul add setrgbcolor eofill } { eoclip originalCTM setmatrix pathbbox /t exch def /r exch def /b exch def /l exch def /w r l sub ceiling cvi def /h t b sub ceiling cvi def /imageByteWidth w 8 div ceiling cvi def /imageHeight h def bgred bggreen bgblue setrgbcolor eofill fgred fggreen fgblue setrgbcolor w 0 gt h 0 gt and { l b translate w h scale w h true [w 0 0 h neg 0 h] { patternproc } imagemask } if } ifelse grestore end } dup 0 8 dict put def /istroke { gsave brushDashOffset -1 eq { [] 0 setdash 1 setgray } { brushDashArray brushDashOffset setdash fgred fggreen fgblue setrgbcolor } ifelse brushWidth setlinewidth originalCTM setmatrix stroke grestore } def /ishow { 0 begin gsave fgred fggreen fgblue setrgbcolor /fontDict printFont findfont printSize scalefont dup setfont def /descender fontDict begin 0 [FontBBox] 1 get FontMatrix end transform exch pop def /vertoffset 0 descender sub printSize sub printFont /Courier ne printFont /Courier-Bold ne and { 1 add } if def { 0 vertoffset moveto show /vertoffset vertoffset printSize sub def } forall grestore end } dup 0 3 dict put def /patternproc { 0 begin /patternByteLength patternString length def /patternHeight patternByteLength 8 mul sqrt cvi def /patternWidth patternHeight def /patternByteWidth patternWidth 8 idiv def /imageByteMaxLength imageByteWidth imageHeight mul stringLimit patternByteWidth sub min def /imageMaxHeight imageByteMaxLength imageByteWidth idiv patternHeight idiv patternHeight mul patternHeight max def /imageHeight imageHeight imageMaxHeight sub store /imageString imageByteWidth imageMaxHeight mul patternByteWidth add string def 0 1 imageMaxHeight 1 sub { /y exch def /patternRow y patternByteWidth mul patternByteLength mod def /patternRowString patternString patternRow patternByteWidth getinterval def /imageRow y imageByteWidth mul def 0 patternByteWidth imageByteWidth 1 sub { /x exch def imageString imageRow x add patternRowString putinterval } for } for imageString end } dup 0 12 dict put def /min { dup 3 2 roll dup 4 3 roll lt { exch } if pop } def /max { dup 3 2 roll dup 4 3 roll gt { exch } if pop } def /arrowhead { 0 begin transform originalCTM itransform /taily exch def /tailx exch def transform originalCTM itransform /tipy exch def /tipx exch def /dy tipy taily sub def /dx tipx tailx sub def /angle dx 0 ne dy 0 ne or { dy dx atan } { 90 } ifelse def gsave originalCTM setmatrix tipx tipy translate angle rotate newpath 0 0 moveto arrowHeight neg arrowWidth 2 div lineto arrowHeight neg arrowWidth 2 div neg lineto closepath patternNone not { originalCTM setmatrix /padtip arrowHeight 2 exp 0.25 arrowWidth 2 exp mul add sqrt brushWidth mul arrowWidth div def /padtail brushWidth 2 div def tipx tipy translate angle rotate padtip 0 translate arrowHeight padtip add padtail add arrowHeight div dup scale arrowheadpath ifill } if brushNone not { originalCTM setmatrix tipx tipy translate angle rotate arrowheadpath istroke } if grestore end } dup 0 9 dict put def /arrowheadpath { newpath 0 0 moveto arrowHeight neg arrowWidth 2 div lineto arrowHeight neg arrowWidth 2 div neg lineto closepath } def /leftarrow { 0 begin y exch get /taily exch def x exch get /tailx exch def y exch get /tipy exch def x exch get /tipx exch def brushLeftArrow { tipx tipy tailx taily arrowhead } if end } dup 0 4 dict put def /rightarrow { 0 begin y exch get /tipy exch def x exch get /tipx exch def y exch get /taily exch def x exch get /tailx exch def brushRightArrow { tipx tipy tailx taily arrowhead } if end } dup 0 4 dict put def /midpoint { 0 begin /y1 exch def /x1 exch def /y0 exch def /x0 exch def x0 x1 add 2 div y0 y1 add 2 div end } dup 0 4 dict put def /thirdpoint { 0 begin /y1 exch def /x1 exch def /y0 exch def /x0 exch def x0 2 mul x1 add 3 div y0 2 mul y1 add 3 div end } dup 0 4 dict put def /subspline { 0 begin /movetoNeeded exch def y exch get /y3 exch def x exch get /x3 exch def y exch get /y2 exch def x exch get /x2 exch def y exch get /y1 exch def x exch get /x1 exch def y exch get /y0 exch def x exch get /x0 exch def x1 y1 x2 y2 thirdpoint /p1y exch def /p1x exch def x2 y2 x1 y1 thirdpoint /p2y exch def /p2x exch def x1 y1 x0 y0 thirdpoint p1x p1y midpoint /p0y exch def /p0x exch def x2 y2 x3 y3 thirdpoint p2x p2y midpoint /p3y exch def /p3x exch def movetoNeeded { p0x p0y moveto } if p1x p1y p2x p2y p3x p3y curveto end } dup 0 17 dict put def /storexyn { /n exch def /y n array def /x n array def n 1 sub -1 0 { /i exch def y i 3 2 roll put x i 3 2 roll put } for } def %%EndProlog %I Idraw 7 Grid 8 %%Page: 1 1 Begin %I b u %I cfg u %I cbg u %I f u %I p u %I t [ 0.8 0 0 0.8 0 0 ] concat /originalCTM matrix currentmatrix def Begin %I MLine %I b 65535 1 0 0 [] 0 SetB %I cfg Black 0 0 0 SetCFg %I cbg White 1 1 1 SetCBg %I p 1 SetP %I t [ 1 0 0 1 20 425.5 ] concat %I 22 66 444 76 436 129 403 153 432 153 135 445 134 447 436 465 412 522 457 466 528 398 549 360 551 335 530 306 522 283 523 264 529 246 546 199 544 120 520 65 444 81 432 82 432 22 MLine End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* /Helvetica-Bold 14 SetF %I t [ 3.30769 0 0 2.11765 493 787.029 ] concat %I [ (FRONT) ] Text End Begin %I MLine %I b 65535 1 0 0 [] 0 SetB %I cfg Black 0 0 0 SetCFg %I cbg White 1 1 1 SetCBg %I p 1 SetP %I t [ 1 0 0 1 22 -58.5 ] concat %I 22 66 444 76 436 129 403 153 432 153 135 445 134 447 436 465 412 522 457 466 528 398 549 360 551 335 530 306 522 283 523 264 529 246 546 199 544 120 520 65 444 81 432 82 432 22 MLine End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* /Helvetica-Bold 14 SetF %I t [ 3.11905 0 0 2.76471 546 262.441 ] concat %I [ (BACK) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* /Helvetica-Bold 14 SetF %I t [ 1 0 0 2.52941 233 362.382 ] concat %I [ (Don't blame me...) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* /Helvetica-Bold 14 SetF %I t [ 1 0 0 1.94118 234 275.676 ] concat %I [ (I didn't vote for that feature.) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* /Helvetica-Bold 14 SetF %I t [ 4.59975 0 0 4.77176 231.144 821.185 ] concat %I [ (HPFF) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* /Helvetica-Bold 14 SetF %I t [ 1.08333 0 0 1 234.581 843.5 ] concat %I [ (HIGH PERFORMANCE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* /Helvetica-Bold 14 SetF %I t [ 1 0 0 1 234.539 766.5 ] concat %I [ (FORTRAN FORUM 1992) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* /Helvetica-Bold 14 SetF %I t [ 2.7907 0 0 4.76471 8.5 923.206 ] concat %I [ (#3) ] Text End End %I eop showpage %%Trailer end From owner-mpi-comm@CS.UTK.EDU Fri Aug 27 13:59:55 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA10208; Fri, 27 Aug 93 13:59:55 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22983; Fri, 27 Aug 93 13:57:52 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 27 Aug 1993 13:57:51 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22973; Fri, 27 Aug 93 13:57:50 -0400 Received: by msr.EPM.ORNL.GOV (4.1/1.34) id AA24054; Fri, 27 Aug 93 13:57:49 EDT Date: Fri, 27 Aug 93 13:57:49 EDT From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9308271757.AA24054@msr.EPM.ORNL.GOV> To: mpi-comm@cs.utk.edu Subject: Re: T-shirts >Who is in charge of the T-shirt design? Al? I am working on it. I have a design down on paper - need to turn into postscript. Al From owner-mpi-comm@CS.UTK.EDU Thu Sep 9 10:27:45 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA18310; Thu, 9 Sep 93 10:27:45 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11629; Thu, 9 Sep 93 10:21:24 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 9 Sep 1993 10:21:21 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sun4.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11606; Thu, 9 Sep 93 10:21:13 -0400 Received: by sun4.epm.ornl.gov (4.1/1.34) id AA21778; Thu, 9 Sep 93 10:21:10 EDT Date: Thu, 9 Sep 93 10:21:10 EDT From: geist@sun4.epm.ornl.gov (Al Geist) Message-Id: <9309091421.AA21778@sun4.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Proposed T-shirt Design Front with back quote. Hi Folks, Here is a possible T-shirt design. There are still a few routines to add that have not been presented to committee yet. The top choice for qoute on the back is: MPI: Defining The Current Practice of Tomorrow Today Runner-up is: Entia non sunt multiplicanda praeter necessitatem I want to stay away from negative comments on MPI or other systems. T-shirts R us Al Geist ------------- %!PS-Adobe-2.0 EPSF-1.2 %%DocumentFonts: Helvetica-Bold %%Pages: 1 %%BoundingBox: 32 2 589 789 %%EndComments /arrowHeight 10 def /arrowWidth 5 def /IdrawDict 51 dict def IdrawDict begin /reencodeISO { dup dup findfont dup length dict begin { 1 index /FID ne { def }{ pop pop } ifelse } forall /Encoding ISOLatin1Encoding def currentdict end definefont } def /ISOLatin1Encoding [ /.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef /.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef /.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef /.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef /space/exclam/quotedbl/numbersign/dollar/percent/ampersand/quoteright /parenleft/parenright/asterisk/plus/comma/minus/period/slash /zero/one/two/three/four/five/six/seven/eight/nine/colon/semicolon /less/equal/greater/question/at/A/B/C/D/E/F/G/H/I/J/K/L/M/N /O/P/Q/R/S/T/U/V/W/X/Y/Z/bracketleft/backslash/bracketright /asciicircum/underscore/quoteleft/a/b/c/d/e/f/g/h/i/j/k/l/m /n/o/p/q/r/s/t/u/v/w/x/y/z/braceleft/bar/braceright/asciitilde /.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef /.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef /.notdef/dotlessi/grave/acute/circumflex/tilde/macron/breve /dotaccent/dieresis/.notdef/ring/cedilla/.notdef/hungarumlaut /ogonek/caron/space/exclamdown/cent/sterling/currency/yen/brokenbar /section/dieresis/copyright/ordfeminine/guillemotleft/logicalnot /hyphen/registered/macron/degree/plusminus/twosuperior/threesuperior /acute/mu/paragraph/periodcentered/cedilla/onesuperior/ordmasculine /guillemotright/onequarter/onehalf/threequarters/questiondown /Agrave/Aacute/Acircumflex/Atilde/Adieresis/Aring/AE/Ccedilla /Egrave/Eacute/Ecircumflex/Edieresis/Igrave/Iacute/Icircumflex /Idieresis/Eth/Ntilde/Ograve/Oacute/Ocircumflex/Otilde/Odieresis /multiply/Oslash/Ugrave/Uacute/Ucircumflex/Udieresis/Yacute /Thorn/germandbls/agrave/aacute/acircumflex/atilde/adieresis /aring/ae/ccedilla/egrave/eacute/ecircumflex/edieresis/igrave /iacute/icircumflex/idieresis/eth/ntilde/ograve/oacute/ocircumflex /otilde/odieresis/divide/oslash/ugrave/uacute/ucircumflex/udieresis /yacute/thorn/ydieresis ] def /Helvetica-Bold reencodeISO def /none null def /numGraphicParameters 17 def /stringLimit 65535 def /Begin { save numGraphicParameters dict begin } def /End { end restore } def /SetB { dup type /nulltype eq { pop false /brushRightArrow idef false /brushLeftArrow idef true /brushNone idef } { /brushDashOffset idef /brushDashArray idef 0 ne /brushRightArrow idef 0 ne /brushLeftArrow idef /brushWidth idef false /brushNone idef } ifelse } def /SetCFg { /fgblue idef /fggreen idef /fgred idef } def /SetCBg { /bgblue idef /bggreen idef /bgred idef } def /SetF { /printSize idef /printFont idef } def /SetP { dup type /nulltype eq { pop true /patternNone idef } { dup -1 eq { /patternGrayLevel idef /patternString idef } { /patternGrayLevel idef } ifelse false /patternNone idef } ifelse } def /BSpl { 0 begin storexyn newpath n 1 gt { 0 0 0 0 0 0 1 1 true subspline n 2 gt { 0 0 0 0 1 1 2 2 false subspline 1 1 n 3 sub { /i exch def i 1 sub dup i dup i 1 add dup i 2 add dup false subspline } for n 3 sub dup n 2 sub dup n 1 sub dup 2 copy false subspline } if n 2 sub dup n 1 sub dup 2 copy 2 copy false subspline patternNone not brushLeftArrow not brushRightArrow not and and { ifill } if brushNone not { istroke } if 0 0 1 1 leftarrow n 2 sub dup n 1 sub dup rightarrow } if end } dup 0 4 dict put def /Circ { newpath 0 360 arc patternNone not { ifill } if brushNone not { istroke } if } def /CBSpl { 0 begin dup 2 gt { storexyn newpath n 1 sub dup 0 0 1 1 2 2 true subspline 1 1 n 3 sub { /i exch def i 1 sub dup i dup i 1 add dup i 2 add dup false subspline } for n 3 sub dup n 2 sub dup n 1 sub dup 0 0 false subspline n 2 sub dup n 1 sub dup 0 0 1 1 false subspline patternNone not { ifill } if brushNone not { istroke } if } { Poly } ifelse end } dup 0 4 dict put def /Elli { 0 begin newpath 4 2 roll translate scale 0 0 1 0 360 arc patternNone not { ifill } if brushNone not { istroke } if end } dup 0 1 dict put def /Line { 0 begin 2 storexyn newpath x 0 get y 0 get moveto x 1 get y 1 get lineto brushNone not { istroke } if 0 0 1 1 leftarrow 0 0 1 1 rightarrow end } dup 0 4 dict put def /MLine { 0 begin storexyn newpath n 1 gt { x 0 get y 0 get moveto 1 1 n 1 sub { /i exch def x i get y i get lineto } for patternNone not brushLeftArrow not brushRightArrow not and and { ifill } if brushNone not { istroke } if 0 0 1 1 leftarrow n 2 sub dup n 1 sub dup rightarrow } if end } dup 0 4 dict put def /Poly { 3 1 roll newpath moveto -1 add { lineto } repeat closepath patternNone not { ifill } if brushNone not { istroke } if } def /Rect { 0 begin /t exch def /r exch def /b exch def /l exch def newpath l b moveto l t lineto r t lineto r b lineto closepath patternNone not { ifill } if brushNone not { istroke } if end } dup 0 4 dict put def /Text { ishow } def /idef { dup where { pop pop pop } { exch def } ifelse } def /ifill { 0 begin gsave patternGrayLevel -1 ne { fgred bgred fgred sub patternGrayLevel mul add fggreen bggreen fggreen sub patternGrayLevel mul add fgblue bgblue fgblue sub patternGrayLevel mul add setrgbcolor eofill } { eoclip originalCTM setmatrix pathbbox /t exch def /r exch def /b exch def /l exch def /w r l sub ceiling cvi def /h t b sub ceiling cvi def /imageByteWidth w 8 div ceiling cvi def /imageHeight h def bgred bggreen bgblue setrgbcolor eofill fgred fggreen fgblue setrgbcolor w 0 gt h 0 gt and { l b translate w h scale w h true [w 0 0 h neg 0 h] { patternproc } imagemask } if } ifelse grestore end } dup 0 8 dict put def /istroke { gsave brushDashOffset -1 eq { [] 0 setdash 1 setgray } { brushDashArray brushDashOffset setdash fgred fggreen fgblue setrgbcolor } ifelse brushWidth setlinewidth originalCTM setmatrix stroke grestore } def /ishow { 0 begin gsave fgred fggreen fgblue setrgbcolor /fontDict printFont printSize scalefont dup setfont def /descender fontDict begin 0 [FontBBox] 1 get FontMatrix end transform exch pop def /vertoffset 1 printSize sub descender sub def { 0 vertoffset moveto show /vertoffset vertoffset printSize sub def } forall grestore end } dup 0 3 dict put def /patternproc { 0 begin /patternByteLength patternString length def /patternHeight patternByteLength 8 mul sqrt cvi def /patternWidth patternHeight def /patternByteWidth patternWidth 8 idiv def /imageByteMaxLength imageByteWidth imageHeight mul stringLimit patternByteWidth sub min def /imageMaxHeight imageByteMaxLength imageByteWidth idiv patternHeight idiv patternHeight mul patternHeight max def /imageHeight imageHeight imageMaxHeight sub store /imageString imageByteWidth imageMaxHeight mul patternByteWidth add string def 0 1 imageMaxHeight 1 sub { /y exch def /patternRow y patternByteWidth mul patternByteLength mod def /patternRowString patternString patternRow patternByteWidth getinterval def /imageRow y imageByteWidth mul def 0 patternByteWidth imageByteWidth 1 sub { /x exch def imageString imageRow x add patternRowString putinterval } for } for imageString end } dup 0 12 dict put def /min { dup 3 2 roll dup 4 3 roll lt { exch } if pop } def /max { dup 3 2 roll dup 4 3 roll gt { exch } if pop } def /midpoint { 0 begin /y1 exch def /x1 exch def /y0 exch def /x0 exch def x0 x1 add 2 div y0 y1 add 2 div end } dup 0 4 dict put def /thirdpoint { 0 begin /y1 exch def /x1 exch def /y0 exch def /x0 exch def x0 2 mul x1 add 3 div y0 2 mul y1 add 3 div end } dup 0 4 dict put def /subspline { 0 begin /movetoNeeded exch def y exch get /y3 exch def x exch get /x3 exch def y exch get /y2 exch def x exch get /x2 exch def y exch get /y1 exch def x exch get /x1 exch def y exch get /y0 exch def x exch get /x0 exch def x1 y1 x2 y2 thirdpoint /p1y exch def /p1x exch def x2 y2 x1 y1 thirdpoint /p2y exch def /p2x exch def x1 y1 x0 y0 thirdpoint p1x p1y midpoint /p0y exch def /p0x exch def x2 y2 x3 y3 thirdpoint p2x p2y midpoint /p3y exch def /p3x exch def movetoNeeded { p0x p0y moveto } if p1x p1y p2x p2y p3x p3y curveto end } dup 0 17 dict put def /storexyn { /n exch def /y n array def /x n array def n 1 sub -1 0 { /i exch def y i 3 2 roll put x i 3 2 roll put } for } def %%EndProlog %%BeginIdrawPrologue /arrowhead { 0 begin transform originalCTM itransform /taily exch def /tailx exch def transform originalCTM itransform /tipy exch def /tipx exch def /dy tipy taily sub def /dx tipx tailx sub def /angle dx 0 ne dy 0 ne or { dy dx atan } { 90 } ifelse def gsave originalCTM setmatrix tipx tipy translate angle rotate newpath arrowHeight neg arrowWidth 2 div moveto 0 0 lineto arrowHeight neg arrowWidth 2 div neg lineto patternNone not { originalCTM setmatrix /padtip arrowHeight 2 exp 0.25 arrowWidth 2 exp mul add sqrt brushWidth mul arrowWidth div def /padtail brushWidth 2 div def tipx tipy translate angle rotate padtip 0 translate arrowHeight padtip add padtail add arrowHeight div dup scale arrowheadpath ifill } if brushNone not { originalCTM setmatrix tipx tipy translate angle rotate arrowheadpath istroke } if grestore end } dup 0 9 dict put def /arrowheadpath { newpath arrowHeight neg arrowWidth 2 div moveto 0 0 lineto arrowHeight neg arrowWidth 2 div neg lineto } def /leftarrow { 0 begin y exch get /taily exch def x exch get /tailx exch def y exch get /tipy exch def x exch get /tipx exch def brushLeftArrow { tipx tipy tailx taily arrowhead } if end } dup 0 4 dict put def /rightarrow { 0 begin y exch get /tipy exch def x exch get /tipx exch def y exch get /taily exch def x exch get /tailx exch def brushRightArrow { tipx tipy tailx taily arrowhead } if end } dup 0 4 dict put def %%EndIdrawPrologue %I Idraw 10 Grid 8 8 %%Page: 1 1 Begin %I b u %I cfg u %I cbg u %I f u %I p u %I t [ 0.799705 0 0 0.799705 0 0 ] concat /originalCTM matrix currentmatrix def Begin %I Pict %I b u %I cfg u %I cbg u %I f u %I p u %I t [ 1 0 0 1 32 4 ] concat Begin %I Text %I cfg Green 0 1 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 2.61386e-08 2.14815 -2.14815 2.61386e-08 637.259 77.1852 ] concat %I [ (Back:) ] Text End Begin %I Text %I cfg Red 1 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 2.06326e-08 1.69565 -1.69565 2.06326e-08 634.783 209 ] concat %I [ ( MPI: Defining ) (The current Practice of Tomorrow) ( Today.) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 129 551 ] concat %I [ (MPI_ADDRESS) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 0.974059 0.226296 -0.226296 0.974059 249.653 331.169 ] concat %I [ (MPI_ALLCAST) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 -3.55271e-14 3.55271e-14 1 364 955 ] concat %I [ (MPI_ALLREDUCE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 239 557 ] concat %I [ (MPI_ALLTOALL) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -0.903278 0.429057 -0.429057 -0.903278 141.341 271.178 ] concat %I [ (MPI_BCAST) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 195 489 ] concat %I [ (MPI_ATTRIBUTE_ALLOC) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 41 523 ] concat %I [ (MPI_ATTRIBUTE_FREE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 458 138 ] concat %I [ (MPI_BARRIER) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -1 2.43359e-08 -2.43359e-08 -1 112 32.0004 ] concat %I [ (MPI_CANCEL) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 373 491 ] concat %I [ (MPI_CART) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -0.971508 0.237007 -0.237007 -0.971508 220.936 69.8524 ] concat %I [ (MPI_COLL_SUBGROUP) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 180 551 ] concat %I [ (MPI_COMM_CONTEXT) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 570 8.99997 ] concat %I [ (MPI_COMM_DUP) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 474 587 ] concat %I [ (MPI_COMM_FREE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 152 554 ] concat %I [ (MPI_COMM_GROUP) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -0.972983 0.230877 -0.230877 -0.972983 333.252 69.5133 ] concat %I [ (MPI_COMM_INIT) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 0.974479 0.224481 -0.224481 0.974479 191.075 417.444 ] concat %I [ (MPI_COMM_MAKE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 569 145 ] concat %I [ (MPI_COMM_NAME_MAKE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 567 345 ] concat %I [ (MPI_CREATE_GROUP) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 -3.55271e-14 3.55271e-14 1 216.5 957.5 ] concat %I [ (MPI_CREATE_RECV) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 563 522 ] concat %I [ (MPI_CREATE_RSEND) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 451 580 ] concat %I [ (MPI_CREATE_SEND) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 0.0137466 0.999906 -0.999906 0.0137466 478.798 237.077 ] concat %I [ (MPI_DELETE_ATTRIBUTE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 564 839 ] concat %I [ (MPI_EXCHANGE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 449 731 ] concat %I [ (MPI_GATHER) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 114 911 ] concat %I [ (MPI_GET_ATTRIBUTE_KEY) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 336 583 ] concat %I [ (MPI_GET_COUNT) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 405 147 ] concat %I [ (MPI_GET_SOURCE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 372 164 ] concat %I [ (MPI_GET_TAG) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 253 737 ] concat %I [ (MPI_GRAPH) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 387 699 ] concat %I [ (MPI_GRAPHDIMS) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 22 545 ] concat %I [ (MPI_GROUP_DUP) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -0.967074 0.254493 -0.254493 -0.967074 372.005 33.4863 ] concat %I [ (MPI_GROUP_FREE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 209 552 ] concat %I [ (MPI_GROUP_RANK) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 563 696 ] concat %I [ (MPI_GROUP_SIZE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 411 585 ] concat %I [ (MPI_INQCART) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 357 731 ] concat %I [ (MPI_INQDIM) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -0.941742 0.336336 -0.336336 -0.941742 284.527 288.788 ] concat %I [ (MPI_INQGRAPH) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -1 2.4336e-08 -2.4336e-08 -1 116 79.0001 ] concat %I [ (MPI_INQLOC) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 114 956 ] concat %I [ (MPI_INQMAP) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 473 724 ] concat %I [ (MPI_INQRANK) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 335 734 ] concat %I [ (MPI_IPROBE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 46 427 ] concat %I [ (MPI_IRECV) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 482 975 ] concat %I [ (MPI_IRSEND) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 212 750 ] concat %I [ (MPI_ISEND) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 454.5 571.5 ] concat %I [ (MPI_ISSEND) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 404 295 ] concat %I [ (MPI_MAKDIM) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 44 101 ] concat %I [ (MPI_MAP_CART) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 352.5 977.5 ] concat %I [ (MPI_MAP_GRAPH) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 176 404 ] concat %I [ (MPI_PARTC) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 0.894427 0.447214 -0.447214 0.894427 62.304 227.242 ] concat %I [ (MPI_PROBE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 324 176 ] concat %I [ (MPI_PROBE_COUNT) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 373 277 ] concat %I [ (MPI_RECV) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 0.0167756 0.99986 -0.99986 0.0167756 270.195 187.091 ] concat %I [ (MPI_REDUCE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 154 401 ] concat %I [ (MPI_RSEND) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 0.966092 0.258195 -0.258195 0.966092 322.997 448.019 ] concat %I [ (MPI_SCAN) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 390 590 ] concat %I [ (MPI_SCATTER) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 193 755 ] concat %I [ (MPI_SEND) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 67 381 ] concat %I [ (MPI_SENDRECV) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 458 252 ] concat %I [ (MPI_SET_ATTRIBUTE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 482 130 ] concat %I [ (MPI_SHIFT_ID) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 108 401 ] concat %I [ (MPI_SSEND) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 232 748 ] concat %I [ (MPI_START) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 114.5 820.5 ] concat %I [ (MPI_TEST) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 523 584 ] concat %I [ (MPI_TEST_ATTRIBUTE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 36 814 ] concat %I [ (MPI_TEST_CANCELLED) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 0.886225 0.463254 -0.463254 0.886225 170.588 192.927 ] concat %I [ (MPI_TESTALL) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 279 726 ] concat %I [ (MPI_TESTANY) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 46 228 ] concat %I [ (MPI_TRANSLATE_RANKS) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 21.9999 239 ] concat %I [ (MPI_TYPE_COMMIT) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 307 908 ] concat %I [ (MPI_TYPE_CONTIGUOUS) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 0.978234 0.207504 -0.207504 0.978234 400.91 467.592 ] concat %I [ (MPI_TYPE_EXTENT) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 330.5 570.5 ] concat %I [ (MPI_TYPE_FREE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 542 589 ] concat %I [ (MPI_TYPE_HINDEXED) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 13 832 ] concat %I [ (MPI_TYPE_HVECTOR) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 433 143 ] concat %I [ (MPI_TYPE_INDEXED) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 113 933 ] concat %I [ (MPI_TYPE_STRUCT) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 348 174 ] concat %I [ (MPI_TYPE_VECTOR) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 0.980581 0.196116 -0.196116 0.980581 209.18 342.635 ] concat %I [ (MPI_USER_ALLREDUCE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -1 2.43359e-08 -2.43359e-08 -1 203.5 57.5003 ] concat %I [ (MPI_USER_ALLREDUCEA) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -0.982102 0.188348 -0.188348 -0.982102 205.261 99.9342 ] concat %I [ (MPI_USER_REDUCE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 49.4999 974.5 ] concat %I [ (MPI_USER_REDUCEA) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 22 104 ] concat %I [ (MPI_USER_SCAN) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 297 180 ] concat %I [ (MPI_USER_SCANA) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 111 799 ] concat %I [ (MPI_WAIT) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 434 303 ] concat %I [ (MPI_WAITALL) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 525 318 ] concat %I [ (MPI_WAITANY) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 504 270 ] concat %I [ (MPI_CREATE_SSEND) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 497 592 ] concat %I [ (MPI_GROUP_UNION) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 584 773 ] concat %I [ (MPI_GROUP_INTERSECTION) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -1 2.43359e-08 -2.43359e-08 -1 206 10.0001 ] concat %I [ (MPI_GROUP_DIFFERENCE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 211.5 976.5 ] concat %I [ (MPI_GROUP_INCL) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 410 692 ] concat %I [ (MPI_GROUP_EXCL) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 546 182 ] concat %I [ (MPI_GROUP_RANGE_INCL) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -0.975232 0.221187 -0.221187 -0.975232 531.491 24.6688 ] concat %I [ (MPI_GROUP_RANGE_EXCL) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 306 691 ] concat %I [ (MPI_GROUP_FREE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 53.5001 777.5 ] concat %I [ (MPI_COMM_SIZE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 428 697 ] concat %I [ (MPI_COMM_RANK) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 16 684 ] concat %I [ (MPI_COMM_SPLIT) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 504 121 ] concat %I [ (MPI_COMM_MERGE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 544 384 ] concat %I [ (MPI_CONTEXTS_ALLOC) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 545 7.00015 ] concat %I [ (MPI_CONTEXTS_FREE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 586 394 ] concat %I [ (MPI_COMM_CONTEXTS) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -0.961524 0.274721 -0.274721 -0.961524 504.683 3.24772 ] concat %I [ (MPI_COMM_STAT) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 585 584 ] concat %I [ (MPI_INTERCOMM_MAKE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 589 185 ] concat %I [ (MPI_INTERCOMM_START) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 527 116 ] concat %I [ (MPI_INTERCOMM_FINISH) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 591 26.0002 ] concat %I [ (MPI_COMM_SPLITL) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -1 2.43359e-08 -2.43359e-08 -1 302.5 30.5001 ] concat %I [ (MPI_INTERCOMM_NAME) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 252 932 ] concat %I [ (MPI_INTERCOMM_NAME_START) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 430 579 ] concat %I [ (MPI_INITIALIZED) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 0.624695 0.780869 -0.780869 0.624695 57.0426 710.807 ] concat %I [ (MPI_INIT) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 87.0002 391 ] concat %I [ (MPI_FINALIZE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 129 399 ] concat %I [ (MPI_ABORT) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 37.0002 696 ] concat %I [ (MPI_VALIDTAGS) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 201 470 ] concat %I [ (MPI_NUMGROUPS) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 360 594 ] concat %I [ (MPI_BUFFERSIZE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -1 2.43359e-08 -2.43359e-08 -1 424 11.0002 ] concat %I [ (MPI_NUMCOMMUNICATORS) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 23.0002 392 ] concat %I [ (MPI_ERRORMODE) ] Text End End %I eop End %I eop showpage %%Trailer end From owner-mpi-comm@CS.UTK.EDU Thu Sep 9 12:25:28 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA19205; Thu, 9 Sep 93 12:25:28 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21509; Thu, 9 Sep 93 12:22:38 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 9 Sep 1993 12:22:36 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21501; Thu, 9 Sep 93 12:22:34 -0400 Received: from snacker.pnl.gov (130.20.186.18) by pnlg.pnl.gov; Thu, 9 Sep 93 09:16 PDT Received: by snacker.pnl.gov (4.1/SMI-4.1) id AA23525; Thu, 9 Sep 93 09:13:49 PDT Date: Thu, 9 Sep 93 09:13:49 PDT From: rj_littlefield@pnlg.pnl.gov Subject: tee-shirt/ghostview problem To: mpi-comm@cs.utk.edu Cc: rj_littlefield@pnlg.pnl.gov Message-Id: <9309091613.AA23525@snacker.pnl.gov> X-Envelope-To: mpi-comm@cs.utk.edu Al Geist sent us a PostScript file and wrote: % Here is a possible T-shirt design. Be aware that you may have trouble previewing and/or printing that file. I tried previewing with ghostview (version 1.4). It failed with the following diagnostic: Error: /undefined in 2.61386e-08 Operand stack: -savetype- -savetype- -savetype- -marktype- Execution stack: %interp_exit --nostringval-- --nostringval-- false --nostringval-- --nostringval-- --nostringval-- Dictionary stack: 399/479 7/200 37/51 1/17 0/17 5/17 Unrecoverable error, exit code 1 However, when I replaced "2.61386e-08" and all similar strings with "0.", it worked fine. Further experiments suggested that ghostview just doesn't accept negative exponents. This problem is being reported to the ghostview people. The file did print correctly on an Apple LaserWriter II and on a DEC PrintServer 20. --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-comm@CS.UTK.EDU Thu Sep 9 13:26:26 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA19595; Thu, 9 Sep 93 13:26:26 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26105; Thu, 9 Sep 93 13:23:25 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 9 Sep 1993 13:23:24 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26097; Thu, 9 Sep 93 13:23:22 -0400 Date: Thu, 9 Sep 93 18:23:04 BST Message-Id: <25359.9309091723@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: tee-shirt To: mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Perhaps the subprograms should have arguments. Also it was suggested that there could be an MPI baseball cap with the header files on it :-) (Perhaps I can get one of these!) Anyway, seriously, it's just a bit noisy, don't you think? And, I hope the "back" text is not really a serious suggestion. Cheers Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Thu Sep 9 13:41:01 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA19705; Thu, 9 Sep 93 13:41:01 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA27264; Thu, 9 Sep 93 13:38:32 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 9 Sep 1993 13:38:31 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from THIALFI.CS.CORNELL.EDU by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA27256; Thu, 9 Sep 93 13:38:29 -0400 Received: from CLOYD.CS.CORNELL.EDU by thialfi.cs.cornell.edu (5.67/I-1.99E) id AA03403; Thu, 9 Sep 93 13:38:27 -0400 Received: from FAFNIR.CS.CORNELL.EDU by cloyd.cs.cornell.edu (5.67/I-1.99D) id AA26508; Thu, 9 Sep 93 13:38:32 -0400 From: ken@cs.cornell.edu (Ken Birman) Date: Thu, 9 Sep 93 13:38:14 -0400 Message-Id: <9309091738.AA21632@fafnir.cs.cornell.edu> Received: by fafnir.cs.cornell.edu (5.67/N-0.13) id AA21632; Thu, 9 Sep 93 13:38:14 -0400 To: mpi-comm@cs.utk.edu Subject: Could I get off of this mailing list, please? My group (Horus) hopes to provide an MPI implementation once the standard settles. But this email list seems to be mostly about t-shirt designs. Thanks! From owner-mpi-comm@CS.UTK.EDU Thu Sep 9 16:35:04 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA20563; Thu, 9 Sep 93 16:35:04 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11788; Thu, 9 Sep 93 16:33:02 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 9 Sep 1993 16:33:01 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11780; Thu, 9 Sep 93 16:33:00 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA01098; Thu, 9 Sep 93 15:32:52 CDT Date: Thu, 9 Sep 93 15:32:52 CDT From: Tony Skjellum Message-Id: <9309092032.AA01098@Aurora.CS.MsState.Edu> To: ken@cs.cornell.edu Subject: Re: Could I get off of this mailing list, please? Cc: mpi-comm@cs.utk.edu Maybe you should wait for our reference implementation. - Tony Skjellum From owner-mpi-comm@CS.UTK.EDU Thu Sep 16 03:48:24 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA26807; Thu, 16 Sep 93 03:48:24 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17304; Thu, 16 Sep 93 03:45:16 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 16 Sep 1993 03:45:14 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from THIALFI.CS.CORNELL.EDU by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17287; Thu, 16 Sep 93 03:45:12 -0400 Received: from CLOYD.CS.CORNELL.EDU by thialfi.cs.cornell.edu (5.67/I-1.99E) id AA04082; Thu, 16 Sep 93 03:45:01 -0400 Received: from ELLI.CS.CORNELL.EDU by cloyd.cs.cornell.edu (5.67/I-1.99D) id AA23988; Thu, 16 Sep 93 03:44:58 -0400 From: elster@cs.cornell.edu (Anne C. Elster) Date: Thu, 16 Sep 93 03:44:56 -0400 Message-Id: <9309160744.AA24612@elli.cs.cornell.edu> Received: by elli.cs.cornell.edu (5.67/N-0.13) id AA24612; Thu, 16 Sep 93 03:44:56 -0400 To: mpi-comm@cs.utk.edu Subject: MPI_LOAD Cc: elster@cs.cornell.edu Well, folks, I promised a nice write-up on my MPI_LOAD suggestion, but am afraid you will have to do with this patch-up job I'm throwing together in this wee hr in the morning. Sorry. I am off to Norway later to day for my Dad's 60th b-day, which also means I will not be able to make next week's meeting in Dallas. Have a nice and productive meeting! Maybe somebody could get a t-shirt sent to me? Take care! Anne ------------------------------------------------------------------------------- Anne C. Elster | (607) 254-8653 [off] 255-6236 [FAX] Xerox Design Research Institute | (607) 272-2986 [home/answ.mach] Cornell University, 502 Theory Ctr.| Ithaca, NY 14853 | e-mail: elster@cs.cornell.edu USA | na.elster@na-net.ornl.gov MPI_(CPU_)LOAD_INFO MOTIVATION Currently, most parallelized algorithms for distributed memory machines assume that they will run on a homogeneous set of processors. (I.e. that each node in the parallel system has approximately the same computing power and I/O speed.) The main focus in parallel computing has thus far been to minimize communication over-head and achieve a load-balance for the particular algorithm or application. However, as each computational node a parallel system become more complex, their individual processor/node performance characteristics will tend to deviate more as some nodes perform I/O while others concentrate on system tasks, etc. This behavior has so far been addressed through multi-tasking on large shared-memory systems such as a network of IBM 3090s, however, no simple, straight-forward solutions exist currently for the distributed memory cases. It is, however, this authors belief that load-balancing application at the ``load-info'' level (i.e. based on each node's dynamic load at run-time) will become very important in the future -- especially if there were mechanisms built into MPI that would facilitate such implementations. CURRRENT OPTIONS Options for getting load-info read into a Fortran or C application on Unix-based systems are not wonderful: System load is typically from the OS-side done by {\bf reading kernel memory (-kmem)}. This will, however, usually require {\em setgid} permissions and grubbing through kernel symbol tables. This brute-force approach might fork and exec something with the required kmem permissions. Another alternative is to use {\bf rstat} which does not require such permission, but unfortunately, is not available on all systems. Rstat typically gives you statistics from a remote kernel that should be current disregarding network delay. The third approach I can think of is to use the output from {\bf uptime}. In C, this could possibly be done though fscanf on the return value of popen. MPI SUGGESTION Since I fond none of the previous methods very programmer-friendly, my suggestion is to add a {\bf MPI\_LOAD\_INFO} that would return the desired information. As I mentioned at the last meeting, I am now convinced that the way ISIS/Horus does "load-info" for CPU load, i.e. to return the no. of jobs the scheduler has outstanding, is a good start. MPI_(CPU_)LOAD_INFO(n) OUT n : the no. of jobs the scheduler has outstanding. This way there is no timing issues associated with the local MPI call (MPI_CPU_LOAD_INFO, if you wish). The notion of needing some kind of time information was the largest objection I heard re. to my suggestion to include an MPI_LOAD_INFO back when I presented it in March. It is up to the applications programmers whether they'd want to do several calls for a statistical average. Global info. can be obtained via an MPI collective comm. call. Users will also be required to know scheduler queue sizes and what they mean for the individual sytstem/node, but at least they will not be stuck with non-standard system-supplied fucntions and "grubbing" through "kmem" tables under super-user priveleges in order to get the information they need. There are of course several other factors that influence the "load" of the processing element, e.g. memory traffic, communication traffic, I/O traffice, etc, all which could be very useful information for applications programmers to get their hands one -- especially in non-homogeneous or multi-user environments. Last time I guess some people objected to the notion of adding another MPI functions, some others felt it was not really an MPI call per se, and hence should be avoided. I can see the arguments for this, but feel the benefits of including such a functions too great to let it pas up. From owner-mpi-comm@CS.UTK.EDU Thu Sep 16 09:03:18 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA28606; Thu, 16 Sep 93 09:03:18 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15652; Thu, 16 Sep 93 09:00:01 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 16 Sep 1993 09:00:00 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.aggregate.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15635; Thu, 16 Sep 93 08:59:56 -0400 Received: from sirius.aggregate.com by aggregate.com (5.0/SMI-SVR4) id AA01583; Thu, 16 Sep 93 08:00:45 CDT Received: by sirius.aggregate.com (4.1/SMI-SVR4) id AA02006; Thu, 16 Sep 93 08:00:58 CDT From: ted@aggregate.com Message-Id: <9309161300.AA02006@sirius.aggregate.com> Subject: Re: MPI_LOAD To: elster@cs.cornell.edu (Anne C. Elster) Date: Thu, 16 Sep 1993 08:00:57 -0500 (CDT) Cc: mpi-comm@cs.utk.edu In-Reply-To: <9309160744.AA24612@elli.cs.cornell.edu> from "Anne C. Elster" at Sep 16, 93 03:44:56 am X-Mailer: ELM [version 2.4 PL22] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 3171 Anne C. Elster writes: > [text deleted] > > MOTIVATION > > Currently, most parallelized algorithms for distributed memory machines > assume that they will run on a homogeneous set of processors. > (I.e. that each node in the parallel system has approximately > the same computing power and I/O speed.) > The main focus in parallel computing has thus far been to minimize > communication over-head and achieve a load-balance for the > particular algorithm or application. > [text deleted] > > MPI SUGGESTION > > Since I fond none of the previous methods very programmer-friendly, > my suggestion is to add a {\bf MPI\_LOAD\_INFO} that would return > the desired information. As I mentioned at the last meeting, I am now > convinced that the way ISIS/Horus does "load-info" for CPU load, i.e. > to return the no. of jobs the scheduler has outstanding, is a good start. > > MPI_(CPU_)LOAD_INFO(n) > > OUT n : the no. of jobs the scheduler has outstanding. > > This way there is no timing issues associated with the local MPI > call (MPI_CPU_LOAD_INFO, if you wish). The notion of needing some > kind of time information was the largest objection I heard re. > to my suggestion to include an MPI_LOAD_INFO back when I presented it > in March. > > It is up to the applications programmers whether they'd want to do several > calls for a statistical average. Global info. can be obtained via an MPI > collective comm. call. Users will also be required to know scheduler queue > sizes and what they mean for the individual sytstem/node, but at least they > will not be stuck with non-standard system-supplied fucntions and "grubbing" > through "kmem" tables under super-user priveleges in order to get the > information they need. [text deleted] > > Last time I guess some people objected to the notion of adding another > MPI functions, some others felt it was not really an MPI call per se, > and hence should be avoided. I can see the arguments for this, but feel > the benefits of including such a functions too great to let it pas up. I'm merely a lurker on this mailing list, so I'll let others discuss what is and isn't appropriate for MPI. However, I have written a generalized network resource monitoring library for the product I'm working on (NetShare SDK, info@aggregate.com if you are curious). In a real heterogeneous network environment you need to track more than just the load average. For example, a SPARCstation 10 with a load average of 0.6 will give a better response than an idle diskless SPARCstation ELC. Other applications may care more about memory than load -- and then physical memory versus virtual memory. Software properties like licenses and OS type and version may be important. (As an example from NetShare, which does remote execution, you may want to select hosts for a large simulation based on memory, SPECfloat rating, simulator software license availablity, and having an idle keyboard) Resource monitoring is important in a heterogeneous environment, however the simple case of monitoring a single property is not useful in practice. -- Ted Stockwell, Aggregate Computing ted@aggregate.com From owner-mpi-comm@CS.UTK.EDU Fri Sep 17 21:10:54 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA12880; Fri, 17 Sep 93 21:10:54 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19499; Fri, 17 Sep 93 20:53:12 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 17 Sep 1993 20:53:11 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from super.super.org by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19491; Fri, 17 Sep 93 20:53:09 -0400 Received: from b125.super.org by super.super.org (4.1/SMI-4.1) id AA17279; Fri, 17 Sep 93 20:53:07 EDT Received: by b125.super.org (4.1/SMI-4.1) id AA04681; Fri, 17 Sep 93 20:53:07 EDT Date: Fri, 17 Sep 93 20:53:07 EDT From: lederman@super.org (Steve Huss-Lederman) Message-Id: <9309180053.AA04681@b125.super.org> To: mpi-comm@cs.utk.edu Subject: subset chapter Hello all, Since the upcomming meeting will have our final vote on each chapter, I thought I would let everyone know that the latest version of the subset chapter is available. To get use anonymous ftp to ftp.super.org in pub/mpi. Only the PostScript is there since there are x-refs, etc. The sections reference might differ from the various drafts of chapters that are out there; I used the latest ones I had for each. The major changes from the last meeting: - all intra-communicator and group functionality is in. all inter-communicator and caching is out. - All derived datatype stuff is in. - the new collective communication function MPI_REDUCE_SCATTER is in but MPI_USER_REDUCE_SCATTER and MPI_USER_REDUCE_SCATTERA are out. (I thought that this new name would be the longest yet in MPI but the context people still have a strong hold on this with MPI_LOCAL_SUBGROUP_EXCL_RANGES and After you add the P for profiling you get an amazing 31 characters. I'm surprised it can fit on the Tee Shirt :-) - profiling is in - Environmental management is still up in the air. We will need to discuss this at the meeting to see what we want in/out. If you have thoughts please post to mpi-iac. See you all soon in Dallas. Steve From owner-mpi-comm@CS.UTK.EDU Sun Sep 19 10:50:29 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA17234; Sun, 19 Sep 93 10:50:29 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA13805; Sun, 19 Sep 93 10:44:09 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sun, 19 Sep 1993 10:44:08 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA13797; Sun, 19 Sep 93 10:44:07 -0400 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA12315; Sun, 19 Sep 1993 10:44:06 -0400 Date: Sun, 19 Sep 1993 10:44:06 -0400 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9309191444.AA12315@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Next MPI meeting ******************************************************************* ******** ******** ******** Subcommittee chairs!! Please send me agenda items ******** ******** ******** ******************************************************************* The next meeting of the Message Passing Interface (MPI) Forum will take place in Dallas, September 22-24, 1993. Details are given below. The ongoing email discussion on MPI standardization issues can be obtained from netlib. To find out what is available via email send the following message to netlib@ornl.gov: send index from mpi or use xnetlib ("send index from xnetlib" for info). MPI Forum Meeting, September 22-24, 1993 ===================================== The next meeting of the MPI forum will take place at Bristol Suites Hotel 7800 Alpha Road Dallas, Texas The meeting will start at 1pm on Wednesday September 22, 1993, and finish at noon on Friday, September 24, 1993. Rooms are $89 per night and reservations may be made by calling (214) 233-7600 (mention MPI meeting). The meeting registration fee will be $75. Please make checks and POs payable to University of Tennessee. The registration fee will be collected at the meeting. The registration fee will go for coffee breaks, meeting rooms, AV and printer rentals. Certain organizations need to see a registration form before giving their employees money for meetings like this. You can get such a form in PostScript after noon on Sept. 20 by sending the following message to netlib@ornl.gov: send mpi-form.ps from mpi There is no need to send me the registration form or bring it to the meeting. TBS Shuttle Service will be providing complimentary shuttle service to and from the airports. If you fly into DFW, use their courtesy telephone and dial 03. If you fly into Love Field, you'll have to use a pay phone. They can be reached at 817-267-5150. Upon boarding the shuttle refer to the MPI meeting. We have NOT been able to make any special arrangements with airlines to get reduced fares. We have secured limited funding from ARPA/NSF for travel expenses of MPI meeting participants who are from U.S. universities. If you would like to apply for financial support to attend the September MPI meeting please send email to me at walker@msr.epm.ornl.gov, with justification of why you need support and an estimate of your travel expenses. Please send comments and/or suggested changes to the agenda below to me at walker@msr.epm.ornl.gov Provisional Agenda for MPI Meeting, September 22-24, 1993 Wednesday TBD Thursday TBD 12:00-1:30 Lunch (provided) TBD 6:00-8:00 Dinner (attendees pay, but hotel provides transport to area restaurant) 8:00-10:00 Continued informal subcommittee meetings if necessary Friday TBD From owner-mpi-comm@CS.UTK.EDU Mon Sep 20 16:41:01 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA26506; Mon, 20 Sep 93 16:41:01 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14331; Mon, 20 Sep 93 16:32:35 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 20 Sep 1993 16:32:34 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14323; Mon, 20 Sep 93 16:32:33 -0400 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA11935; Mon, 20 Sep 1993 16:32:32 -0400 Date: Mon, 20 Sep 1993 16:32:32 -0400 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9309202032.AA11935@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Agenda for MPI meeting Here is the agenda for this week's MPI meeting. Please let me know if you want any changes made. David Agenda for MPI Meeting September 22-24, 1993 Wednesday 1:00-3:00 Environmental Inquiry Subcommittee Meeting 3:00-4:00 Language Binding Subcommittee Meeting 4:00-5:30 Full MPIF meeting to discuss Collective Communication (Geist) 5:30-6:00 The status of the MPI reference implementation (Lusk and Gropp) 6:00-7:30 Dinner break 7:30-10:30 Break up for subcommittee meetings if necessary Thursday 9:00-12:00 Full meeting to discuss contexts (2nd reading?) (Skjellum) 12:00-1:30 Lunch (provided) 1:30-3:30 Full meeting to discuss contexts (continued from morning) (Skjellum) 3:30-3:45 Break 3:45-5:00 Full meeting to discuss topologies (Hempel) 5:00-6:00 Full meeting to discuss language binding (Lusk?) 6:00-8:00 Dinner (attendees pay, but hotel provides transport to area restaurant) 8:00-10:30 Continued informal subcommittee meetings if necessary Friday 9:00-10:30 Full group meeting for second reading of subset chapter. (Huss-Lederman) 10:30-10:45 Break 10:45-12:00 Any other business From owner-mpi-comm@CS.UTK.EDU Tue Sep 21 09:20:06 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA01211; Tue, 21 Sep 93 09:20:06 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25312; Tue, 21 Sep 93 09:14:34 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 21 Sep 1993 09:14:33 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25303; Tue, 21 Sep 93 09:14:31 -0400 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA12662; Tue, 21 Sep 1993 09:14:29 -0400 Date: Tue, 21 Sep 1993 09:14:29 -0400 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9309211314.AA12662@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: MPI Agenda: Second Version Agenda for MPI Meeting September 22-24, 1993 I can't remember what is general discussion, first reading, or second reading. I assume the subcommittee chairs know this. Wednesday 1:00-1:30 Review of the Introduction chapter, and what needs to be done to it. (Dongarra) 1:30-3:00 Full group meeting to discuss point-to-point (Snir) 3:00-4:00 Language Binding Subcommittee Meeting 4:00-5:30 Full MPIF meeting to discuss Collective Communication (Geist) 5:30-6:00 The status of the MPI reference implementation (Lusk and Gropp) 6:00-7:30 Dinner break 7:30-10:30 Environmental Inquiry Subcommittee Meeting. Other subcommittee meetings if necessary. Thursday 9:00-12:00 Full meeting to discuss contexts (2nd reading?) (Skjellum) 12:00-1:30 Lunch (provided) 1:30-3:30 Full meeting to discuss contexts (continued from morning) (Skjellum) 3:30-3:45 Break 3:45-5:00 Full meeting to discuss topologies (Hempel) 5:00-6:00 Full meeting to discuss language binding (Lusk?) 6:00-8:00 Dinner (attendees pay, but hotel provides transport to area restaurant) 8:00-10:30 Continued informal subcommittee meetings if necessary Friday 9:00-10:30 Full group meeting for second reading of subset chapter. (Huss-Lederman) 10:30-10:45 Break 10:45-12:00 Any other business From owner-mpi-comm@CS.UTK.EDU Tue Sep 21 11:50:42 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA02719; Tue, 21 Sep 93 11:50:42 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA08505; Tue, 21 Sep 93 11:46:29 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 21 Sep 1993 11:46:28 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA08404; Tue, 21 Sep 93 11:45:34 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA08316; Tue, 21 Sep 93 10:45:04 CDT Date: Tue, 21 Sep 93 10:45:04 CDT From: Tony Skjellum Message-Id: <9309211545.AA08316@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu, walker@rios2.EPM.ORNL.GOV Subject: MPI meeting ... Context Ladies/Gentlemen: I am coming to Dallas on Wednesday, in time for a post-dinner Context committee meeting. We are reading the chapter on Thursday. The Bristol Suites cancelled my reservation for Tuesday night, so I am coming on Wednesday; this also means I will be bringing the polished (:-)) chapter with me. Those of you who want to see it Wednesday will be able to see copies, or come to the meeting after dinner. I am sorry for the late arrival, but events have conspired against me. Please be patient. - Tony Skjellum From owner-mpi-comm@CS.UTK.EDU Tue Sep 21 12:01:14 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA02958; Tue, 21 Sep 93 12:01:14 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09341; Tue, 21 Sep 93 11:57:29 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 21 Sep 1993 11:57:26 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09325; Tue, 21 Sep 93 11:57:24 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA08373; Tue, 21 Sep 93 10:56:55 CDT Date: Tue, 21 Sep 93 10:56:55 CDT From: Tony Skjellum Message-Id: <9309211556.AA08373@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, walker@rios2.epm.ornl.gov Subject: Re: MPI Agenda: Second Version The agenda puts a "question mark" after the word second reading with regard to the context chapter. Why? - Tony ----- Begin Included Message ----- From owner-mpi-comm@CS.UTK.EDU Tue Sep 21 08:37:15 1993 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 21 Sep 1993 09:14:33 EDT Date: Tue, 21 Sep 1993 09:14:29 -0400 From: walker@rios2.epm.ornl.gov (David Walker) To: mpi-comm@cs.utk.edu Subject: MPI Agenda: Second Version Content-Length: 1445 Agenda for MPI Meeting September 22-24, 1993 I can't remember what is general discussion, first reading, or second reading. I assume the subcommittee chairs know this. Wednesday 1:00-1:30 Review of the Introduction chapter, and what needs to be done to it. (Dongarra) 1:30-3:00 Full group meeting to discuss point-to-point (Snir) 3:00-4:00 Language Binding Subcommittee Meeting 4:00-5:30 Full MPIF meeting to discuss Collective Communication (Geist) 5:30-6:00 The status of the MPI reference implementation (Lusk and Gropp) 6:00-7:30 Dinner break 7:30-10:30 Environmental Inquiry Subcommittee Meeting. Other subcommittee meetings if necessary. Thursday 9:00-12:00 Full meeting to discuss contexts (2nd reading?) (Skjellum) 12:00-1:30 Lunch (provided) 1:30-3:30 Full meeting to discuss contexts (continued from morning) (Skjellum) 3:30-3:45 Break 3:45-5:00 Full meeting to discuss topologies (Hempel) 5:00-6:00 Full meeting to discuss language binding (Lusk?) 6:00-8:00 Dinner (attendees pay, but hotel provides transport to area restaurant) 8:00-10:30 Continued informal subcommittee meetings if necessary Friday 9:00-10:30 Full group meeting for second reading of subset chapter. (Huss-Lederman) 10:30-10:45 Break 10:45-12:00 Any other business ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Tue Sep 21 22:55:33 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA06018; Tue, 21 Sep 93 22:55:33 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20080; Tue, 21 Sep 93 22:52:05 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 21 Sep 1993 22:52:03 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares9.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20059; Tue, 21 Sep 93 22:51:58 -0400 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA27329 (5.65c/IDA-1.4.4 for ); Tue, 21 Sep 1993 21:51:54 -0500 Message-Id: <199309220251.AA27329@antares.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: minutes of August meeting Date: Tue, 21 Sep 1993 21:51:51 -0500 From: Rusty Lusk Here are the minutes of the August meeting. Bob and Rusty Minutes of the Message Passing Interface Forum Dallas, Texas August 11 - 13, 1993 The MPI Forum met August 11-13, 1993, at the Bristol Suites Hotel in North Dallas. This was the eigth meeting of the MPIF and the sixth regular working meeting in Dallas. There were both general meetings of the committee as a whole and meetings of several of the subcommittees. This meeting included the first reading of the Communication Contexts, Environmental Management and Subset chapters, the second reading of the Process Topologies chapter and formal consideration of various topics in the Point-to-point and Collective Communication chapters. There were a substantial number of formal votes taken at this meeting as well as a few straw votes. All of the votes are recorded in these minutes (and can be found by searching for VOTE) and have also been published in summary form to the mpi-core mailing list. These minutes were written by Bob Knighten (knighten@ssd.intel.com) and Rusty Lusk (lusk@anl.gov). These minutes are quite long. If you want to see the important topics you can search for --- and this will quickly lead to each topic (and a few other things.) The basic document that was used at this meeting are: + DRAFT Document for a Standard Message-Passing Interface (August 10,1993) + MPI Environmental Management section Attendees: --------- Robert G. Babb II U. of Denver babb@cs.du.edu Doreen Cheng NASA/Ames dcheng@nas.nasa.gov Lyndon Clarke University.of Edinburgh lyndon@epcc.ed.ac.uk James Cownie Meiko jim@meiko.co.uk Jack Dongarra UT/ORNL dongarra@cs.utk.edu Anne C. Elster Cornell University. elster@cs.cornell.edu Jim Feeney IBM Endicott feenyj@vnet.endicott.ibm.com Al Geist ORNL gst@ornl.gov Ian Glendinning University. of Southampton igl@ecs.soton.ac.uk Brian K. Grant LLNL bkg@llnl.gov Adam Greenberg TMC moose@think.com Leslie Hart NOAA/FSL hart@fsl.noaa.gov Don Heller Shell Development heller@shell.com Tom Henderson NOAA/FSL hender@fsl.noaa.gov Alex Ho IBM Almaden wkh@almaden.ibm.com Gary Howell Florida Tech howell@zach.fit.edu Steven Huss-Lederman SRC lederman@super.org Bob Knighten Intel SSD knighten@ssd.intel.com Rik Littlefield PNL rj_littlefield@pnl.gov Rusty Lusk ANL lusk@mcs.anl.gov Peter Madams nCube pmadams@ncube.com Dan Nessett LLNL nessett@llnl.gov Steve Otto Oregon Graduate Instiute otto@cse.ogi.edu Peter Pacheco U. of San Francisco peter@sun.math.usfca.edu Anthony Skjellum Mississippi State U. tony@cs.msstate.edu Marc Snir IBM, T.J. Watson snir@watson.ibm.com Alan Sussman University. of Maryland als@cs.umd.edu Bob Tomlinson LANL bob@lanl.gov Eric T. Van de Velde CalTech evdv@ama.caltech.edu David Walker ORNL walker@msr.epm.ornl.gov Joel Williamson Convex Computer joelw@convex.com Wednesday, August 11 --------- --------- ------------------------------------------------------------------------------- General Meeting ------------------------------------------------------------------------------- Jack Dongarra opened the meeting by presenting the agenda that was previously sent out by David Walker. AGENDA ------ Wednesday 1:00 - 2:00 Subcommittee meetings 2:00 - 4:00 Point-to-point communications Snir 4:00 - 5:00 Collective communications Geist 6:00 - 7:30 Dinner 7:30 - 10:00 Subcommittee meetings Thursday 9:00 - 12:00 Context Skjellum 12:00 - 1:30 lunch 1:30 - 2:30 Context 3-4 subset Huss 4-6 topology Huss 6-8 dinner 8-10 subcommittees Friday 9-10:30 env Lusk 10:30-12 language Lusk Status of Readings sec\date May June August September --------- +-------------------------------------------------------------- p-p | 2 coll | 1 2 2 profile | 1 2 context | 1 2 topology | 1 2 subset | 1 2 2 lang | 1 2 env | 1 2 The next meeting will be September 22-24. It will again be here in Dallas. Started at 2:10 ------------------------------------------------------------------------------- Report From the Point-to-Point Communication Subcommittee ------------------------------------------------------------------------------- Marc Snir presided. Marc reorganized the chapter to make it more readable. He also added the material in Section 4.13 (Derived datatypes) in line with the straw vote at the last meeting. In response to a question, Marc noted that he welcomes editorial comments, and asks that they be sent to him in e-mail. Derived datatypes (4.3) ----------------------- Marc began by describing the ideas behind "Derived datatypes". What is the relation of this for Fortran 90 data types? Largely orthogonal. Organization count: 20 STRAW VOTE: Should "Derived datatypes" be added to MPI? ---------- Yes: 22 No: 0 Abstain: 0 Introduction (4.13.0) --------------------- VOTE: Approve 4.13.0 (Introduction)? ---- Yes: 19 No: 0 Abstain: 1 Datatype constructors (4.13.1) ------------------------------ Marc gave brief descriptions of the five functions in this section and contrasted them with the earlier buffer construction functions. VOTE: Approve 4.13.1 (Datatype constructors) ---- Yes: 20 No: 0 Abstain: 0 Additional functions (4.13.2) ----------------------------- It was noted that the sentence starting at line 33 on p. 77 is wrong and contradicts what follows. Marc agreed and will repair this. There was some discussion of exactly what is passed in a message using a datatype that contains gaps. There was no disagreement and this will be clarified. The fact that in MPI_ADDRESS has in integer OUT parameter that provides the byte address of location is a problem on some architectures was briefly discussed. One proposal was to use a suitable implementation specific definition of the return type. There was also a discussion of MPI_TYPE_COMMIT primarily to better understand what was intended here. The principal confusion was that MPI_TYPE_COMMIT has to do with completing the definition of the type, NOT committing data. There is an alternative form of MPI_TYPE_COMMIT with only one paramater (type). Then would need commit before communication; can use type in constructors after commit. Yet another alternative is a lazy commit, i.e. a datatype object becomes commited at first use in a communication. There are several issues in considering these alternatives. One is ease of use for the programmer (lazy commit is easy); use of resources (the original allows reclaiming resources as soon as they are not being used); the cost of using a datatype buffer in a communication. Adam Greenberg proposed yet another option - provide an optional function, MPI_TYPE_COMPILE, which can be used for efficiency but otherwise there is lazy commit. The objection to this was efficiency of the communication. Adam's counter was that he expected this to be in the noise of the general overhead of communication. Marc noted that we need to write up full proposals and have a more focused discussion. VOTE: Approve 4.13.2 (Additional functions) EXCEPT MPI_TYPE_COMMIT and ---- MPI_TYPE_FREE? Yes: 16 No: 0 Abstain: 2 Use of general datatypes in communication (4.13.3) -------------------------------------------------- This section specifies how datatypes are used in send and receive, including how type matching working. In particular datatypes match if they are structurally equivalent (i.e. the type signatures are the same.) Is there some function that gives a count (of some suitable sort) of elements in a send when using datatypes? {{{I am confused?}}} VOTE: Have query function that returns ther number of top-most elements ---- received? Yes: 14 No: 0 Abstain: 4 VOTE: Have ONLY a query function that returns the count of top-most level ---- elements in a receive. Yes: No: Abstain: VOTE: Approve 4.13.3 (Use of general datatypes in communication) as amended. ---- Yes: 15 No: 0 Abstain: 4 Examples (4.13.4) ----------------- There are no requirements here and so no vote was taken. Correct use of addresses (4.13.5) --------------------------------- This section is about dealing with addressing when the system does not have limits in using addresses systems which do not have a flat address space. VOTE: Approve section 4.13.5 (Correct use of addresses)? ---- Yes: 18 No: 0 Abstain: 1 Message data (4.2.1) -------------------- Marc asked if the table in this section needs to be expanded to include all of the native C data types. YES! Marc reviewed all of the changes (other than order) that he made in this chapter. More detail in the discussion of the semantics of point-to-point communication. Has added Progress (some guaranteed) and Fairness (none guaranteed) requirements. {{{Where?}}} Can use null when nothing is needed. On p. 61 there is a new function, MPI_TESTALL, as suggested by David Walker. It is included for reason of completeness and symmetry. {{{See Discussion on p. 62}}} VOTE: Allow null pointers in an array of pointers (with the system ---- required to do the right thing? Yes: 20 No: 0 Abstain: 1 On p. 63 there is a new function, MPI_PROBE_COUNT, which uses a datatype to interpret the result of a probe and get a count. On p. 64 there is a typo in MPI_PROBE. There should not be a datatype parameter. On p. 65, the name of MPI_IS_CANCELLED has been changed to MPI_TEST_CANCELLED. The time arrived to make a decision on SENDRECV (section 4.11). Adam Greenberg suggested that there should be two tag arguments (for sending and for receiving) rather than only one. One effect of this is that a wild card can now be used in the receive_tag. VOTE: Have two tag arguments (send_tag and receive_tag) in MPI_EXCHANGE? ---- Yes: 6 No: 3 Abstain: 10 Section 4.12 (Null processes) is a proposal to the suggestion from Jon Flower that was supported in a straw vote at the last meeting. There was a substantial discussion of the utility and cost of MPI_PROCNULL. Steve Huss-Lederman, as a proxy for Rolf Hempel, argued for the value of this in use with process topologies. Various people argued that there would a universal overhead if MPI_PROCNULL were allowed as a source/destination. Three positions: Use as legitimate source/destination; only in send/receive/exchange; not legitimate source/destination. time (based on the suggestion from Jon Flower.) VOTE: (1) Allow MPI_PROCNULL as a source or destination for all communication operations. (2) Allow MPI_PROCNULL only for MPI_SENDRECV and MPI_EXCHANGE. (3) Never allow MPI_PROCNULL as a source or destination in communication operations. (3) Yes: 3 No: 9 Abstain: 8 (2) Yes: 11 No: 5 Abstain: 6 Section 4.14 (Universal communication functions) includes one new convenience function, MPI_COMM_INIT. This was added by Marc Snir to make it part of the base functions in terms of which all other functions can be defined. An alternative approach is to put section 4.14 in an Annex and NOT require these functions as part of VOTE: Approve Chapter 4 (Point to Point Communication) as ammended. ---- Yes: 18 No: 0 Abstain: 1 Note that all that remains to consider in Chapter 4 are the type_commit and free functions. ------------------------------------------------------------------------------- Report From the Collective Communication Subcommittee ------------------------------------------------------------------------------- Al Geist presided. This was a continuation of the second reading of this chapter that was begun at the last meeting. The number of changes was small. The format of the buffer arguments have all been changed to agree with those in the Point-to-point Communication chapter. The material on user_reduce has been rewritten to provide two variants, one assuming commutativity of the user operation and one not. Steve Huss-Lederman asked the question if there are ANY collective operations that are guaranteed to give the same result for repeated runs with the same initial conditions. Al Geist replied "Broadcast. Next question." A long discussion ensued. As in previous discussions of this topic, there was a wide spectrum of opinions expressed. Some insisted that reproducibility, at least in a debug mode, is required. Others insisted this is a quality of implementation issue. Other opinions expressed included that this is outside of the scope of MPI; an implementation must document the extent to which reproducibility is available; etc. In line with the Discussion paragraph on p. 95, it was agreed that it is unneeded to have the completely general ALLTOALL. Rik Littlefield noted that there is a useful kind of reduce that is missing, a "scatter-reduce" that does a reduction of sections of an array to an array of processes. He will write up a proposal and distribute it to the collective communication mailing list. Al Geist noted the change mentioned in the paragraph labeled Missing on p. 99. There was some confusion, so Al Geist promised to add an example to clarify this. Jim Cownie pointed out that it is important that implementors should provide an implementation guide that specifies which of the collective operations that may/may not be synchronizing actually are synchronizing. Adam Greenberg countered that users must assume that these operations are not synchronizing and therefore such a document serves no purpose. Adam Greenberg also expressed unease with section 5.6 and STRAW VOTE: Should MPI require documentation of the implementation ---------- variations in synchronization properties of collective operations. Yes: 18 No: 2 Abstain: 2 Eric Van de Velde asked why two versions of user_reduce were being provided. A brief review of the arguments of last time ensued. VOTE: Approve Chapter 5 (Collective Communications) ---- Yes: 21 No: 0 Abstain: 0 ----------------------------------------------------------------------------- The group adjourned for dinner at 6:10pm ============================================================================= Thursday, August 12, 1993 -------- --------------- ------------------------------------------------------------------------------- Report From the Communication Contexts Subcommittee ------------------------------------------------------------------------------- Anthony Skjellum presented. Group, Contexts and Communicators (Chapter 3) - First Reading ------------------------------------------------------------- Introduction (3.1) ------------------ Objection to use of term intra-communication and inter-communication without definition. This is editorial and will be addressed outside the meeting. There are no requirements in this section, so no vote was taken. Context (3.2) ------------- There was confusion about the term "hypertag". This too is editorial. There are no requirements in this section, so no vote was taken. Groups (3.3) ------------ Predefined Groups (3.3.1) ------------------------- It was proposed to change the wording describing MPI_GROUP_ALL to be "all processes at moment of process creation" and to add another group, MPI_GROUP_SIBLING which is "all processes with the same program text." Don Heller asked about system defined server processes and the like. It was agreed to modify the wording to include these. Jim Cownie noted that there is a problem with the notion of HOST because the host would have to have many different versions of MPI_GROUP_HOST. As an alternative Jim proposed that host should acessible via some constant or function that would give the rank of the host in MPI_ALL. After various proposals for additional predefined groups {MPI_GROUP_PEER "all processes except the host (if there is one)", MPI_GROUP_PARENT "parent of all children spawned"} it was proposed that this section be revised to say something like "There are no predefined groups. The effect of predefined groups is gotten by using the groups associated with the predefined communicators." Adam Greenberg asked that a vote on this section NOT be taken until after the decision on which communicators are predefined. Communicators (3.4) ------------------- Predefined Communicators (3.4.1) -------------------------------- MPI_COMM_ALL MPI_COMM_SIBLING MPI_COMM_PEER MPI_COMM_PARENT MPI_COMM_SELF After dealing for a time with the complexity and inclarity of this situation various alternatives were offered. The simplest was to have only MPI_COMM_ALL. Dan Nessett Organization count: 22 VOTE: Revise sections 3.3 and 3.4 as follows: ---- (1) There are no predefined groups (2) The only predefined communicators are MPI_COMM_ALL and MPI_COMM_PEER. (3) There is a predefined MPI_HOST_RANK which gives the rank of the host in the ALL group. It is MPI_UNDEFINED if there is no host. Yes: 17 No: 2 Abstain: 3 VOTE: Approve sections 3.3 and 3.4 as amended. ---- Yes: 18 No: 0 Abstain: 3 Group Management (3.5) ---------------------- Local Operations (3.5.1) ------------------------ There was some discussion of how MPI manages the coordination between various groups (always by relation to the ALL group), which of the inquiry functions in this section properly belong in the environmental sections (none) and general confusion about what the various functions do. No changes were made. VOTE: Approve section 3.5.1 (Local Operations)? ---- Yes: 19 No: 0 Abstain: 3 Local Group Constructors (3.5.2) -------------------------------- Most of the discussion had to do with clarification and editorial corrections. Eric V. asked for a function to reorder the ranks in a group. After some discussion as to exactly what is needed and why it was noted that the MPI_LOCAL_SUBGROUP provides the desired function. It was agreed to add a remark to this effect. It was noted that all of the functions in section 3.5 need descriptions rather than just names and parameters. Marc Snir asked for a clarifying note that of the functions in this section only MPI_LOCAL_SUBGROUP and MPI_LOCAL_SUBGROUP_RANGES can change the ranks. This has as a side effect that the ranges must not overlap. This was agreed. Lyndon Clarke asked for a clarification that the effect of MPI_LOCAL_SUBGROUP_RANGES is as though the ranges were expanded to a list of ranks and MPI_LOCAL_SUBGROUP were called with these ranks. There should be a similar statement for the relation between MPI_LOCAL_SUBGROUP_EXCL_RANGES and MPI_LOCAL_EXCL_SUBGROUP. This was agreed as well. VOTE: Approve section 3.5.2 (Local Group Constructors) as clarified? ---- Yes: 17 No: 0 Abstain: 5 Collective Group Constructors (3.5.3) ------------------------------------- The phrase "a stable sort is used to determine rank order" on line 23 of p. 18 will be change to say that in the event of ties the rank in the comm group will be used to determine the rank in new_group. After discussion of the meaning of MPI_COLL_SUBGROUP it was proposed to have instead: MPI_COLL_SUBCOMM(comm, key, color, new_comm) which will then appear in section 3.7.3. The effect on section 3.5.3 is that it would simply say there there are no collective group constructors. VOTE: Approve section 3.5.3. (Collective Group Constructors) as amended? ---- Yes: 18 No: 0 Abstain: 3 --- break 10:30 - 11 --- Sections 3.6 and 3.7 (pp. 18-21) were handled by giving a function by function discussion followed by an overview of a "tuning" proposal by Marc Snir. Operations on Contexts (3.6) ---------------------------- Local Operations (3.6.1) ------------------------ Collective Operations (3.6.2) ----------------------------- In MPI_CONTEXTS_ALLOC, the len parameter is removed. The "void *" in the descriptions is removed and better words will be provided. The idea of quiescence that was prominent in the context proposal at the last meeting has largely disappeared. The manner of dealing with the problem that quiescence was designed to solve is discussed at length on p. 19. A discussion of the relation of point-to-point and collective communication was prompted by a dispute between Marc Snir and Jim Cownie. Jim made the point that the collective communication routines can be written using the point-to-point communication routines and the material in the context chapter. It was noted that there is one collective communication routine in the context chapter - MPI_CONTEXTS_ALLOC - and some system magic must insure that this works correctly. Marc Snir noted that the system can provide similar magic throughout for optimization purposes. Operations on Communicators (3.7) --------------------------------- Local communicator Operations (3.7.1) ------------------------------------- There were no issues in this section. Local Constructors (3.7.2) -------------------------- There is a MPI_COMM_BIND functions missing. It was accidently deleted and will be added. The form is: MPI_COMM_BIND(group, context, new_comm) IN group IN context OUT new_comm The details will be provided in the next draft. The name of the function MPI_COMM_UNBIND will be changed to MPI_COMM_FREE (and the function of this name in the point-to-point chapter will be renamed.) The frequent reference to MPI_COMM_GROUP(comm) will be changed to MPI_COMM_GROUP(comm, group). Collective Communicator Constructors (3.7.3) -------------------------------------------- The one collective operation for communicators is MPI_COMM_MAKE. Adam Greenberg noted that as currently written every member of the group associated with sync_comm gets comm_new which has At this point Marc Snir An out of context proposal - Only use of context is for local creation of communicators. - Result can be achieved without explicit context object (some loss of safety) - Either case needs ruls for coordinated context allocation. Communication context - specified by communicator - can be "preallocated" and then locally bound to communicator. MPI_CONTEXTS_ALLOC(comm, n) - preallocates n contexts and "caches" them with comm. {This can be called repeatedly and adds the number of contexts specified on each call. This is a collective operation.} MPI_CONTEXTS_FREE(comm, n) - releases up to n preallocated contexts. {This can be called repeatedly. It is a local operation.} MPI_COMM_CONTEXT(comm, n) - queries the number of available preallocated contexts. {This is a local operation.} MPI_COMM_DUP(comm, new_comm) - duplicates a comunicator. Uses locally cached context, if available, otherwise this is a collective operation {It is erroneous if some but not all have locally cached context available. Note that new_comm does NOT have any cached context.) MPI_COM_LDUP(comm, new_comm) - duplicates a communicator. Uses locally cached context and returns NULL if none available. MPI_COMM_MAKE(sync_comm, comm_group, comm_new) MPI_COMM_LMAKE(sync_comm, comm_group, comm_new) - both of these create new communicator associated with the comm_group which is a subgroup of the group of sync_comm. This must be call by all members of the group of sync_comm. Some unease was expressed about sometime collective sometimes not. Safety? In response Marc noted that there could also be MPI_COMM_GDUP which would always do a collective operation. Correctness rule ---------------- All processes in a comm group must execute the same sequence of calls to MPI_CONTEXT_ALLOC, MPI_COTEXT_FREE, MPI_COMM_xDUP, MPI_COMM_xMAKE with comm as argument. - simple to state - same as for collective communication - too conservative? Note: This is compatible with the existence of static preallocated contexts. At this point Lyndon Clarke, having been waving his hand in the air for several minutes, stood on his chair to try and get Marc to address his question. After a brief discuss between Adam, Marc and Tony, Lyndon was allowed to speak. Lyndon Clarke noted that there is no way for the system to check for proper usage of arguments, so this offers no additional security compared with earlier proposals. Others noted that this did provide some additional safety, but it is hard to make direct comparison. Steve Huss asked if any one in the contex comm wanted to keep the current material. Tony said no, but that would likely not be true if Mark Sears were here. STRAW VOTE: Make this into a full proposal to replace the current 3.7.2 & ---------- 3.7.3 Yes: 25 No: 0 Abstain: 2 -- break for lunch 12:10 - 1:40 -- Cacheing (3.9) -------------- Rik Littlefield presented this material. Attribute Cacheing Function: Safely attach arbitrary informatio to groups (and communicators). Purpose: Allow modules to retain or exchange gropup-specific information WITHOUT complicated calling sequences or correctness ruls for use of module Examples Basic Capabilities: - Attributes are local - Attribute value can be pointer to arbitrary structure - Attributes are referenced by a key value obtained from MPI - Attributes can be defined and retrieved - Destructur routine is called when the group (communicator) is freed - Propagation routine is called when the group (communicator) is duplicated. Funtions: MPI_GET_ATTRIBUTE_KEY( OUT keyval) MPI_GROUP_ATTR(IN group, OUT attr_set_handle) MPI_COMM_ATTR(IN comm, OUT attr_set_handle) MPI_SET_ATTRIBUTE( IN attr_set_handle IN keyval IN attribute_value IN *attribute_destructor_routine IN *attribute_propagation_routine ) MPI_TEST_ATTRIBUTE( IN attr_set_handle IN keyval OUT attribute_value OUT result_status ) attribute_destructor_routine(IN attribute_value) attribute_propagation_routine(IN attribute_value, ......) This list is an updated and organized variant of the text. In particular the two routines, MPI_ATTRIBUTE_ALLOC and MPI_ATTRIBUTE_FREE, have been eliminated. What is the rationale for these functions altogether? These provide a method for managing resources associated with groups and communicators. For example this provides facilities to implement the topology facilities on top of MPI. Rik observed that this allows one to effectively extend MPI, e.g. to provide a user written collective operation that can be safely use with MPI and which looks like an MPI routine. Marc Snir asked for a routine to change the value of attributes without having to provide the destructor and propagation routines. There was a question if this introduced a degree of insecurity. Jim Cownie noted that one might well want at attribute with null destructor and propagation routines. Such a reset routine will be provided in the next draft of this chapter. Do we need attributes for both groups and communicators? Why not just on groups? This would allow elimination of attribute handles. There do seem to be situations where it is need on communicators, not just on groups. Adam noted that this proposal puts a resource burden on the system, so he asked about the possiblity of providing only a single system slot with the remainder of the storage provided by the user. Adam is concerned about the admixture of resources at both user and system level. {{{I'm confused}}} Tony proposed adding MPI_ATTRIBUTE_ALLOC and MPI_ATTRIBUTE_FREE back. It was claimed that providing attribute allocate and free routines together with a call back mechanism associated with the free is sufficient to provide all of the functions in section 3.9. Various people countered that this would introduce new problems of coordination and safety. In particular each library might have independent attribute mechanisms and this would require using multiple callback on each call of free. It was noted that this is very similar to the that was solved in X by using a register of callback functions. That can provide a model for this group to use. STRAW VOTE: Do we want a cacheing mechanism? ---------- Yes: 14 No: 4 Abstain: 7 How would toplogy need this? Steve Huss, as virtual Rolf Hemple, noted that topologies need a variety of information (e.g. dimensions, periodicities) that need to be associated with group (and communicators?). As topologies are a part of MPI, a general cacheing mechanism is not required. But without it there is likely to be conflicts between topology and user written libraries. VOTE: Approve section 3.9 (Cacheing) as amended in the presentation? ---- Yes: 8 No: 7 Abstain: 6 -------- break 2:45 - 3:10 Adam Greenberg asked about having a User's Guide meeting tonight. There are enough to have a meeting here after dinner -------- Introduction to Inter-Communication (3.8) ----------------------------------------- Tom Henderson presented on Inter-communication ( A ) <=> ( B ) <=> ( C ) / / Arank Brank Arank: ----- send(...,Brank,tag, commAB) Brank: ----- recv(...,Arank,tag,commAB,...) Want to be able to send from process in one group to process in another group using the rank in the target group. ALTERNATIVES Local group has acess to remote group and have a rank translation in some common ancestor. User maintains tables and communicators for group-pairs. {{{SLIDES HANDED OUT}}} STRAW VOTE: Hear the full proposal? ---------- Yes: 25 No: 1 Abstain: 4 Basic concepts Local Group Remote Group local group leader remote group leader Peer-group that contains both group leaders. {Not to be confused with MPI_COMM_PEER introduced earlier today.} - All members of both groups must call MPI_COMM_PEER_MAKE. What is reason for tag? - serves as the identifier for this particular inter-communicator. Joel Williamson asked why do this rather than just working in MPI_ALL? So can use names that are convenient for the problem at hand. This is also suitable for generalization to a dynamic situation rather than the static situation that is now in MPI. - Why does everyone call PEER_MAKE? To get a common communicator. - Discussion slide Add an IN argument, my_leader_rank, to MPI_COMM_PEER_MAKE(). This allows later addition of dynamic process creation. The IN argument peer_comm need only be valid in the local group leader. Only the group leaders need to be members of peer_comm. - LOOSELY-SYNCHRONOUS INTER-COMMUNICATOR CONSTRUCTOR - SYNCHRONIZATION PROPERTIES OF MPI_COMM_PEER_MAKE_START() AND MPI_COMM_PEER_MAKE_FINISH() - COMMUNICATOR STATUS (convenience function - make go away) - Synchronization issue: Can a process using an inter-communicator send a message using that inter-communicator as soon as it has the inter-communicator? Something needs to be said. - INTER-COMMUNICATOR SUPPORT - EXAMPLE 1 - "UNDER THE HOOD" - INTRA-COMMUNICATION INTER-COMMUNICATION - POSSIBLE IMPLEMENTATION OF MPI_COMM_PEER_MAKE() - POSSIBLE IMPLEMENTATION OF MPI_COMM_PEER_MAKE_START() - POSSIBLE IMPLEMENTATION OF MPI_COMM_PEER_MAKE_FINISH() - What is comparison with using the common ancestor approach? To do this one would create a union group STRAW VOTE: Have inter-communication? ---------- Yes: 14 No: 3 Abstain: 9 VOTE: Approve section 3.8 (Introduction to Inter-Communication) as ---- amended but minus the name service? Yes: 8 No: 2 Abstain: 10 - ? - EXAMPLE 2 - EXAMPLE 3 - EXAMPLE 4 - Al Geist noted all of the examples should be expanded to show at least one message being sent! - POSSIBLE IMPLEMENTATION OF MPI_COMM_NAME_MAKE() - POSSIBLE IMPLEMENTATION OF MPI_COMM_NAME_MAKE_START() - POSSIBLE IMPLEMENTATION OF MPI_COMM_NAME_MAKE_FINISH() - VOTE: Approve the name serverice material in section 3.8 (as amended)? ---- Yes: 8 No: 1 Abstain: 10 ------------------------------------------------------------------------------- Report From the Process Topology Subcommittee ------------------------------------------------------------------------------- Process Topology - Second Reading --------------------------------- Steve Huss presented. {He put on a pair of Birkenstocks to emphasize that he was acting as a virtual Rolf Hemple.} A couple of proposals that were made verbally at the last meeting were not written and so have not been included. Introduction (6.1) ------------------ VOTE: Approve section 6.1 (Introduction) ---- Yes: 13 No: 0 Abstain: 5 Virtual Topologies (6.2) ------------------------ Terminology - change the name to "process topologies" (Eric Van ) or "application topologies" (David Walker). VOTE: Approve section 6.2 (Virtual Topologies)? ---- Yes: 10 No: 0 Abstain: 6 Embedding in MPI (6.3) ---------------------- There were various editorial remarks which Steve recorded to relay to Rolf Hemple. VOTE: Approve section 6.3 (Embedding in MPI)? ---- Yes: 9 No: 0 Abstain: 8 Overview of the proposed MPI functions (6.4) -------------------------------------------- The initial part of 6.4 (before 6.4.1) will go away. VOTE: Specify that we use row major order? ---- Yes: 4 No: 0 Abstain: 13 Steve Huss pointed out that the extent to which these functions are global functions is not specified. Lyndon Clarke offered the amendment that they be specified as collective. There was a discussion of the group parameters in these functions. Steve agreed to propose to Rolf that they be systematically replaced by communicators. VOTE: Replace all group parameters throughout by communicator? ---- Yes: 8 No: 1 Abstain: 7 VOTE: MPI_MAP_CART and MPI_MAP_GRAPH shall be global routines? ---- Yes: 4 No: 2 Abstain: 12 The large number of abstentions in the recent votes led to a discussion of the value of topology and also to a discussion of our procedures. There was no strong interest in discussing including topology. Neither was there any strong interest in changing procedures. VOTE: Postpone the second reading of the topology chapter until the next ---- meeting? Yes: 3 No: 12 Abstain: 2 VOTE: Approve chapter 6 as amended? ---- Yes: 12 No: 3 Abstain: 2 ------------------------------------------------------------------------------- Report from the Subset Subcommittee ------------------------------------------------------------------------------- Steve Huss presided. This was NOT a second reading. Jim Cownie argued that the profiling material should be included in the subset because this provides important facilities and the cost of providing it in an initial implementation is not large. Several people agreed with this, so there was a quick vote. STRAW VOTE: Include profiling in the subset? ---------- Yes: 24 No: 0 Abstain: 1 A discussion of the parts of environmental management and inquiry to be included lead to an agreement that this should be deferred until the presentation on this material. There was nothing to be said about language binding - there will be F77 and C bindings. STRAW VOTE: Exclude topology from the subset? ---------- Yes: 21 No: 2 Abstain: 3 It was agreed that the list for collective communication in the document is OK. In considering the Point-to-point Functions, the list in the document includes MPI_SENDRECV but excludes MPI_EXCHANGE. It was generally agreed that this is sensible. Steve Otto argued for including hvec type functions in the subset because of common usage. In considering this and the issue of derived datatype several posibilities were considered. The one that got general support is to include all of the material in section 4.13. It was noted that data conversion is not a subset issue - heterogeneous systems have to have it; homogeneous systems do not need it. --- break for dinner at 6:05 --- Subcommitte meetings - starting about 8:30 Subset - immediately after dinner User Guide - after subset meeting Context - after subset meeting ============================================================================= Friday, June 25, 1993 ------ ------- ---- ------------------------------------------------------------------------------- Report from the Environmental Management Subcommittee ------------------------------------------------------------------------------- Rusty Lusk presided. Rusty began by handing out a new version Environmental Management and Inquiry 1 Initialization 2 Environmental query 3 Others Initialization and Termination Current draft: MPI_INIT() "idempotent" MPI_FINALIZE() "last MPI call" Discussion: How does a library know whether to call MPI_FINALIZE? Is MPI_FINALIZE optional? MPI_INIT requires some state not attached to any object; why not a communicator? MPI_INIT requires some state not attached to any object; why not a communicator? A proposed amendment: MPI_INIT(old_comm, new_comm) {if old_comm is null, gets first communicator} MPI_FINALIZE(current_comm, old_comm) Nests MPI invocations Attaches MPI state to communicator STRAW VOTE: We shall provide a mechanism that allows a library written ---------- using to be called from either within or without MPI? Yes: 11 No: 8 Abstain: 3 Steve Huss asks what happens if two libraries are invoked using different numbers of processors, then what is the ALL group? Jim Cownie offered a very simple proposal - All processes must call MPI_INIT at the start and all processes must call MPI_FINALIZE at the end. Note that the picture is that by a vendor provided miracle the MPI system is started and only after this can MPI_INIT be called. This is likely to be a global barrier. STRAW VOTE: MPI_INIT and MPI_FINALIZE must be called exactly once in each ---------- process. Yes: 18 No: 0 Abstain: 5 organization count: 20 Lotes of continuing discussion. It was agreed that, in the context of the straw vote, any program that violated the requirement is erroneous. VOTE: Have an MPI_INITIALIZED flag? Yes: 16 No: 1 Abstain: 2 VOTE: MPI_INIT and MPI_FINALIZE must be called exactly once in each ---- process. MPI_INIT is a global operation. It must be called before any other MPI routine. MPI_FINALIZE is the last MPI call. Yes: 16 No: 1 Abstain: 3 Rusty offered a proposal for MPI_ABORT(error_code) which terminates every process in the ALL group. VOTE: Have MPI_ABORT? ---- Yes: 13 No: 3 Abstain: 2 MPI-Specific (1.1) [Section numbers from chapter handed out at meeting] ------------------ Why are there communicator arguments in MPI_NumCommunicator? Rusty did not know. No one had a convincing argument. VOTE: Remove communicator arguments from MPI_ValidTags and ---- MPI_NumCommunicator? Yes: 14 No: 0 Abstain: 5 There was a fair amount of confusion about the intent and value of the MPI_BufferParams routine. In particular, various alternative proposals were mentioned. Rik Littlefield has proposed that the user be able to specify buffering capability STRAW VOTE: Should there be some way of asking the system about buffering? ---------- Yes: 15 No: 2 Abstain: 7 STRAW VOTE: Should there be some way of telling the system about buffer ---------- requirements? Yes: 6 No: 3 Abstain: 15 Rik will provide a proposal at the next meeting. VOTE: Remove MPI_IOmode? ---- Yes: 18 No: 1 Abstain: 2 Discussion of MPI_Errormode? There was again uncertainty about the communicator argument leading to: VOTE: Remove the communicator argument from MPI_Errormode? ---- Yes: 7 No: 4 Abstain: 8 After further discussion it was realized that while something of this sort is desirable, that is much more detail (e.g. how are error handlers established) that is essential before accepting this function. STRAW VOTE: Should there be a facility to set and query error mode? ---------- Yes: 18 No: 0 Abstain: 0 There was quick agreement that MPI_Has_Nonblocking and MPI_Has_Heterogeneous are not useful. VOTE: Remove MPI_Has_Nonblocking and MPI_Has_Heterogeneous? ---- Yes: 9 No: 0 Abstain: 4 STRAW VOTE: Have functions to inspect receive queue and other interesting ---------- internal structures? Yes: 12 No: 3 Abstain: 2 Anne Elster again asked for MPI_LOAD_INFO. She was proposing a modified version that had less time-specific information, but no written proposal was available at the meeting. A concrete proposal will be seen at the next meeting. VOTE: Remove sections 1.2 (Parallel programming) and 1.3 (non-MPI) except keep for MPI_Ge ---- Yes: 8 No: 3 Abstain: 2 VOTE: Accept MPI Environmental Management chapter as amended. ---- Yes: 8 No: 2 Abstain: 3 ------------------------------------------------------------------------------ MPI Sound Bites Jim Cownie Where's David? Oh No! Don't think about that one too much What's the question again? Those in favor of going to the bar? Shall we accept the chapter as eviscerated? ------------------------------------------------------------------------------ Report from the Language Binding Subcommittee ------------------------------------------------------------------------------- Rusty Lusk presided. Language Binding 7.1-7.4 will go into another chapter (MPI Terms and Conventions) 7.5 will go into an Appendix We need to: 1. Update and read 7.1-7.4 2. Decide on principles for binding presentation 3. Decide on format of definitions in the chapters 4. Decide on format and order of definitions in appendix. 5. Choose a procedure for agreeing on names of C functions, Fortran subroutines, named constants, types. 6. Enforce consistency 1. Modification to 7.1-7.4 (see draft) 2. Presenting the bindings a. Named constants (MPI_SUCCESS) b. Named types (MPI_COMMUNICATOR) c. Functions and arguments i. C - use ANSI C style ii. Fortran - use prototypes and declarations d. Consistency of formal argument names e. IN arguments before OUT arguments f. return code last argument in FORTRAN g. others? 3. Current 7.5 OK modulo name updating? Jim Cownie argued that the principle that "all C functions should return an error code" should be relaxed for those functions that would best be implemented using macros. VOTE: Accept chapter and annex with the modifications outline by Rusty ---- Yes: 12 No: 0 Abstain: 1 Format In the chapters Current Format <- match C binding in order and number of arguments + Fortran Binding <- match appendix + C Binding <- match appendix In the appendix, Sort by appearance order? Alphabetically within chapter? <-- this one chosen Alphabetically? Keep appendix after using if for consistency check? (Note alphabetical index) As a technical issue, Steve Otto would like to have the bindings appear in the chapter source, but only appear printed in the appendices. There was general agreement that in the appendix the functions should appear alphabetically within each chapter. ------------------------------------------------------------------------------ From owner-mpi-comm@CS.UTK.EDU Wed Sep 22 14:31:24 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA10658; Wed, 22 Sep 93 14:31:24 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02812; Wed, 22 Sep 93 14:27:56 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 22 Sep 1993 14:27:54 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sun2.nsfnet-relay.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02781; Wed, 22 Sep 93 14:27:52 -0400 Via: uk.ac.daresbury.cxa; Wed, 22 Sep 1993 19:26:34 +0100 From: "R.J. Allan" Date: Wed, 22 Sep 93 19:22:20 +0100 Message-Id: <21256.9309221822@uk.ac.dl.cxa> To: owner-mpi-comm@CS.UTK.EDU Cc: mpi-comm@cs.utk.edu, elster@cs.cornell.edu In-Reply-To: Anne C. Elster's message of Thu, 16 Sep 93 03:44:56 -0400 <9309160744.AA24612@elli.cs.cornell.edu> Subject: MPI_LOAD A quick comment on load balancing and message-passing harnesses. We have done load balancing using our own Fortnet harness on, for instance, a domain-decomposed CFD code. This is easy to do (given an algorithm!) providing you have access to CPU time and WALL time, and can time over computation and communication parts of the code. I am pleased to see that MPI has this facility already, more general load tools are very debatable... and probably program dependent in some way Cheers, Robert J. Allan Advanced Research Computing Group, Science and Engineering Research Council, Daresbury Laboratory, Daresbury, Warrington, WA4 4AD, U.K. 'phone +44 925 603207 FAX +44 925 603634 e-mail r.j.allan@daresbury.ac.uk From owner-mpi-comm@CS.UTK.EDU Wed Sep 22 16:19:13 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA11864; Wed, 22 Sep 93 16:19:13 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11362; Wed, 22 Sep 93 16:11:06 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 22 Sep 1993 16:11:05 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11354; Wed, 22 Sep 93 16:11:03 -0400 Received: from elephant (elephant.parasoft.com) by sampson.ccsf.caltech.edu with SMTP id AA26994 (5.65c/IDA-1.4.4 for mpi-comm@cs.utk.edu); Wed, 22 Sep 1993 13:11:01 -0700 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA03022; Wed, 22 Sep 93 13:04:11 PDT Received: by lion.parasoft (4.1/SMI-4.1) id AA11731; Wed, 22 Sep 93 13:13:44 PDT Date: Wed, 22 Sep 93 13:13:44 PDT From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9309222013.AA11731@lion.parasoft> To: mpi-comm@cs.utk.edu Subject: MPI_LOAD I agree with Robert Allan -- we have CPU and wallclock statistics inside Express and actually use it when computing grid mappings (i.e., virtual toplogies). It works remarkably well - better than I think it would be reasonable to expect. If anyone remembers the "wall of SGI's" that we had at Supercomputing '92 we had a CFD code that would migrate to and fro between 8 SGI's, and HP and an RS/6000 depending on what other "stuff" was chewing up their resources. So..... I think that MPI would probably benefit a lot from standardizing this type of call because it would encourage people to play, and potentially learn a lot about the rather under-explored field of dynamic load balancing. HOWEVER, having said that I don't think it's really in MPI's charter to include this stuff, just as it doesn't include high resolution timers. Of course, I was one of the people that wanted that feature too! Jon From owner-mpi-comm@CS.UTK.EDU Tue Oct 19 07:36:42 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA02869; Tue, 19 Oct 93 07:36:42 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930922/2.8s-UTK) id AA25225; Tue, 19 Oct 93 07:32:12 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 19 Oct 1993 07:32:11 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930922/2.8s-UTK) id AA25217; Tue, 19 Oct 93 07:32:07 -0400 Date: Tue, 19 Oct 93 12:31:48 BST Message-Id: <29074.9310191131@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: mpi; future availability To: mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Howdy y'all This may sound a little premature, but its something are frequently asked, so I just thought I'd see if there is an answer. I'd be really grateful for any real answers. We know that there is planned a demonstration of MPI at Supercomputing, and an MPI workshop. What will be the availability status of the demonstrator implementation, and of the TCP/IP mpid which is being written (by Tony et al, I recall)? Are any of the vendors able to give quotable information about when implementations of MPI will be released for their machines? Its actually quite important to know these things for acceptance of MPI in project planning, and we are doing our bit to promote acceptance of MPI, and have a lot of project planning here at the moment. Thanks and Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Sat Nov 27 04:50:26 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id EAA29159; Sat, 27 Nov 1993 04:50:25 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id EAA19651; Sat, 27 Nov 1993 04:48:33 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sat, 27 Nov 1993 04:48:32 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id EAA19644; Sat, 27 Nov 1993 04:48:31 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA00442; Sat, 27 Nov 93 03:48:45 CST Date: Sat, 27 Nov 93 03:48:45 CST From: Tony Skjellum Message-Id: <9311270948.AA00442@Aurora.CS.MsState.Edu> To: mpi-comm@CS.UTK.EDU Subject: MPI Libraries Paper Dear colleagues, following up on my promise at SC'93 minisymposium, the paper "Writing Libraries in MPI" is now available on anonymous ftp. site: aurora.cs.msstate.edu directory: pub/reports file: mpi_lib_27nov93.ps.Z This will appear in the Proceedings of the Scalable Libraries Conference, to be published by IEEE Press. -Tony Skjellum From owner-mpi-comm@CS.UTK.EDU Wed Jan 12 08:54:53 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id IAA14110; Wed, 12 Jan 1994 08:54:53 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id IAA15141; Wed, 12 Jan 1994 08:53:44 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 12 Jan 1994 08:53:42 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from dasher.cs.utk.edu by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id IAA15134; Wed, 12 Jan 1994 08:53:41 -0500 From: Jack Dongarra Received: by dasher.cs.utk.edu (5.61+IDA+UTK-930922/2.7c-UTK) id AA26974; Wed, 12 Jan 94 08:52:54 -0500 Date: Wed, 12 Jan 94 08:52:54 -0500 Message-Id: <9401121352.AA26974@dasher.cs.utk.edu> To: mpi-comm@CS.UTK.EDU Subject: ``Final'' MPI meeting We are planning to have the ``Final'' MPI Meeting in Knoxville, Tennessee February 23rd-25th. Plan to fly into the Knoxville airport, called the Knoxville McGhee-Tyson airport. We have made arrangements with the Hilton Hotel in downtown Knoxville. Hilton Hotel 501 W. Church Street Knoxville, TN Phone: 615-523-2300 The room rate is $64.00. When making a reservation tell them you are with the MPI meeting. Rooms under MPI for the above rate will be available until Feb. 1, after that, they will be on an availability basis. The hotel provides a shuttle service from the airport to the hotel. There is a hotel phone located in the airport lower level at baggage pickup. Let the hotel know you have arrived and are with MPI and they will arrange to pick you up; or you can call the hotel in advance, give your flight # and arrival time, and the hotel will go ahead and schedule a pick up. The hotel shuttle leaves the hotel on-the-hour and picks up at the airport on-the-half-hour so there could be a delay. You can rent a car or get a cab from the airport to the hotel as well. Its about a 15-20 minute ride. (There is a Hilton located at the Knoxville airport. Do not confuse this hotel with the meeting Hilton located in downtown Knoxville.) We should plan to start at 1:00 pm Wednesday February 23rd and finish about 12:00 pm on Friday February 25th. The registration fee for the meeting will be $75. Please make checks and POs payable to University of Tennessee. We will collect this at the meeting. The registration fee will go for coffee breaks, meeting rooms, AV. The format of the meeting is as before: Wednesday, February 23rd 1:00 pm to 8:00 pm 2 Breakouts for 10 people 1 Breakout for 20 people 5:00 pm to 7:00 pm--Dinner. Thursday, February 24th 8:00 am to 10:00 pm 1 U-shape room for 40 people 6:00 pm to 8:00 pm-- The group dinner somewhere in the area. Friday, February 25th 8:00 am to 12:00 pm 1 U-shape room for 40 people If you have any questions, please feel free to contact me. Jack From owner-mpi-comm@CS.UTK.EDU Sat Jan 15 10:01:22 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id KAA10048; Sat, 15 Jan 1994 10:01:16 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id KAA00645; Sat, 15 Jan 1994 10:00:22 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sat, 15 Jan 1994 10:00:16 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id JAA00574; Sat, 15 Jan 1994 09:59:52 -0500 Received: from WATSON by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 0223; Sat, 15 Jan 94 09:59:53 EST Received: from YKTVMH by watson.vnet.ibm.com with "VAGENT.V1.0" id 4470; Sat, 15 Jan 1994 09:59:24 EST Received: from snir.watson.ibm.com by yktvmh.watson.ibm.com (IBM VM SMTP V2R3) with TCP; Sat, 15 Jan 94 09:59:15 EST Received: by snir.watson.ibm.com (AIX 3.2/UCB 5.64/930311) id AA40957; Sat, 15 Jan 1994 09:59:14 -0500 From: snir@watson.ibm.com (Marc Snir) Message-Id: <9401151459.AA40957@snir.watson.ibm.com> To: mpi-comm@CS.UTK.EDU Subject: a proposal for MPI-IO (don't worry -- this is for MPI-II) Reply-To: snir@watson.ibm.com Date: Sat, 15 Jan 94 09:59:11 -0500 -:) -:) -:) -:) -:) %!PS-Adobe-2.0 %%Creator: dvips 5.47 (RS/6000 1.0) Copyright 1986-91 Radical Eye Software %%Title: mpi-io-v1.dvi %%Pages: 28 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: tex.pro /TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{ ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image} imagemask restore}B /D{/cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval(Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{ moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{ S p tail}B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w }B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{clear SS restore}B end %%EndProcSet %%BeginProcSet: special.pro TeXDict begin /SDict 200 dict N SDict begin /@SpecialDefaults{/hs 612 N /vs 792 N /ho 0 N /vo 0 N /hsc 1 N /vsc 1 N /ang 0 N /CLIP false N /BBcalc false N /p 3 def}B /@scaleunit 100 N /@hscale{@scaleunit div /hsc X}B /@vscale{ @scaleunit div /vsc X}B /@hsize{/hs X /CLIP true N}B /@vsize{/vs X /CLIP true N}B /@hoffset{/ho X}B /@voffset{/vo X}B /@angle{/ang X}B /@rwi{10 div /rwi X} B /@llx{/llx X}B /@lly{/lly X}B /@urx{/urx X}B /@ury{/ury X /BBcalc true N}B /magscale true def end /@MacSetUp{userdict /md known{userdict /md get type /dicttype eq{md begin /letter{}N /note{}N /legal{}N /od{txpose 1 0 mtx defaultmatrix dtransform S atan/pa X newpath clippath mark{transform{ itransform moveto}}{transform{itransform lineto}}{6 -2 roll transform 6 -2 roll transform 6 -2 roll transform{itransform 6 2 roll itransform 6 2 roll itransform 6 2 roll curveto}}{{closepath}}pathforall newpath counttomark array astore /gc xdf pop ct 39 0 put 10 fz 0 fs 2 F/|______Courier fnt invertflag{ PaintBlack}if}N /txpose{pxs pys scale ppr aload pop por{noflips{pop S neg S TR pop 1 -1 scale}if xflip yflip and{pop S neg S TR 180 rotate 1 -1 scale ppr 3 get ppr 1 get neg sub neg ppr 2 get ppr 0 get neg sub neg TR}if xflip yflip not and{pop S neg S TR pop 180 rotate ppr 3 get ppr 1 get neg sub neg 0 TR}if yflip xflip not and{ppr 1 get neg ppr 0 get neg TR}if}{noflips{TR pop pop 270 rotate 1 -1 scale}if xflip yflip and{TR pop pop 90 rotate 1 -1 scale ppr 3 get ppr 1 get neg sub neg ppr 2 get ppr 0 get neg sub neg TR}if xflip yflip not and{TR pop pop 90 rotate ppr 3 get ppr 1 get neg sub neg 0 TR}if yflip xflip not and{TR pop pop 270 rotate ppr 2 get ppr 0 get neg sub neg 0 S TR}if} ifelse scaleby96{ppr aload pop 4 -1 roll add 2 div 3 1 roll add 2 div 2 copy TR .96 dup scale neg S neg S TR}if}N /cp{pop pop showpage pm restore}N end}if} if}N /normalscale{Resolution 72 div VResolution 72 div neg scale magscale{ DVImag dup scale}if}N /psfts{S 65536 div N}N /startTexFig{/psf$SavedState save N userdict maxlength dict begin /magscale false def normalscale currentpoint TR /psf$ury psfts /psf$urx psfts /psf$lly psfts /psf$llx psfts /psf$y psfts /psf$x psfts currentpoint /psf$cy X /psf$cx X /psf$sx psf$x psf$urx psf$llx sub div N /psf$sy psf$y psf$ury psf$lly sub div N psf$sx psf$sy scale psf$cx psf$sx div psf$llx sub psf$cy psf$sy div psf$ury sub TR /showpage{}N /erasepage{}N /copypage{}N /p 3 def @MacSetUp}N /doclip{psf$llx psf$lly psf$urx psf$ury currentpoint 6 2 roll newpath 4 copy 4 2 roll moveto 6 -1 roll S lineto S lineto S lineto closepath clip newpath moveto}N /endTexFig{end psf$SavedState restore}N /@beginspecial{SDict begin /SpecialSave save N gsave normalscale currentpoint TR @SpecialDefaults}N /@setspecial{CLIP{newpath 0 0 moveto hs 0 rlineto 0 vs rlineto hs neg 0 rlineto closepath clip}if ho vo TR hsc vsc scale ang rotate BBcalc{rwi urx llx sub div dup scale llx neg lly neg TR}if /showpage{}N /erasepage{}N /copypage{}N newpath}N /@endspecial{grestore clear SpecialSave restore end}N /@defspecial{SDict begin}N /@fedspecial{end}B /li{lineto}B /rl{rlineto}B /rc{rcurveto}B /np{/SaveX currentpoint /SaveY X N 1 setlinecap newpath}N /st{stroke SaveX SaveY moveto}N /fil{fill SaveX SaveY moveto}N /ellipse{/endangle X /startangle X /yrad X /xrad X /savematrix matrix currentmatrix N TR xrad yrad scale 0 0 1 startangle endangle arc savematrix setmatrix}N end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 3 50 df<00600000600000600000600000600000 6000006000006000006000006000FFFFF0FFFFF000600000600000600000600000600000600000 600000600000600000600014167E9119>43 D<0FC01FE0387070386018E01CE01CE01CE01CE01C E01CE01CE01CE01C6018703838701FE00FC00E137F9211>48 D<06001E00FE00EE000E000E000E 000E000E000E000E000E000E000E000E000E000E00FFE0FFE00B137D9211>I E /Fb 1 2 df<60F0F06004047E880A>1 D E /Fc 10 119 df<60F0F07030303060E040040A7E 830A>59 D<07FFE007FFF801C03801C01C01C01C01C01C0380380380700381E003FFC007FFE007 00700700700700380E00700E00700E00E00E03C0FFFF80FFFE0016147F9319>66 D<07FC7FC007FC7FC001C01C0001C01C0001C01C0001C01C0003803800038038000380380003FF F80007FFF0000700700007007000070070000E00E0000E00E0000E00E0000E00E000FF8FF800FF 8FF8001A147F931B>72 D<3E003E000C000C000C000C00180019E01FF01E303830383030303030 7060606660C660CCC0FCC0700F147E9313>104 D<03000380030000000000000000001C003E00 6600C600CC000C000C0018001980318033003F001C00091480930C>I<00600070006000000000 00000000038007C00CE018C018C000C000C0018001800180018003000300030003006600EE00FC 0078000C1A81930E>I<387C007CFE006F8600CF0600CE06000E06000C06001C0C00180CC01818 C0181980301F80300E00120D808C15>110 D<38E07DF06F30CE70CE300C000C00180018001800 1800300030000C0D808C0F>114 D<07001F8019C0318030001E001F0003804180E180C300FE00 7C000A0D7E8C10>I<1C083E186618C618CC180C180C181830183018201C600FC007800D0D808C 11>118 D E /Fd 27 123 df<70F8F8F87005057C840D>58 D<70F8FCFC7C0C0C0C1818383060 40060E7C840D>I<000001C00000078000001E00000078000001E00000078000000E0000003C00 0000F0000003C000000F0000003C000000F0000000F00000003C0000000F00000003C0000000F0 0000003C0000000E0000000780000001E0000000780000001E0000000780000001C01A1A7C9723 >I<00030003000700060006000E000C000C001C0018001800380030003000700060006000E000 C000C001C00180018001800380030003000700060006000E000C000C001C001800180038003000 3000700060006000E000C000C000102D7DA117>II<00FFFFE000FFFFF8000F007C000F003C000F003E001E003E 001E003E001E003E001E003E003C007C003C007C003C00F8003C01F0007807E0007FFF80007FFF C0007803E000F001F000F001F000F000F000F000F001E001F001E001F001E001F001E001E003C0 03E003C007C003C00F8003C01F007FFFFE00FFFFF0001F1F7E9E22>66 D<00FFF9FFF000FFF9FF F0000F001E00000F001E00000F001E00001E003C00001E003C00001E003C00001E003C00003C00 7800003C007800003C007800003C007800007800F000007FFFF000007FFFF000007800F00000F0 01E00000F001E00000F001E00000F001E00001E003C00001E003C00001E003C00001E003C00003 C007800003C007800003C007800003C00780007FFCFFF800FFF8FFF800241F7E9E26>72 D<00FFFC01FFF8000F00000F00000F00001E00001E00001E00001E00003C00003C00003C00003C 0000780000780000780000780000F00000F00000F00000F00001E00001E00001E00001E00003C0 0003C00003C00003C000FFFC00FFFC00161F7F9E14>I86 D<00F18003FDC0078F800E07801C07803C07803C0700780700780700780700F00E00F00E00 F00E00F00E18F01C30F03C30707C3078FC603FCFC00F078015147E9318>97 D<07803F803F000700070007000E000E000E000E001C001C001CF01FFC3F1E3E0E3C0F380F700F 700F700F700FE01EE01EE01EE03CE03CE038607071E03FC01F0010207E9F14>I<007C01FE0787 0F0F1E0F1C0E3C04780078007800F000F000F000F000F00070027007381E1FF80FE010147E9314 >I<007C01FE07870E031E033C033C06781C7FF87FC0F000F000F000F000700070027007381E1F F807E010147E9315>101 D<01E0000FE0000FC00001C00001C00001C000038000038000038000 038000070000070000071F00077F800EE1C00F81C00F00E00E00E01E01C01C01C01C01C01C01C0 38038038038038038038070670070C700E0C700E18700638E007F06003C017207E9F1A>104 D<007000F000F00060000000000000000000000000000007001F8039C031C061C061C0C3800380 0380070007000E000E000E181C301C301C601CE00FC007000D1F7F9E10>I<0000C00001E00001 E00001C0000000000000000000000000000000000000000000003E00007F0000C3800183800303 80030380060700000700000700000700000E00000E00000E00000E00001C00001C00001C00001C 0000380000380000380000380000700000700030700078E000F3C0007F80003E00001328819E13 >I<01E0000FE0000FC00001C00001C00001C00003800003800003800003800007000007000007 01C00707E00E0C300E18700E30F00E60F01CC0E01F80001F80001FE00038F00038380038380038 1C307038607038607038407018C0E01F80600F0014207E9F18>I<03C01FC01F80038003800380 07000700070007000E000E000E000E001C001C001C001C00380038003800380070007000700071 80E300E300E300E6007E003C000A207E9F0E>I<0F03E03E001F8FF0FF0031DC39C38061F03B03 8061E01E01C061C01C01C0C3C03C03800380380380038038038003803803800700700700070070 070007007007000700700E0C0E00E00E180E00E01C180E00E01C300E00E00C701C01C00FE00C00 C0078026147F9328>I<0F07C0001F8FE00031D8700061F0380061E0380061C03800C3C0700003 80700003807000038070000700E0000700E0000700E0000701C1800E01C3000E0383000E038600 0E018E001C01FC000C00F00019147F931B>I<007C0001FF000383800F01C01E01C01C01E03C01 E07801E07801E07801E0F003C0F003C0F003C0F00780F00700700F00701E003838001FF00007C0 0013147E9316>I<03C1E007E7F80C7E3C187C1C18781E18701E30E01E00E01E00E01E00E01E01 C03C01C03C01C03C01C07803C07803C07003C0E003E3C0077F80071E000700000700000E00000E 00000E00000E00001C0000FFC000FFC000171D819317>I<0F0F001F9F8031F1C061E1C061E3C0 61C3C0C3C3800380000380000380000700000700000700000700000E00000E00000E00000E0000 1C00000C000012147F9315>114 D<007C01FE0387070F070F070E0F00078007F803FC01FE001E 000F700FF00EF00EE01C70383FF01FC010147E9315>I<00C000E001C001C001C001C003800380 FFF8FFF807000700070007000E000E000E000E001C001C001C001C1838303830386038E01FC00F 000D1C7F9B10>I<0780C01FC1E038C1F030E0F060E0F061C070C1C06003806003806003806007 00C00700C00700C0070180070180070300070200078E0003FC0000F00014147F9316>118 D<01E06003F0C007FFC00E1F800C0300000600000C0000180000300000600000C0000180000300 000600C00C01801803803F87007FFE0061FC00C0700013147E9315>122 D E /Fe 5 81 df<0018007800F001E003C007800F001F001E003E003C007C007C007800F800F8 00F800F800F800F800F800F800F800F800F800F800F800F800F800F800F800F800F800F800F800 F800F8000D25707E25>56 D58 D<007C007C007C007C007C007C007C007C007C007C007C00 7C007C007C007C007C007C007C007C007C007C007C007C007C00F800F800F800F001F001E003E0 03C0078007000E001C003800F000C000F00038001C000E000700078003C003E001E001F000F000 F800F800F8007C007C007C007C007C007C007C007C007C007C007C007C007C007C007C007C007C 007C007C007C007C007C007C007C0E4D798025>60 D62 D80 D E /Ff 1 106 df105 D E /Fg 5 22 df0 D<70F8F8F87005057C8D0D>I<07E01FF8 3FFC7FFE7FFEFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF7FFE7FFE3FFC1FF807E010127D9317>15 D<000000C0000003C000000F0000003C000000F0000003C000000F0000001C00000078000001E0 0000078000001E00000078000000E0000000780000001E0000000780000001E000000078000000 1C0000000F00000003C0000000F00000003C0000000F00000003C0000000C00000000000000000 0000000000000000000000000000000000000000FFFFFFC0FFFFFFC01A247C9C23>20 DI E /Fh 56 123 df<007000F001F003C007800F001E001C003C003800780070007000F000E000E0 00E000E000E000E000E000E000F00070007000780038003C001C001E000F00078003C001F000F0 00700C24799F18>40 D<6000F00078003C001E000F000780038003C001C001E000E000E000F000 70007000700070007000700070007000F000E000E001E001C003C0038007800F001E003C007800 F00060000C247C9F18>I<01C00001C00001C00001C000C1C180F1C780F9CF807FFF001FFC0007 F00007F0001FFC007FFF00F9CF80F1C780C1C18001C00001C00001C00001C00011147D9718>I< 3C7E7F7F7F3F0F0E1E7CF870080C788518>44 D<000300000F80001F80003F0000FE0001FC0003 F00007E0001FC0003F80007E0000FC0000FC00007E00003F80001FC00007E00003F00001FC0000 FE00003F00001F80000F8000030011187D9918>60 D<600000F80000FC00007E00003F80001FC0 0007E00003F00001FC0000FE00003F00001F80001F80003F0000FE0001FC0003F00007E0001FC0 003F80007E0000FC0000F8000060000011187D9918>62 D<00700000F80000F80000D80000D800 01DC0001DC0001DC00018C00038E00038E00038E00038E00030600070700070700070700070700 0FFF800FFF800FFF800E03800E03801C01C01C01C07F07F0FF8FF87F07F0151C7F9B18>65 DI<01FCE003FEE007FFE00F07E01E03E03C01E07800E07000E070 00E0F00000E00000E00000E00000E00000E00000E00000E00000E00000F000007000E07000E078 00E03C01E01E01C00F07C007FF8003FF0001FC00131C7E9B18>I<7FF800FFFE007FFF001C0F80 1C03C01C03C01C01E01C00E01C00E01C00F01C00701C00701C00701C00701C00701C00701C0070 1C00701C00F01C00E01C00E01C01E01C01C01C03C01C0F807FFF00FFFE007FF800141C7F9B18> III<01F9C007FFC00FFFC01F0FC0 1C03C03C03C07801C07001C07001C0F00000E00000E00000E00000E00000E00000E00FF0E01FF0 E00FF0F001C07001C07003C07803C03C03C01C07C01F0FC00FFFC007FDC001F9C0141C7E9B18> I<7F07F0FF8FF87F07F01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01FFF C01FFFC01FFFC01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C07F07 F0FF8FF87F07F0151C7F9B18>I<7FFF00FFFF807FFF0001C00001C00001C00001C00001C00001 C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001 C00001C00001C00001C0007FFF00FFFF807FFF00111C7D9B18>I<7F07F0FF87F87F07F01C03C0 1C07801C07001C0F001C1E001C3C001C38001C78001CF0001DF0001DF8001FF8001FBC001F1C00 1E1E001E0E001C0F001C07001C07801C03801C03C01C01C07F03F0FF87F87F03F0151C7F9B18> 75 D<7FE000FFE0007FE0000E00000E00000E00000E00000E00000E00000E00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E00700E00700E00700E00700E0070 7FFFF0FFFFF07FFFF0141C7F9B18>II<7E07F0FF0FF87F07F01D 81C01D81C01D81C01DC1C01CC1C01CC1C01CE1C01CE1C01CE1C01C61C01C71C01C71C01C31C01C 39C01C39C01C39C01C19C01C19C01C1DC01C0DC01C0DC01C0DC07F07C0FF87C07F03C0151C7F9B 18>I<0FF8003FFE007FFF00780F00700700F00780E00380E00380E00380E00380E00380E00380 E00380E00380E00380E00380E00380E00380E00380E00380E00380E00380F00780700700780F00 7FFF003FFE000FF800111C7D9B18>II<0FF8003FFE007FFF0078 0F00700700F00780E00380E00380E00380E00380E00380E00380E00380E00380E00380E00380E0 0380E00380E00380E00380E1E380E1E380F0E78070F700787F007FFF003FFE000FFC00001C0000 1E00000E00000F0000070000070011227D9B18>I<7FF800FFFE007FFF001C0F801C03801C03C0 1C01C01C01C01C01C01C03C01C03801C0F801FFF001FFE001FFE001C0F001C07801C03801C0380 1C03801C03801C03801C039C1C039C1C039C7F03FCFF81F87F00F0161C7F9B18>I<07F3801FFF 803FFF807C1F80700780F00380E00380E00380E00000F000007800003F00001FF0000FFE0001FF 00001F800003C00001E00000E00000E06000E0E000E0E001E0F001C0FC07C0FFFF80FFFF00E7FC 00131C7E9B18>I<7FFFF8FFFFF8FFFFF8E07038E07038E07038E0703800700000700000700000 700000700000700000700000700000700000700000700000700000700000700000700000700000 700000700007FF0007FF0007FF00151C7F9B18>IIII<7F8FE07F9FE07F8FE00E07000F0700070E00 078E00039C0003DC0001F80001F80000F00000F00000700000F00000F80001F80001DC00039E00 038E00070F000707000E07800E03801E03C07F07F0FF8FF87F07F0151C7F9B18>II<3FFFE07FFFE07FFFE07001C07003C0700780700700000F00001E00001C0000 3C0000780000700000F00001E00001C00003C0000780000700000F00001E00E01C00E03C00E078 00E07000E0FFFFE0FFFFE0FFFFE0131C7E9B18>I<1FE0003FF8007FFC00783E00300F00000700 00070001FF000FFF003FFF007F0700780700F00700E00700E00700F00F00783F007FFFF03FFBF0 0FE1F014147D9318>97 D<7E0000FE00007E00000E00000E00000E00000E00000E00000E3E000E FF800FFFC00FE3E00F80F00F00700F00780E00380E00380E00380E00380E00380F00380F00780F 00700F80F00FC3E00FFFC00EFF80067E00151C809B18>I<01FE0007FF001FFF803F07803C0300 780000700000F00000E00000E00000E00000E00000F000007000007801C03C01C03F07C01FFF80 07FF0001FC0012147D9318>I<001F80003F80001F8000038000038000038000038000038003F3 800FFB801FFF803E1F80780F80700780F00780E00380E00380E00380E00380E00380E00780F007 80700780780F803E3F801FFFF00FFBF803E3F0151C7E9B18>I<03F0000FFC001FFE003E1F0078 0780700380F003C0E001C0E001C0FFFFC0FFFFC0FFFFC0F000007000007801C03C01C03F07C01F FF8007FF0001FC0012147D9318>I<001FC0007FE000FFE001F1E001C0C001C00001C00001C000 7FFFC0FFFFC0FFFFC001C00001C00001C00001C00001C00001C00001C00001C00001C00001C000 01C00001C00001C00001C0007FFF007FFF007FFF00131C7F9B18>I<03F1F007FFF80FFFF81E1F 303C0F003807003807003807003807003807003C0F001E1E001FFC003FF8003BF0003800003C00 001FFF001FFFC03FFFE07801F0F00078E00038E00038E00038F000787800F07E03F03FFFE00FFF 8003FE00151F7F9318>I<7E0000FE00007E00000E00000E00000E00000E00000E00000E3F000E FF800FFFC00FE1E00F80E00F00E00F00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E 00E00E00E00E00E07FC3FCFFE7FE7FC3FC171C809B18>I<03800007C00007C00007C000038000 0000000000000000000000007FC000FFC0007FC00001C00001C00001C00001C00001C00001C000 01C00001C00001C00001C00001C00001C00001C00001C000FFFF00FFFF80FFFF00111D7C9C18> I107 D<7FE000FFE0007FE00000E00000E00000E00000E00000E0 0000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E0 0000E00000E00000E00000E0007FFFC0FFFFE07FFFC0131C7E9B18>I<7DF1F000FFFBF8007FFF FC001F1F1C001E1E1C001E1E1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C 1C1C001C1C1C001C1C1C001C1C1C001C1C1C007F1F1F00FFBFBF807F1F1F001914819318>I<7E 3F00FEFF807FFFC00FE1E00F80E00F00E00F00E00E00E00E00E00E00E00E00E00E00E00E00E00E 00E00E00E00E00E00E00E07FC3FCFFE7FE7FC3FC1714809318>I<01F0000FFE001FFF003E0F80 3803807001C07001C0E000E0E000E0E000E0E000E0E000E0F001E07001C07803C03C07803E0F80 1FFF000FFE0001F00013147E9318>I<7E3E00FEFF807FFFC00FE3E00F80F00F00700F00780E00 380E00380E00380E00380E00380F00380F00780F00700F80F00FC3E00FFFC00EFF800E7E000E00 000E00000E00000E00000E00000E00000E00007FC000FFE0007FC000151E809318>I<03F3800F FB801FFF803E1F80780F80700780F00780E00380E00380E00380E00380E00380E00380F0078070 0780780F803E1F801FFF800FFB8003F38000038000038000038000038000038000038000038000 3FF8003FF8003FF8151E7E9318>I<7F87E0FF9FF87FBFF803FC7803F03003E00003C00003C000 03C0000380000380000380000380000380000380000380000380007FFE00FFFF007FFE0015147F 9318>I<0FF7003FFF007FFF00F81F00E00700E00700F007007C00007FF0001FFC0007FE00001F 00600780E00380F00380F00780FC0F00FFFF00FFFE00E7F80011147D9318>I<01800003800003 80000380000380007FFFC0FFFFC0FFFFC003800003800003800003800003800003800003800003 80000380000380400380E00380E00381E003C3C001FFC000FF80007E0013197F9818>I<7E07E0 FE0FE07E07E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E0 0E00E00E01E00F07E007FFFC03FFFE01FCFC1714809318>I<7F8FF0FF8FF87F8FF01E03C00E03 800E03800E0380070700070700070700038E00038E00038E00038E0001DC0001DC0001DC0000F8 0000F80000700015147F9318>II<7F8FF07F9FF07F8FF0070700078E00039E0001DC0001F80000F80000700000F00000F800 01DC00039E00038E000707000F07807F8FF0FF8FF87F8FF015147F9318>I<7F8FF0FF8FF87F8F F00E01C00E03800E0380070380070700070700038700038600038E0001CE0001CE0000CC0000CC 0000DC0000780000780000780000700000700000700000F00000E00079E0007BC0007F80003F00 001E0000151E7F9318>I<3FFFF07FFFF07FFFF07001E07003C0700780000F00001E00007C0000 F80001F00003E0000780000F00701E00703C0070780070FFFFF0FFFFF0FFFFF014147F9318>I E /Fi 2 63 df<0000038000000F0000003C000000F0000003C000000F0000003C000000F00000 03C000000F0000003C000000F0000000F00000003C0000000F00000003C0000000F00000003C00 00000F00000003C0000000F00000003C0000000F000000038019187D9520>60 D62 D E /Fj 43 122 df<0003C001F3C003F3C007F3C00700000E00000E00000E00000E00000E00000E00000E 0000FFF1C0FFF1C0FFF1C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E01C00E01C0121E809D16>12 D<7878787838307060E005097D830C >44 D<00030003000700060006000E000C001C0018001800380030003000700060006000E000C0 00C001C001800380030003000700060006000E000C000C001C0018001800380030007000600060 00E000C000C00010297E9E15>47 D<001C0000003E0000003E0000002E00000067000000670000 00E7800000C7800000C3800001C3C0000183C0000181C0000381E0000381E0000700F0000700F0 000600F0000E0078000FFFF8000FFFF8001FFFFC001C003C0018003C0038001E0038001E007000 1F0070000F0070000F00E0000780191D7F9C1C>65 DI<00 3FC000FFF003FFF007C0700F80001E00003E00003C0000780000780000780000F00000F00000F0 0000F00000F00000F00000F00000F00000F000007800007800007800003C00003E00001E00000F 800807C07803FFF800FFF0003F80151F7D9D1B>IIII<003F8001FFF003FFF8 07C0F80F00181E00003E00003C0000780000780000780000F00000F00000F00000F00000F00000 F00000F007F8F007F8F007F87800387800387800383C00383E00381E00380F003807C07803FFF8 01FFF0003F80151F7D9D1C>III75 DIII<003F000001FFE00003FFF00007C0F8000F807C001E001E003E00 1F003C000F00780007807800078078000780F00003C0F00003C0F00003C0F00003C0F00003C0F0 0003C0F00003C0F00003C0F80007C078000780780007807C000F803C000F003E001F001F003E00 0F807C0007C0F80003FFF00001FFE000003F00001A1F7E9D1F>II<003F000001FFE00003FFF00007C0F8000F807C001F003E003E001F003C000F00780007 807800078078000780F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F000 03C0F00003C07800078078000780781E07803C0F0F003E079F001E07DE000F83FC0007C1F80003 FFF00001FFF800003F7C0000003C0000003E0000001F0000000F801A237E9D1F>II<03F8000FFE001FFF003E0F00780300780000F00000F00000F00000F0 0000F800007C00007F00003FE0001FFC0007FE0001FF00001F800007800007C00003C00003C000 03C00003C00003C0C00780E00780FC1F007FFE001FFC0007F000121F7E9D17>II< F00070F00070F00070F00070F00070F00070F00070F00070F00070F00070F00070F00070F00070 F00070F00070F00070F00070F00070F00070F00070F00070F00070F000F07800E07801E03C03C0 1F07800FFF8007FE0001F800141E7C9C1D>III<78000E007C001E003C003C001E0038000F00 70000F00F0000781E00003C1C00001C3C00001E7800000F70000007E0000003E0000003C000000 3C0000007E00000077000000E7800001E3800003C1C0000381E0000700F0000F00F8000E007800 1C003C003C003E0078001F0070000F00F0000F80191D7F9C1C>I<0FC03FF07FF87038401C001C 001C00FC0FFC3FFC781CE01CE01CE01CF07C7FFC7FDC3F1C0E127E9114>97 D<07E00FF81FFC3C1C70047000E000E000E000E000E000E000700070043C1C1FFC0FF807E00E12 7E9112>99 D<000E000E000E000E000E000E000E000E000E000E000E0F8E1FEE3FFE7C3E700E70 0EE00EE00EE00EE00EE00EE00EF00E701E7C3E3FFE1FEE0F8E0F1D7E9C15>I<07C01FE03FF078 787018601CFFFCFFFCFFFCE000E000E000700070043C1C3FFC1FF807E00E127E9112>I<00FC01 FC03FC07000E000E000E000E000E000E000E00FFE0FFE0FFE00E000E000E000E000E000E000E00 0E000E000E000E000E000E000E000E000E1D809C0D>I105 D108 DII<03F0000FFC001FFE003C0F0078 0780700380E001C0E001C0E001C0E001C0E001C0F003C07003807807803C0F001FFE000FFC0003 F00012127F9115>II114 D<1FC03FF07FF0F030E000E000F0007F003FC01FE000F0003800388038F078FFF07FE01FC00D12 7F9110>I<1C001C001C001C001C001C00FFE0FFE0FFE01C001C001C001C001C001C001C001C00 1C001C001C001C201FF00FF007C00C187F970F>III121 D E /Fk 53 123 df<003F0F0000FFBF8003C3F3C00703E3C00703C1 800E01C0000E01C0000E01C0000E01C0000E01C0000E01C000FFFFFC00FFFFFC000E01C0000E01 C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E 01C0000E01C0000E01C0007F87FC007F87FC001A1D809C18>11 D<003F0000FF8003C1C00703C0 0703C00E01800E00000E00000E00000E00000E0000FFFFC0FFFFC00E01C00E01C00E01C00E01C0 0E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C07F87F87F87F8151D80 9C17>I<70F8FCFC7C0C0C0C1818306040060D7D9C0C>39 D<00C00180030006000E000C001C00 18003800300030007000700060006000E000E000E000E000E000E000E000E000E000E000E000E0 00600060007000700030003000380018001C000C000E0006000300018000C00A2A7D9E10>II<70F0F8F8781818183030706040050D7D840C>44 DI<70F8F8F87005057D840C>I<00030003000700060006000E000C 001C0018001800380030003000700060006000E000C000C001C001800380030003000700060006 000E000C000C001C001800180038003000700060006000E000C000C00010297E9E15>I<70F8F8 F870000000000000000070F8F8F87005127D910C>58 D<70F8F8F870000000000000000070F0F8 F8781818183030706040051A7D910C>I<0FE03FF8703C601CF01EF01EF01E001E003C007800E0 01C001800380030003000300030003000200000000000000000007000F800F800F8007000F1D7E 9C14>63 D<00060000000F0000000F0000000F0000001F8000001F8000001F8000001F80000033 C0000033C0000033C0000061E0000061E0000061E00000C0F00000C0F00000C0F0000180780001 80780001FFF80003FFFC0003003C0003003C0006001E0006001E0006001E001F001F00FFC0FFF0 FFC0FFF01C1D7F9C1F>65 D68 D70 D72 DI77 DI<003F800000FFE00003E0F80007803C000E000E00 1E000F003C00078038000380780003C0780003C0700001C0F00001E0F00001E0F00001E0F00001 E0F00001E0F00001E0F00001E0F00001E0780003C0780003C0780003C03C0007803C0007801E00 0F000F001E0007803C0003E0F80000FFE000003F80001B1E7E9C20>II<07E0801FF9803C1F80700780700380E00380E00180E00180E00180F00000F000007C0000 7FC0003FF8001FFE0007FF0000FF80000F800003C00003C00001C0C001C0C001C0C001C0E00180 E00380F00300FC0E00CFFC0083F800121E7E9C17>83 D<7FFFFFC07FFFFFC0780F03C0700F01C0 600F00C0E00F00E0C00F0060C00F0060C00F0060C00F0060000F0000000F0000000F0000000F00 00000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F 0000000F0000000F000003FFFC0003FFFC001B1C7F9B1E>IIII<06000F001F80 39C070E0E07040200C077C9C15>94 D<0FE0001FF8003C3C003C1E00180E00000E00001E0007FE 001FFE003E0E00780E00F00E00F00E60F00E60F01E60783E603FFFC01F878013127F9115>97 DI<03F00FF81E3C383C78187000F000F000F000F000F000 F000780078063C061E0C0FF803E00F127F9112>I<001F80001F80000380000380000380000380 00038000038000038000038000038003E3800FFB801E0F80380780780380700380F00380F00380 F00380F00380F00380F003807003807803803807801E1F800FFBF007E3F0141D7F9C17>I<03E0 0FF01C38381C781E700EFFFEFFFEF000F000F000F000700078063C061E0C0FF803E00F127F9112 >I<007801FC039E071E0E0C0E000E000E000E000E000E00FFE0FFE00E000E000E000E000E000E 000E000E000E000E000E000E000E000E007FE07FE00F1D809C0D>I<00038007E7C00FFDC03C3D C0381C00781E00781E00781E00781E00381C003C3C003FF00037E0007000007000003000003FFC 001FFF003FFF80700780E001C0E001C0E001C0E001C07003803C0F001FFE0007F800121C7F9215 >II<18003C007C003C0018000000000000000000000000 00FC00FC001C001C001C001C001C001C001C001C001C001C001C001C001C001C00FF80FF80091D 7F9C0C>I<01C003E003E003E001C00000000000000000000000000FE00FE000E000E000E000E0 00E000E000E000E000E000E000E000E000E000E000E000E000E000E000E060E0F1C0F1C07F803E 000B25839C0D>IIIII<03F0000FFC001E1E00380700780780700380F003C0F003C0F003 C0F003C0F003C0F003C07003807807803807001E1E000FFC0003F00012127F9115>II<03E1800FF9801E1F803C0780780780780380F00380F00380F00380F00380F00380F00380 7803807807803C07801E1F800FFB8007E380000380000380000380000380000380000380001FF0 001FF0141A7F9116>II<1F903FF07070E030E030E030F8007F803FE00FF000F8C038C0 38E038E038F070DFE08FC00D127F9110>I<0C000C000C000C000C001C001C003C00FFE0FFE01C 001C001C001C001C001C001C001C001C301C301C301C301C301E600FC007800C1A7F9910>IIII<7F8FF07F8FF00F0780070600038E0001DC0001D80000F00000700000780000F80001 DC00038E00030E000607000F0380FF8FF8FF8FF81512809116>II<7FFC7FFC78 38707060F060E061C063C00380070C0F0C0E0C1C1C3C1838187078FFF8FFF80E127F9112>I E /Fl 20 119 df<387CFEFEFE7C3807077D860D>46 D<387CFEFEFE7C3800000000387CFEFEFE 7C3807127D910D>58 D<001FE02000FFF8E003F80FE007E007E00FC001E01F8001E03F0000E03F 0000E07E0000607E000060FE000060FE000000FE000000FE000000FE000000FE000000FE000000 FE0000007E0000607E0000603F0000603F0000C01F8000C00FC0018007E0030003F80E0000FFFC 00001FE0001B1C7D9B22>67 DI73 D<0FF8001FFE003E1F803E07803E07C01C07C00007C003FFC01FFFC03F87C07E07C0FC07C0FC07 C0FC07C0FC0FC07E1FC03FFBF80FE1F815127F9117>97 D<03FC000FFE001F1F003E1F007C1F00 7C0E00FC0000FC0000FC0000FC0000FC0000FC00007C00007E01803E03801F07000FFE0003F800 11127E9115>99 D<01FC000FFF001F0F803E07C07C03C07C03E0FC03E0FFFFE0FFFFE0FC0000FC 0000FC00007C00007E00603E00C01F81C00FFF0001FC0013127F9116>101 D104 D<1E003F003F007F003F003F001E00000000000000 00000000FF00FF001F001F001F001F001F001F001F001F001F001F001F001F001F001F00FFE0FF E00B1E7F9D0E>I108 DII<01FC000FFF801F07C03E03E07C01F07C01F0FC01F8FC01F8FC01F8FC01F8FC01F8FC01F8 7C01F07C01F03E03E01F07C00FFF8001FC0015127F9118>II114 D<1FD83FF87038E018E018F000FF807FE07FF01FF807FC 007CC01CC01CE01CF038FFF0CFC00E127E9113>I<030003000300070007000F000F003F00FFFC FFFC1F001F001F001F001F001F001F001F001F001F0C1F0C1F0C1F0C0F9807F003E00E1A7F9913 >III E /Fm 55 123 df<007F07F001FF1FF003FF3FF007807800070070000F00F000 0F00F0000F00F0000F00F0000F00F0000F00F0000F00F000FFF8FF80FFF8FF80FFF8FF800F00F0 000F00F0000F00F0000F00F0000F00F0000F00F0000F00F0000F00F0000F00F0000F00F0000F00 F0000F00F0000F00F0000F00F0000F00F0000F00F0000F00F0001C20809F1B>11 D<0000F000F8F001F8F003F8F00780000700000F00000F00000F00000F00000F00000F00000F00 00FFF8F0FFF8F0FFF8F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00 F00F00F00F00F00F00F00F00F00F00F00F00F00F00F0142180A018>I<007000E001C003800780 07000E001E001E003C003C003C0078007800780078007000F000F000F000F000F000F000F000F0 00F000F000F000F000700078007800780078003C003C003C001E001E000E0007000780038001C0 00E000700C2E7EA112>40 DI<000600000006 000000060000000600000006000000060000000600000006000000060000000600000006000000 060000000600000006000000060000FFFFFFF0FFFFFFF000060000000600000006000000060000 000600000006000000060000000600000006000000060000000600000006000000060000000600 00000600001C207D9A23>43 D<787878781830306060E0050A7D830D>II<03F00007F8001FFE001E1E003C0F00380700780780780780700380F003C0F003C0F003 C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C07807807807 807807803807003C0F001E1E001FFE000FFC0003F00012207E9E17>48 D<00C001C00FC0FFC0FF C0F3C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C0 03C003C003C003C0FFFEFFFEFFFE0F1F7C9E17>I61 D<001F0000001F0000003F8000003B8000003B8000007BC0000073C0000071C00000F1E00000E1 E00000E0E00001E0F00001E0F00001C0F00003C0780003C078000380780007803C0007803C0007 003C000FFFFE000FFFFE000FFFFE001E000F001E000F003C000F803C0007803C000780780007C0 780003C0780003C0F00003E01B207F9F1E>65 DI<001FC000FFF801FFFC03E03C07800C0F00001E00003E00003C00007C0000780000 780000780000F00000F00000F00000F00000F00000F00000F00000F00000780000780000780000 7C00003C00003E00001E00000F000207800E03E03E01FFFC00FFF0001FC017227DA01D>IIII<001FE000FF F801FFFE03E03E07800E0F00001E00003E00003C00007C0000780000780000780000F00000F000 00F00000F00000F00000F00000F003FEF003FE7803FE78001E78001E7C001E3C001E3E001E1E00 1E0F001E07801E03E03E01FFFE00FFF8001FC017227DA01E>III75 DIII<003F000000FFC00003 FFF00007E1F8000F807C001F003E001E001E003C000F003C000F00780007807800078078000780 F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F80007 C07800078078000780780007803C000F003C000F001E001E001F003E000F807C0007E1F80003FF F00000FFC000003F00001A227DA021>II82 D<01FC0007FF800FFFC01F 03C03C00C03C00007800007800007800007800007800007C00003C00003F00001FE0000FFC0007 FE0001FF00003F800007C00003C00003E00001E00001E00001E00001E00001E00001C0C003C0F0 07C0FC0F807FFF001FFE0003F80013227EA019>II< F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0 F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F07801E0 7801E03C03C03C03C01F0F800FFF0007FE0001F80014217B9F1F>III89 D<07E03FF87FFC701E401F000F000F000F003F07FF1FFF7E0FF80FF00FF00FF00FF83F7FFF3FEF 1F8F10147E9316>97 DI<03F00FFC 1FFE3E0E3C0278007800F000F000F000F000F000F000780078003C013E0F1FFF0FFE03F010147E 9314>I<0007800007800007800007800007800007800007800007800007800007800007800007 8007C7800FF7801FFF803E1F807C0780780780F80780F00780F00780F00780F00780F00780F007 80F00780780780780F803E1F801FFF800FF78007C78011207E9F17>I<03F0000FFC001FFE003E 1F003C0700780700700380FFFF80FFFF80FFFF80F00000F00000F000007000007800003C01003E 07001FFF0007FE0001F80011147F9314>I<007E01FE03FE078007000F000F000F000F000F000F 000F00FFF0FFF0FFF00F000F000F000F000F000F000F000F000F000F000F000F000F000F000F00 0F000F000F20809F0E>I104 DI108 DII<01F80007FE001FFF803F0FC03C03 C07801E07801E0F000F0F000F0F000F0F000F0F000F0F000F07801E07801E03C03C03F0FC01FFF 8007FE0001F80014147F9317>II<03C7800FF7801FFF80 3E1F807C0F80780780780780F00780F00780F00780F00780F00780F00780F807807807807C0F80 3E1F801FFF800FF78007C780000780000780000780000780000780000780000780000780000780 111D7E9317>II<07F01FFC3FFC780C7800780078007C003FC01FF00FF803F8 007C003C003CC03CF07CFFF87FF00FC00E147F9311>I<1E001E001E001E001E001E00FFF0FFF0 FFF01E001E001E001E001E001E001E001E001E001E001E001E001E001E201FF00FF007C00C1A7F 9910>II II<7801E07C03C03E07801E0F000F0F00079E0003FC0003F80001F80000F0 0001F00001F80003FC00079E000F0F000E0F001E07803C03C07801E0F801F01414809315>II<7FFF7FFF7FFF003E003C007800F800F001E003E007C00780 0F001F001E003C007C00FFFFFFFFFFFF10147F9314>I E /Fn 35 121 df<000FE000007FF800 00F81C0001E07C0003E07C0007C07C0007C07C0007C0380007C0000007C0000007C0000007C1FE 00FFFFFE00FFFFFE0007C03E0007C03E0007C03E0007C03E0007C03E0007C03E0007C03E0007C0 3E0007C03E0007C03E0007C03E0007C03E0007C03E0007C03E0007C03E0007C03E003FF9FFC03F F9FFC01A20809F1D>12 D<0018007000E001C00380038007000E000E001E001C003C003C007800 780078007800F800F000F000F000F000F000F000F000F000F000F80078007800780078003C003C 001C001E000E000E0007000380038001C000E0007000180D2D7DA114>40 DI45 D<00700000F0000FF000FFF000F3F00003F00003F00003F00003F00003F00003F00003F00003F0 0003F00003F00003F00003F00003F00003F00003F00003F00003F00003F00003F00003F00003F0 0003F000FFFF80FFFF80111D7C9C1A>49 D<07F0001FFE00383F007C1F80FE0FC0FE0FC0FE0FE0 FE07E07C07E03807E0000FE0000FC0000FC0001F80001F00003E0000780000F00000E00001C060 0380600700600C00E01FFFE01FFFC03FFFC07FFFC0FFFFC0FFFFC0131D7D9C1A>I<3803803FFF 803FFF003FFE003FFC003FF0003F800030000030000030000030000033F8003FFE003C1F00380F 80300FC0000FC0000FE0000FE0780FE0FC0FE0FE0FE0FE0FE0FC0FC0780FC0601F80383F001FFC 0007F000131D7D9C1A>53 D<387CFEFEFE7C38000000000000387CFEFEFE7C3807147C930F>58 D<0007FC02003FFF0E00FE03DE03F000FE07E0003E0FC0001E1F80001E3F00000E3F00000E7F00 00067E0000067E000006FE000000FE000000FE000000FE000000FE000000FE000000FE0000007E 0000007E0000067F0000063F0000063F00000C1F80000C0FC0001807E0003803F0007000FE01C0 003FFF800007FC001F1F7D9E26>67 D72 D77 D80 D<03FC080FFF381E03F83C00F8780078780038F80038F80018FC0018FC0000FF0000FFF8007FFF 007FFFC03FFFE01FFFF00FFFF803FFF8001FFC0001FC0000FC0000FCC0007CC0007CC0007CE000 78E00078F800F0FE01E0E7FFC081FF00161F7D9E1D>83 D85 DI<07FC001FFF003F0F803F07C03F03E03F03E00C03E00003E001FFE00FFFE03F83 E07E03E07C03E0F803E0F803E0F803E0FC07E07E0DE03FF9FE07E07E17147F9319>97 DI<01FE0007FF801F0FC03E0FC03E 0FC07C0FC07C0300FC0000FC0000FC0000FC0000FC0000FC00007C00007E00003E00603F00C01F 81C007FF0001FC0013147E9317>I<0007F80007F80000F80000F80000F80000F80000F80000F8 0000F80000F80000F80000F801F8F80FFEF81F83F83E01F87E00F87C00F87C00F8FC00F8FC00F8 FC00F8FC00F8FC00F8FC00F87C00F87C00F87E00F83E01F81F07F80FFEFF03F8FF18207E9F1D> I<01FE0007FF801F83E03F01F07E00F07E00F8FC00F8FC00F8FFFFF8FFFFF8FC0000FC0000FC00 007C00007E00003E00183F00380F807007FFE000FF8015147F9318>I<001F8000FFC001F3E003 E7E003C7E007C7E007C3C007C00007C00007C00007C00007C000FFFC00FFFC0007C00007C00007 C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007 C0003FFC003FFC0013207F9F10>I<01FC3C07FFFE0F07DE1E03DE3E03E03E03E03E03E03E03E0 3E03E01E03C00F07800FFF0019FC001800001800001C00001FFF801FFFF00FFFF83FFFFC7C007C 70003EF0001EF0001EF0001E78003C78003C3F01F80FFFE001FF00171E7F931A>II<1C003F007F007F007F003F001C000000000000 00000000000000FF00FF001F001F001F001F001F001F001F001F001F001F001F001F001F001F00 1F001F00FFE0FFE00B217EA00E>I108 DI< FE0FC0FE3FE01E61F01EC0F81F80F81F00F81F00F81F00F81F00F81F00F81F00F81F00F81F00F8 1F00F81F00F81F00F81F00F81F00F8FFE3FFFFE3FF18147D931D>I<01FF0007FFC01F83F03E00 F83E00F87C007C7C007CFC007EFC007EFC007EFC007EFC007EFC007E7C007C7C007C3E00F83E00 F81F83F007FFC001FF0017147F931A>II114 D<0FE63FFE701E600EE006E006F800FF C07FF83FFC1FFE03FE001FC007C007E007F006F81EFFFCC7F010147E9315>I<01800180018003 800380038007800F803F80FFFCFFFC0F800F800F800F800F800F800F800F800F800F800F860F86 0F860F860F8607CC03F801F00F1D7F9C14>III120 D E /Fo 30 120 df<000078007C7800FC7801FC7803C40003800007800007800007800007800007800007 8000078000078000FFFC78FFFC78FFFC7807807807807807807807807807807807807807807807 8078078078078078078078078078078078078078078078078078078078078078078078152480A3 1A>12 D46 D<00C001C007C0FFC0FFC0FBC003C003C003C003C003C003 C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C0 FFFFFFFFFFFF10227CA118>49 D<03F0000FFC001FFE003C1F003807807007C07003C0F003E0E0 01E06001E02001E00001E00001E00001E00003C00003C0000780000780000F00001E00003C0000 780000F00001E00001C0000380000700000E00001C0000380000700000FFFFE0FFFFE0FFFFE013 227EA118>I<01F00007FC001FFF003E0F003807807003C02003C02003C00003C00003C00003C0 000780000780000F00001E0003FC0003F80003FE00000F000007800003C00003C00001E00001E0 0001E00001E00001E08001E0C003C0E003C07007803C0F801FFF000FFC0003F00013237EA118> I<3FFF803FFF803FFF803C00003C00003C00003C00003C00003C00003C00003C00003C00003CF8 003FFE003FFF003F0F803E07803C03C03803C00001E00001E00001E00001E00001E00001E00001 E04003C04003C0E003C07007807C1F003FFE000FFC0003F00013227EA018>53 D<007E0001FF0003FF0007C1000F00001E00001E00003C00003C000078000078000078F800F3FE 00F7FF00FF0F80FC0780F803C0F803C0F801C0F001E0F001E0F001E0F001E0F001E07001E07001 E07801E07803C03803C03C03801C07801F0F000FFE0007FC0001F00013237EA118>I<01F00007 FC000FFE001E0F003C07803C07807803C07803C07803C07803C07803C03803803C07801E0F000F 1E0007FC0003F8000FFE001E0F003C07807803C07803C0F001E0F001E0F001E0F001E0F001E0F0 01E07803C07803C03C07803E0F801FFF0007FC0001F00013237EA118>56 D<001F0000001F0000003F8000003F8000003B8000007BC0000073C0000071C00000F1E00000F1 E00000E0E00001E0F00001E0F00001C0F00003C0780003C078000380780007803C0007803C0007 003C000F001E000F001E000FFFFE001FFFFF001FFFFF001C000F003C0007803C00078038000780 780003C0780003C0700003C0F00001E0F00001E0E00001E01B237EA220>65 D68 D70 D86 D<07E01FF83FFC3C1E301E200F000F000F000F01FF0FFF3FFF7F0F780FF00FF00FF00FF80F7C3F 7FFF3FFF1F8F10167E9517>97 DI<01FC0007FF000FFF801F07803C0180780000780000700000F00000F00000 F00000F00000F00000F000007800007800007800403C00C01F07C00FFFC007FF8001FC0012167E 9516>I<0003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003 C00003C003F3C00FFFC01FFFC03F0FC03C07C07803C07803C0F003C0F003C0F003C0F003C0F003 C0F003C0F003C0F003C07803C07803C03C07C03E0FC01FFFC00FFBC003E3C012237EA219>I<03 F00007FC001FFE003E0F003C0780780380780380F001C0FFFFC0FFFFC0FFFFC0F00000F00000F0 00007000007800007800003C00801F07800FFF8007FF0001F80012167E9516>I<01F0F807FFF8 0FFFF81F1F801E0F003C07803C07803C07803C07803C07801E0F001F1F001FFE001FFC0039F000 3800003800003C00003FFE001FFFC01FFFE03FFFF07801F0F800F8F00078F00078F00078F00078 7800F03E03E01FFFC00FFF8001FC0015217F9518>103 D105 D107 DIII<01FC0007FF000FFF801F07C03C01E07800F07800 F0700070F00078F00078F00078F00078F00078F000787800F07800F07C01F03E03E01F07C00FFF 8007FF0001FC0015167F9518>II< F0E0F3E0F7E0FFE0FF00FC00F800F800F000F000F000F000F000F000F000F000F000F000F000F0 00F000F0000B167C9511>114 D<07F01FFE3FFF3C0F7803780078007C003F003FF01FF80FFC01 FE001F000F000F400FE00FF81EFFFE3FFC0FF010167F9513>I<0F000F000F000F000F000F00FF F8FFF8FFF80F000F000F000F000F000F000F000F000F000F000F000F000F000F000F080F1C07FC 07F803E00E1C7F9B12>II 119 D E /Fp 26 87 df<3078F8787005057C840D>46 D<00000300000700000600000E00000C 00001C0000180000380000300000700000600000E00000C00001C0000180000380000300000700 000600000E00000C00001C0000180000380000300000700000600000E00000C00001C000018000 0380000300000700000600000E00000C00001C0000180000380000300000700000600000E00000 C00000182D7FA117>I<000C001C00FC0FF80F3800380038003800380070007000700070007000 7000E000E000E000E000E000E001C001C001C001C001C001C003C07FFEFFFE0F1E7C9D17>49 D<007F0001FFC00383E00701E00700F00F00F00F01F00F01F00001E00001E00003C0000780000F 0000FE0000F800000E000007000007800003C00003C00003C03007C07807C0F807C0F807C0F00F 80C00F00601E00383C001FF80007E000141F7D9D17>51 D<0001C00003C00003C0000780000F80 001B80003B8000738000638000C7000187000307000707000E07000C0700180E00300E00600E00 E00E00FFFFF0FFFFF0001C00001C00001C00001C00001C00001C00003C0003FFC003FFC0141E7D 9D17>I<03007003FFF003FFE003FF8003FE000600000600000600000600000600000600000C7C 000DFF000F87800E03800C01C00C01C00001E00001E00001E00001E07003C0F003C0F003C0F003 80C00780E00700600E00383C001FF0000FC000141F7D9D17>I<000F80007FC000F0E001C0E003 81E00701E00E00C01E00001C00003C000038000079FC007BFE007E07007C0380F80380F803C0F0 03C0F003C0F003C0F003C0F00780E00780E00780E00700E00F00700E00701C003838001FF00007 C000131F7C9D17>I<003F0000FF8001C1E00380E00700700E00700E00700E00700E00E00F00E0 0F81C00FC38007F70003FC0001FC0003FF000F3F801C0F803807C03803C07001C07001C0E001C0 E001C0E001C0E003807003007007003C1C001FF80007E000141F7D9D17>56 D<0000180000003800000038000000780000007C000000FC000000FC000001BC000001BC000003 3C0000033E0000061E0000061E00000C1E00000C1E0000181E0000181F0000300F0000300F0000 600F00007FFF0000FFFF0000C00F00018007800180078003000780030007800600078006000780 1F0007C0FFC07FFCFFC07FFC1E207E9F22>65 D<0003F808000FFE18003F0738007801F801F000 F803C000F007C00070078000700F0000701F0000701E0000703E0000603C0000607C0000007C00 00007C0000007C000000F8000000F8000000F8000000F8000000F80000C0780000C0780001807C 0001803C0003003C0003001E0006001E000C000F80180007E0700001FFC000007F00001D217B9F 21>67 D<07FFFF0007FFFFE0003C01F0003C00F80078007C0078003C0078001E0078001E007800 1E0078001F00F0001F00F0001F00F0001F00F0001F00F0001F00F0001F01E0001E01E0003E01E0 003E01E0003E01E0003C01E0007C03C0007803C000F003C000F003C001E003C003C003C00F8007 C03F007FFFFC00FFFFE000201F7E9E23>I<07FFFFF807FFFFF8003C00F8003C00780078003800 78003800780038007800380078003800780C3000F0183000F0180000F0180000F0380000FFF800 00FFF80001E0700001E0300001E0300001E0301801E0303001E0003003C0003003C0006003C000 6003C000E003C001C003C003C007C00FC07FFFFF80FFFFFF801D1F7E9E1F>I<07FFFFF807FFFF F8003C00F8003C0078007800380078003800780038007800380078003800780C3000F0183000F0 180000F0180000F0380000FFF80000FFF80001E0700001E0300001E0300001E0300001E0300001 E0000003C0000003C0000003C0000003C0000003C0000003C0000007C000007FFE0000FFFE0000 1D1F7E9E1E>I<0001FC04000FFF0C003F039C007800FC00F000FC03E0007803C0007807800038 0F0000381F0000381E0000383E0000303E0000307C0000007C0000007C0000007C000000F80000 00F8000000F8007FFCF8007FFCF80001E0780001E0780003C07C0003C03C0003C03C0003C01E00 03C00F0007C007800F8003E0398001FFF080003F80001E217B9F24>I<07FFC7FFC007FFC7FFC0 003C007800003C007800007800F000007800F000007800F000007800F000007800F000007800F0 0000F001E00000F001E00000F001E00000F001E00000FFFFE00000FFFFE00001E003C00001E003 C00001E003C00001E003C00001E003C00001E003C00003C007800003C007800003C007800003C0 07800003C007800003C007800007C00F80007FFCFFF800FFF8FFF800221F7E9E22>I<07FFE007 FFE0003C00003C0000780000780000780000780000780000780000F00000F00000F00000F00000 F00000F00001E00001E00001E00001E00001E00001E00003C00003C00003C00003C00003C00003 C00007C000FFFC00FFFC00131F7F9E10>I<07FFF00007FFF000003C0000003C00000078000000 7800000078000000780000007800000078000000F0000000F0000000F0000000F0000000F00000 00F0000001E0000001E0000001E0000001E0018001E0018001E0030003C0030003C0030003C007 0003C0060003C00E0003C01E0007C07E007FFFFC00FFFFFC00191F7E9E1C>76 D<07FC0000FFC007FC0001FFC0003E0001F800003E00037800006E0003F000006E0006F000006E 0006F000006E000CF0000067000CF00000670018F00000C70019E00000C70031E00000C70031E0 0000C70061E00000C38061E00000C380C1E000018380C3C00001838183C00001838183C0000181 C303C0000181C303C0000181C603C0000301C60780000301CC0780000301CC0780000300F80780 000300F80780000700F00780000F80F00F80007FF0E0FFF800FFF0E1FFF8002A1F7E9E2A>I<07 FC03FFC007FC03FFC0003E007C00003E003800006F003000006F003000006F0030000067803000 00678030000063C0300000C3C0600000C3C0600000C1E0600000C1E0600000C0F0600000C0F060 000180F0C000018078C000018078C00001803CC00001803CC00001803CC00003001F800003001F 800003000F800003000F800003000F800007000780000F800700007FF0030000FFF0030000221F 7E9E22>I<0003F800001FFE00003C1F0000F0078001E003C003C001E0078001E00F8000F00F00 00F01F0000F01E0000F83E0000F83C0000F87C0000F87C0000F87C0000F87C0000F8F80001F0F8 0001F0F80001F0F80001F0F80003E0780003E0780007C07C0007C07C000F803C000F003E001E00 1E003C000F00780007C1F00003FFC00000FE00001D217B9F23>I<07FFFF0007FFFFC0003C03E0 003C01F0007800F0007800F8007800F8007800F8007800F8007800F800F001F000F001F000F001 E000F003C000F00F8000FFFE0001FFF80001E0000001E0000001E0000001E0000001E0000003C0 000003C0000003C0000003C0000003C0000003C0000007C000007FFC0000FFFC00001D1F7E9E1F >I<07FFFC0007FFFF00003C07C0003C03E0007801E0007801F0007801F0007801F0007801F000 7801E000F003E000F003C000F0078000F01F0000FFFC0000FFF00001E0380001E03C0001E01C00 01E01C0001E01E0001E01E0003C03E0003C03E0003C03E0003C03E0003C03E0603C03E0607C03E 0C7FFC1F18FFFC0FF8000003E01F207E9E21>82 D<003F04007FCC01E0FC03807C03003C070038 0600180E00180E00180E00180E00000F00000F00000FE00007FE0003FF8001FFC0007FE00007E0 0001E00000E00000F00000F06000E06000E06000E06000E07001C0700180F80380FE0F00C7FC00 81F80016217D9F19>I<1FFFFFF81FFFFFF81E03C0F83803C03838078038300780387007801860 0780186007803860078030C00F0030000F0000000F0000000F0000000F0000000F0000001E0000 001E0000001E0000001E0000001E0000001E0000003C0000003C0000003C0000003C0000003C00 00003C0000007C00001FFFF0003FFFF0001D1F7B9E21>III E /Fq 15 116 df<60F0F0703030306060C0 40040B7D830B>44 D<03000700FF00FF0007000700070007000700070007000700070007000700 07000700070007000700070007000700FFF0FFF00C197D9813>49 D<0F801FE038F07070787878 38787830780070007000E00FC00F8000E000700038003C003C703CF83CF83CF038607870F03FE0 0F800E1A7E9813>51 D<0070007000F000F001F003F00370067006700C701C7018703070307060 70E070FFFFFFFF0070007000700070007007FF07FF10197F9813>I<07801FE0387030306038E0 18E018E01CE01CE01CE01CE01C603C703C307C3FDC1F9C00180018003830307870786071C03F80 1F000E1A7E9813>57 D<0FFF0FFF00780078007800780078007800780078007800780078007800 7800780078007800787078F878F878F878F0F070E03FC00F80101B7F9914>74 D86 D<3F80FFC0F0E0F070607000700FF03FF07870E070E070E073E07370F37FFE3E3C10107E8F13> 97 D<07C01FF038387018701CFFFCFFFCE000E000E000E0007000300C3C180FF007E00E107F8F 11>101 D<00F801FC03BC073C0E180E000E000E000E000E00FFC0FFC00E000E000E000E000E00 0E000E000E000E000E000E000E007FE07FE00E1A80990C>I<18003C003C001800000000000000 000000000000FC00FC001C001C001C001C001C001C001C001C001C001C001C001C00FF80FF8009 1A80990A>105 D110 D<07E01FF8381C700E6006E007E0 07E007E007E007E007700E700E3C3C1FF807E010107F8F13>I114 D<1F207FE070E0E060E060F0 007F003FC01FE001F0C070C070E070F0E0FFC08F800C107F8F0F>I E /Fr 10 58 df<1F003F8060C04040C060C060C060C060C060C060C060C06060C060C03F801F000B10 7F8F0F>48 D<18007800F80098001800180018001800180018001800180018001800FF80FF8009 107E8F0F>I<3F007F80F1C0F0E06060006000E000C00180030006001C0038606060FFC0FFC00B 107F8F0F>I<1F003F8071C071C031C001800F800F0001C000E060E0F0E0F0E0F1C07F801F000B 107F8F0F>I<070007000F001F001B003B0033006300E300FFE0FFE00300030003001FE01FE00B 107F8F0F>I<60807F807F007C00600060006F007F8070C060E000E060E0E0E0E1C07F803F000B 107F8F0F>I<07801FC039C061C06000C000DF80FFC0E060C060C060C060606060C03F801F000B 107F8F0F>I<60007FE07FE0C0C0C1800180030006000E000C000C001C001C001C001C001C0008 000B117E900F>I<1F003F8061C060C060C079C03F801F803FC063E0C0E0C060C06060C03F801F 000B107F8F0F>I<1F003F8060C0C0C0C060C060C060C0E07FE03F60006000C070C071807F003E 000B107F8F0F>I E /Fs 49 122 df<1C3E7E7E3E060C0C18183060C080070E769F0E>39 D<7FF0FFE0FFE00C037D8A10>45 D<70F8F8F0E005057B840E>I<0000006000000060000000E0 000000C0000001C00000038000000300000007000000060000000E0000001C0000001800000038 000000300000007000000060000000E0000001C000000180000003800000030000000700000006 0000000E0000001C0000001800000038000000300000007000000060000000E0000001C0000001 800000038000000300000007000000060000000E0000001C000000180000003800000030000000 70000000E0000000C00000001B2D80A117>I<000F80003FE000F0F001C0700380700380380700 780F00780F00780E00781E00781E00703C00F03C00F03C00F03C00F07801E07801E07801E07801 C07003C0F003C0F00380F00780F00700700F00700E00701C003878001FF0000FC000151F7C9D17 >I<000200060006000E003C00FC07DC071C0038003800380038007000700070007000E000E000 E000E001C001C001C001C00380038003800380FFF8FFF80F1E7B9D17>I<000F80003FC00070E0 00C0700180780330380638380618780C30780C30780C60780CE0780FC0F00701E00001C0000380 000700001E0000380000E00001C0000700000E00300C00301800603000E07F81C07FFFC061FF80 C07F00C03C00151F7D9D17>I<001F80007FE000E0E00180700300380660380670380630380660 7007C0700380600000E00001C000078000FE0000FE000007000007000007800007800007803007 80780780780780F00F00C00F00601E00603C007078003FE0001F8000151F7C9D17>I<001F0000 7F8000E1C001C1C00380E00700E00F00E00F01E01E01E01E01E01E01E01E01C01C03C03C03C03C 07C01C07C01C0F801C1F800F778007E700000F00000E00001E00001C00603C00F03800F07000E0 E000C1C0007F80003E0000131F7B9D17>57 D<070F1F1F0E0000000000000000000070F8F8F0E0 08147B930E>I<000007000000070000000F0000000F0000001F0000003F0000003F0000006F00 00006F000000CF000000CF0000018F0000038F0000030F0000060F0000060F00000C0F80000C07 800018078000180780003FFF80007FFF800060078000C0078000C0078001800780018007800300 0780070007800F0007807FC07FF8FFC07FF81D207E9F22>65 D<0000FC020007FF06001F818C00 3C00DC0070007C01E0007C03C0003803800038070000380F0000381E0000301E0000303C000030 3C00000078000000780000007800000078000000F0000000F0000000F0000000F00000C0F00001 807000018070000300780003007800060038000C001C0018000E0030000780E00003FF800000FE 00001F217A9F21>67 D<01FFFFFE01FFFFFC001E003C001E001C001E001C003C000C003C000C00 3C000C003C001800780C1800780C1800780C0000781C0000F0380000FFF80000FFF80000F03800 01E0300001E0300001E0303001E0306003C0006003C0006003C000C003C001C007800180078003 800780070007801F00FFFFFF00FFFFFE001F1F7D9E1F>69 D<01FFFFFC01FFFFF8001E0078001E 0038001E0038003C0018003C0018003C0018003C003000780C3000780C3000780C0000781C0000 F0380000FFF80000FFF80000F0380001E0300001E0300001E0300001E0300003C0000003C00000 03C0000003C0000007800000078000000780000007800000FFFC0000FFF800001E1F7D9E1E>I< 0001FC040007FE0C001F0398003C01B800F000F801E000F803C0007007800070070000700F0000 701E0000601E0000603C0000603C00000078000000780000007800000078000000F0000000F000 FFF0F000FFF0F0000780F0000F00F0000F0070000F0078000F0078001E0038001E003C003E001E 007E000F81EC0003FF840000FE00001E217A9F23>I<01FFF3FFE001FFF3FFE0001E003C00001E 003C00001E003C00003C007800003C007800003C007800003C007800007800F000007800F00000 7800F000007800F00000F001E00000FFFFE00000FFFFE00000F001E00001E003C00001E003C000 01E003C00001E003C00003C007800003C007800003C007800003C007800007800F000007800F00 0007800F000007800F0000FFF9FFF000FFF1FFE000231F7D9E22>I<01FFF001FFF0001E00001E 00001E00003C00003C00003C00003C0000780000780000780000780000F00000F00000F00000F0 0001E00001E00001E00001E00003C00003C00003C00003C000078000078000078000078000FFF8 00FFF800141F7D9E12>I<01FF00FFE001FF00FFE0001F001E00001F800C00001F800C00003780 18000033C018000033C018000033C018000063C030000061E030000061E030000061E0300000C0 F0600000C0F0600000C0F0600000C0786000018078C000018078C00001803CC00001803CC00003 003D800003001F800003001F800003001F800006000F000006000F000006000F00000E000F0000 FFE0060000FFC0060000231F7D9E22>78 D<0001FC000007FF00001F07C0003801E000F000E001 E0007003C0007007800078070000380F0000381E0000381E0000383C0000383C00003878000078 780000787800007878000078F00000F0F00000F0F00000E0F00001E0F00001C0F00003C0700007 807800070078000F0038001E003C003C001E00F0000F83E00007FF800001FC00001D217A9F23> I<01FFFF8001FFFFE0001E00F0001E0078001E0038003C003C003C003C003C003C003C003C0078 00780078007800780070007800E000F001E000F0078000FFFF0000FFF80001E0000001E0000001 E0000001E0000003C0000003C0000003C0000003C0000007800000078000000780000007800000 FFF80000FFF000001E1F7D9E1F>I<01FFFF0001FFFFC0001E01E0001E00F0001E0078003C0078 003C0078003C0078003C0078007800F0007800F0007801E0007803C000F00F0000FFFE0000FFF8 0000F03C0001E01E0001E00E0001E00F0001E00F0003C01E0003C01E0003C01E0003C01E000780 3C0007803C1807803C1807803C30FFF81E30FFF00FE0000007C01D207D9E21>82 D<0007E040001FF8C0003C1D8000700F8000E0078001C007800180030003800300038003000380 0300038000000380000003C0000003F8000001FF800001FFE000007FF000001FF0000001F80000 00780000003800000038000000380030003800300038003000300070007000700060007800E000 7801C000EE078000C7FE000081F800001A217D9F1A>I<0FFFFFF01FFFFFE01E0781E0180780E0 38078060300F0060300F0060600F0060600F00C0C01E00C0C01E00C0001E0000001E0000003C00 00003C0000003C0000003C00000078000000780000007800000078000000F0000000F0000000F0 000000F0000001E0000001E0000001E0000003E00000FFFF0000FFFF00001C1F789E21>I<7FFC 3FF87FFC3FF80780078007800300078003000F0006000F0006000F0006000F0006001E000C001E 000C001E000C001E000C003C0018003C0018003C0018003C001800780030007800300078003000 78003000F0006000F0006000F0006000F000C000F00080007001800070030000380600003C1C00 001FF8000007E000001D20779E22>III<00 F18003FDC0078F800E07801C07803C07803C0700780700780700780700F00E00F00E00F00E00F0 0E30F01C60F03C60707C6078FCC03FCFC00F078014147C9317>97 D<07803F803F000700070007 000E000E000E000E001C001C001CF01FFC3F1E3E0E3C0F380F700F700F700F700FE01EE01EE01E E03CE03CE038607071E03FC01F0010207B9F15>I<007E0001FF000383800F07801E07801C0700 3C0200780000780000780000F00000F00000F00000F00000F00000700200700700381E001FF800 07E00011147C9315>I<0000780003F80003F00000700000700000700000E00000E00000E00000 E00001C00001C000F1C003FDC0078F800E07801C07803C07803C0700780700780700780700F00E 00F00E00F00E00F00E30F01C60F03C60707C6078FCC03FCFC00F078015207C9F17>I<007C0001 FF000783000F01801E01803C01803C0300780E007FFC007FE000F00000F00000F00000F0000070 00007002007807003C1E001FF80007E00011147C9315>I<0000F80001FC0003BC00033C000718 000700000700000E00000E00000E00000E00000E0001FFE001FFE0001C00001C00001C00003800 00380000380000380000380000700000700000700000700000700000700000E00000E00000E000 00E00001C00001C00001C0000180003380007B8000F300007E00003C00001629829F0E>I<003C 6000FF7001E3E00381E00701E00F01E00F01C01E01C01E01C01E01C03C03803C03803C03803C03 803C07003C0F001C1F001E3F000FFE0003CE00000E00000E00001C00001C00301C00783800F0F0 007FE0003F8000141D7E9315>I<01E0000FE0000FC00001C00001C00001C00003800003800003 8000038000070000070000073E00077F000EC3800F81C00F01C00E01C01E03801C03801C03801C 0380380700380700380700380E18700E30700E30701C60700C60E00FC060078015207D9F17>I< 006000F000F000E000000000000000000000000000000F001F80318031C063806380C380070007 0007000E000E000E001C301C601C6038C018C01F800F000C1F7D9E0E>I<01E0000FE0000FC000 01C00001C00001C0000380000380000380000380000700000700000703C00707E00E0C600E10E0 0E21E00E61E01CC1C01F80001F00001FC00039E0003870003870003838607070C07070C07070C0 703180E03F00601E0013207D9F15>107 D<03C01FC01F8003800380038007000700070007000E 000E000E000E001C001C001C001C0038003800380038007000700070007180E300E300E300E600 7E003C000A207C9F0C>I<1E07C0F8003F1FE1FC0033B8730E0063E076070063C03C0700638038 0700C780780E000700700E000700700E000700700E000E00E01C000E00E01C000E00E01C000E00 E038601C01C038C01C01C038C01C01C071801C01C031803803803F001801801E0023147D9325> I<1E07C03F1FE033B87063E07063C038638038C780700700700700700700700E00E00E00E00E00 E00E01C31C01C61C01C61C038C1C018C3801F81800F018147D931A>I<007C0001FF000383800F 01C01E01C01C01E03C01E07801E07801E07801E0F003C0F003C0F003C0F00780F00700700F0070 1E003838001FF00007C00013147C9317>I<03C1E007E7F8067E3C0C7C1C0C781E0C701E18E01E 00E01E00E01E00E01E01C03C01C03C01C03C01C07803C07803C07003C0E003E3C0077F80071E00 0700000700000E00000E00000E00000E00001C0000FFC000FFC000171D809317>I<1E0F003F3F 8033F1C063C1C063C3C06383C0C783800700000700000700000E00000E00000E00000E00001C00 001C00001C00001C000038000018000012147D9313>114 D<00FC03FE07070E0F0E0F0E0E1E00 0F800FF007F803FC003E001E701EF01CF01CE03860703FE01F8010147D9313>I<018001C00380 03800380038007000700FFF0FFF00E000E000E000E001C001C001C001C00380038003800383070 60706070C071803F001E000C1C7C9B0F>I<0F00601F80703180E031C0E06380E06380E0C381C0 0701C00701C00701C00E03800E03800E03800E038C0E07180E07180E0F180E1F3007F3F003E1E0 16147D9318>I<0F01C01F83C03183E031C1E06380E06380E0C380C00700C00700C00700C00E01 800E01800E01800E03000E03000E06000E06000F0C0007F80001E00013147D9315>I<0F0060E0 1F8071E03180E1F031C0E0F06380E0706380E070C381C0600701C0600701C0600701C0600E0380 C00E0380C00E0380C00E0381800E0381800E0781000E078300070F860007F9FC0001F0F8001C14 7D931E>I<0787800FCFC018F8E03070E06071E06071E0C0E1C000E00000E00000E00001C00001 C00001C00071C060F380C0F380C0E38180C7C3007CFE00387C0013147D9315>I<0F00601F8070 3180E031C0E06380E06380E0C381C00701C00701C00701C00E03800E03800E03800E03800E0700 0E07000E0F000E1F0007FE0003EE00000E00000E00001C00781C0078380070700060E0003FC000 1F0000141D7D9316>I E /Ft 81 124 df<000FC3E0007FEFF000E07C7801C0F8780380F83007 007000070070000700700007007000070070000700700007007000FFFFFF80FFFFFF8007007000 070070000700700007007000070070000700700007007000070070000700700007007000070070 0007007000070070000700700007007000070070007FE3FF007FE3FF001D20809F1B>11 D<001F80007FC001E0E00381E00381E00701E0070000070000070000070000070000070000FFFF E0FFFFE00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700 E00700E00700E00700E00700E07FC3FE7FC3FE1720809F19>I<001FE0007FE001E1E00381E003 80E00700E00700E00700E00700E00700E00700E00700E0FFFFE0FFFFE00700E00700E00700E007 00E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E07F E7FE7FE7FE1720809F19>I<000F80F800007FE7FE0000E06E060001C0FC0F000380F80F000700 F00F00070070000007007000000700700000070070000007007000000700700000FFFFFFFF00FF FFFFFF000700700700070070070007007007000700700700070070070007007007000700700700 070070070007007007000700700700070070070007007007000700700700070070070007007007 0007007007007FE3FE3FF07FE3FE3FF02420809F26>I<7038F87CFC7EFC7E7C3E0C060C060C06 180C180C381C3018603040200F0E7E9F17>34 D<70F8FCFC7C0C0C0C181838306040060E7C9F0D >39 D<006000C001800300070006000C001C00180038003800300070007000700060006000E000 E000E000E000E000E000E000E000E000E000E000E0006000600070007000700030003800380018 001C000C00060007000300018000C000600B2E7DA112>II<000600000006000000060000000600000006000000060000000600000006000000 060000000600000006000000060000000600000006000000060000FFFFFFF0FFFFFFF000060000 000600000006000000060000000600000006000000060000000600000006000000060000000600 00000600000006000000060000000600001C207D9A23>43 D<70F8FCFC7C0C0C0C181838306040 060E7C840D>II<70F8F8F87005057C840D>I<0003000300070006 0006000E000C000C001C0018001800380030003000700060006000E000C000C001C00180018001 800380030003000700060006000E000C000C001C0018001800380030003000700060006000E000 C000C000102D7DA117>I<03F0000FFC001E1E001C0E00380700780780700380700380700380F0 03C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C070 03807003807003807807803807001C0E001E1E000FFC0003F000121F7E9D17>I<018003801F80 FF80E3800380038003800380038003800380038003800380038003800380038003800380038003 8003800380038003800380FFFEFFFE0F1E7C9D17>I<03F0000FFC00183E00300F00600F80C007 80F007C0F807C0F803C0F803C02003C00007C0000780000780000F00000E00001C000038000070 0000E00001C0000380000300000600C00C00C01800C03001807FFF80FFFF80FFFF80121E7E9D17 >I<03F0000FFC001C1E00300F00780F80780F80780780380F80000F80000F00000F00001E0000 3C0003F80003F000001C00000E00000F000007800007800007C02007C0F807C0F807C0F807C0F0 0780600F80700F003C1E001FFC0003F000121F7E9D17>I<000E00000E00001E00003E00003E00 006E0000EE0000CE00018E00038E00030E00060E000E0E000C0E00180E00380E00300E00600E00 E00E00FFFFF0FFFFF0000E00000E00000E00000E00000E00000E00000E0000FFE000FFE0141E7F 9D17>I<3803003FFF003FFE003FF8003FE00030000030000030000030000030000030000031F0 0037FC003E0E003C07003807803003800003800003C00003C00003C06003C0F003C0F003C0F003 80C00780600700700E003C1E000FF80007E000121F7E9D17>I<007C0001FE000783000E03800C 07801C0780380300380000780000700000700000F3F800F7FE00FC0E00F80700F80380F80380F0 03C0F003C0F003C0F003C0F003C07003C07003C07003803803803807001C07000E1E0007FC0001 F000121F7E9D17>I<6000007FFFC07FFFC07FFF80600180C00300C00600C00600000C00001800 00180000300000600000600000E00000C00001C00001C00001C000038000038000038000038000 078000078000078000078000078000078000078000030000121F7D9D17>I<03F0000FFC001E1E 003807003007007003807003807003807003807807003E07003F0E001FDC000FF80003F80007FC 001E7E00383F00700F80700780E003C0E001C0E001C0E001C0E001C0F001807003803807001E0E 000FFC0003F000121F7E9D17>I<03F00007F8001E1C00380E00380700700700700380F00380F0 0380F003C0F003C0F003C0F003C0F003C07007C07007C0380FC01C0FC01FFBC007F3C000038000 0380000780000700300700780E00780E00701C003078001FF0000FC000121F7E9D17>I<70F8F8 F8700000000000000000000070F8F8F87005147C930D>I<70F8F8F87000000000000000000000 70F8F8F878181818183030606040051D7C930D>I<7FFFFFE0FFFFFFF000000000000000000000 00000000000000000000000000000000000000000000FFFFFFF07FFFFFE01C0C7D9023>61 D<0003800000038000000380000007C0000007C0000007C000000DE000000DE000000DE0000018 F0000018F0000018F00000307800003078000030780000603C0000603C0000603C0000E01E0000 C01E0000FFFE0001FFFF0001800F0001800F0003800F800300078003000780070007C0070003C0 0F8003C0FFE03FFEFFE03FFE1F207F9F22>65 DI<001FC04000 7FF0C001F839C003C00DC0078007C00F0003C01E0003C03E0001C03C0001C07C0001C07C0000C0 780000C0F80000C0F8000000F8000000F8000000F8000000F8000000F8000000F8000000F80000 00780000C07C0000C07C0000C03C0000C03E0001801E0001800F0003000780030003C00E0001F8 1C00007FF000001FC0001A217D9F21>IIII<000FC020007FF86001F81CE003E006E0078003 E00F0001E01E0001E01E0000E03C0000E07C0000E07C00006078000060F8000060F8000000F800 0000F8000000F8000000F8000000F8000000F8007FFCF8007FFC780001E07C0001E07C0001E03C 0001E01E0001E01E0001E00F0001E0078003E003E003E001F80E60007FFC20000FE0001E217D9F 24>III<0FFFC00FFFC0003C00003C00003C00003C00003C00003C00003C00003C0000 3C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C0020 3C00F83C00F83C00F83C00F0780060780070F0003FE0000F800012207E9E17>IIIII<001F800000FFF00001E0780007C03E000F801F000F000F001E000780 3C0003C03C0003C07C0003E07C0003E0780001E0F80001F0F80001F0F80001F0F80001F0F80001 F0F80001F0F80001F0F80001F0F80001F0780001E07C0003E07C0003E03C0003C03E0007C01E00 07800F000F000F801F0007C03E0001F0F80000FFF000001F80001C217D9F23>II<001F800000FFF00001E0780007C03E000F801F000F000F001E0007803E0007 C03C0003C07C0003E07C0003E0780001E0F80001F0F80001F0F80001F0F80001F0F80001F0F800 01F0F80001F0F80001F0F80001F0780001E07C0003E07C0003E03C0003C03E0F07C01E1F87800F 38CF000FB0DF0007F0FE0001F0780000FFF010001FF01000007010000078300000387000003FF0 00003FE000001FE000000FC0000007801C297D9F23>II<07E0800FF9801C1F80300F80700780600380E00380E00180E00180E00180F00000F0000078 00007F00003FF0001FFC000FFE0003FF00001F800007800003C00003C00001C0C001C0C001C0C0 01C0C001C0E00180F00380F80700FE0E00CFFC0081F80012217D9F19>I<7FFFFFE07FFFFFE078 0F01E0700F00E0600F0060600F0060E00F0070C00F0030C00F0030C00F0030C00F0030000F0000 000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F00 00000F0000000F0000000F0000000F0000000F0000000F0000000F000007FFFE0007FFFE001C1F 7E9E21>IIII89 D<7FFFF87FFFF87E00F07801F07001E06003E0 E003C0E00780C00780C00F00C01F00001E00003C00003C0000780000F80000F00001E00001E00C 03C00C07C00C07800C0F001C0F001C1E00183E00183C00387C00787801F8FFFFF8FFFFF8161F7D 9E1C>II<0804180C3018703860306030C060C060C060F87CFC7E FC7E7C3E381C0F0E7B9F17>II<1FE0003FF000783800781C0030 0E00000E00000E00000E0003FE001FFE003E0E00780E00F80E00F00E30F00E30F00E30F01E3078 3F703FEFE00F878014147E9317>97 D<0E0000FE0000FE00000E00000E00000E00000E00000E00 000E00000E00000E00000E00000E3F000EFF800FC3C00F00E00E00F00E00700E00780E00780E00 780E00780E00780E00780E00780E00700E00F00F00F00F01E00FC3C00CFF800C3E0015207F9F19 >I<03F80FFC1E1E3C1E380C78007000F000F000F000F000F000F0007000780038033C031F0E0F FC03F010147E9314>I<000380003F80003F800003800003800003800003800003800003800003 8000038000038003E3800FFB801E0F803C0780780380780380700380F00380F00380F00380F003 80F00380F003807003807003807807803807801E1F800FFBF803E3F815207E9F19>I<03F0000F FC001E1E003C0F00380700780700700380F00380FFFF80FFFF80F00000F00000F0000070000078 00003801801C03800F070007FE0001F80011147F9314>I<003C00FE01CF038F03060700070007 000700070007000700FFF0FFF00700070007000700070007000700070007000700070007000700 0700070007007FF07FF01020809F0E>I<0001E003E3F00FFF701C1C703C1E00380E00780F0078 0F00780F00780F00380E003C1E001C1C003FF80033E0003000003000003800003FFE001FFF803F FFC07003E07000F0E00070E00070E00070F000F07801E03E07C00FFF0003FC00141F7F9417>I< 0E0000FE0000FE00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E3E00 0EFF000FC3800F01C00F01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0 0E01C00E01C00E01C00E01C0FFE7FCFFE7FC16207F9F19>I<1C001E003E001E001C0000000000 00000000000000000E007E007E000E000E000E000E000E000E000E000E000E000E000E000E000E 000E000E00FFC0FFC00A1F809E0C>I<00E001F001F001F000E000000000000000000000000000 7007F007F000F00070007000700070007000700070007000700070007000700070007000700070 00700070007000706070F0E0F1C07F803E000C28829E0E>I<0E0000FE0000FE00000E00000E00 000E00000E00000E00000E00000E00000E00000E00000E1FF00E1FF00E0F800E0E000E0C000E18 000E30000E70000EF0000FF8000F38000E1C000E1E000E0E000E07000E07800E03800E03C0FFCF F8FFCFF815207F9F18>I<0E00FE00FE000E000E000E000E000E000E000E000E000E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00FFE0FFE00B2080 9F0C>I<0E3F03F000FEFF8FF800FFC1DC1C000F80F80E000F00F00E000E00E00E000E00E00E00 0E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E 000E00E00E000E00E00E000E00E00E00FFE7FE7FE0FFE7FE7FE023147F9326>I<0E3E00FEFF00 FFC3800F01C00F01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0 0E01C00E01C00E01C0FFE7FCFFE7FC16147F9319>I<01F80007FE001E07803C03C03801C07000 E07000E0F000F0F000F0F000F0F000F0F000F0F000F07000E07801E03801C03C03C01E078007FE 0001F80014147F9317>I<0E3F00FEFF80FFC3C00F01E00E00F00E00F00E00780E00780E00780E 00780E00780E00780E00780E00700E00F00F00F00F01E00FC3C00EFF800E3E000E00000E00000E 00000E00000E00000E00000E0000FFE000FFE000151D7F9319>I<03E1800FF9801E1F803C0780 780780780380700380F00380F00380F00380F00380F00380F003807003807803807807803C0F80 1E1F800FF38003E380000380000380000380000380000380000380000380003FF8003FF8151D7E 9318>I<0E78FEFCFF9E0F1E0F0C0F000E000E000E000E000E000E000E000E000E000E000E000E 00FFE0FFE00F147F9312>I<1F903FF07070E030E030E030F00078007F803FE00FF000F0C078C0 38C038E038E038F070DFE08F800D147E9312>I<06000600060006000E000E001E003E00FFF8FF F80E000E000E000E000E000E000E000E000E000E000E180E180E180E180E18073007E003C00D1C 7F9B12>I<0E01C0FE1FC0FE1FC00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E03C00E03C0070FC007FDFC01F1FC16147F9319>III<7FC7FC7FC7FC0703 E007038003830001C70000EE0000EC00007800003800003C00007C0000EE0001C7000187000303 800701C00F01E0FF87FEFF87FE1714809318>II<3FFF3F FF380E301C703C6038607060F001E001C0038007830F030E031C073C063806701EFFFEFFFE1014 7F9314>II E /Fu 34 119 df46 D<00000C00001C00001C00003C0000380000380000780000700000700000F00000E00000E00001 E00001C00001C00003C0000380000380000780000700000700000F00000E00000E00001E00001C 00001C00003C0000380000780000700000700000F00000E00000E00001E00001C00001C00003C0 000380000380000780000700000700000F00000E00000E00001E00001C00001C00003C00003800 00380000780000700000700000F00000E00000E00000E00000163C7DAC1D>I<00180000380000 F80007F800FFF800FFF800FFF800F8F80000F80000F80000F80000F80000F80000F80000F80000 F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000 F80000F80000F80000F80000F80000F80000F80000F8007FFFF07FFFF07FFFF07FFFF014287CA7 1D>49 D<00FE0003FF800FFFC01FFFE01F07F03C01F87800FC78007C70007CF0003EF0003E6000 3E20003E00003E00003E00003E00007C00007C0000780000F80001F00001E00003E00007C0000F 80001F00003E00007C0000F80001F00003E00003C0000780000F00001E00003C00007FFFFE7FFF FE7FFFFE7FFFFE17287EA71D>I<007E0003FF8007FFE00FFFF01F81F83E00F87C007C78007C30 007C10007C00007C00007C0000780000F80000F80001F00007E000FFC000FF8000FFC000FFF000 01F80000FC00007C00003E00003E00001F00001F00001F00001F00001F80001FC0003EE0003EF0 007E7C00FC3F01F81FFFF00FFFE003FFC000FE0018297EA71D>I<0007E0000FE0000BE0001BE0 001BE0003BE0003BE00073E00073E000F3E000F3E001E3E001E3E003E3E003C3E007C3E00F83E0 0F83E01F03E01F03E03E03E03E03E07C03E07C03E0F803E0FFFFFFFFFFFFFFFFFFFFFFFF0003E0 0003E00003E00003E00003E00003E00003E00003E00003E00003E018277EA61D>I<3FFFF83FFF F83FFFF83FFFF83E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E3E 003EFF803FFFE03FFFF03FC1F03F80F83F007C3E007C00007C00003E00003E00003E00003E0000 3E00003E00003E20003E60007C70007CF800F87C01F87F07F03FFFE00FFFC007FF0001FC001728 7EA61D>I<000FE0003FF800FFF801FFF803F81807E0000F80000F80001F00003E00003E00003C 00007C00007C02007C3FC0F8FFE0F9FFF0FBFFF8FF80FCFE007CFE003EFC003EFC003FFC001FF8 001FF8001FF8001FF8001F78001F7C001F7C001F7C001E3C003E3E003E1E007C1F00FC0FC1F807 FFF003FFE001FF80007E0018297EA71D>II<007E0003FFC007FFE00FFFF01F81 F83F00FC3E007C7E007E7C003E7C003E7C003E7C003E7C003E3E007C3E007C1F00F80FC3F007FF E003FFC003FFC00FFFF01F81F83F00FC3E007C7C003E7C003EF8001FF8001FF8001FF8001FF800 1FF8001F7C003E7C003E7E007E3F00FC1F81F80FFFF007FFE003FFC0007E0018297EA71D>I<00 03FE00000FFFC0003FFFF000FFFFF001FE03F003F8006007E0002007C000000F8000001F800000 1F0000003E0000003E0000007C0000007C0000007C0000007C000000F8000000F8000000F80000 00F8000000F8000000F8000000F8000000F8000000F8000000F80000007C0000007C0000007C00 00007C0000003E0000003E0000001F0000001F8000000F80000007C0000007E0000803F8003801 FE01F800FFFFF8003FFFF0000FFFC00003FE001D2C7CAA25>67 D69 DI73 D<0003FC0000000FFF0000003FFFC00000FFFFF00001FE07F80003F801FC0007E0007E000FC000 3F000F80001F001F00000F801F00000F803E000007C03E000007C07C000003E07C000003E07C00 0003E078000001E0F8000001F0F8000001F0F8000001F0F8000001F0F8000001F0F8000001F0F8 000001F0F8000001F0F8000001F0F8000001F07C000003E07C000003E07C000003E07E000007E0 3E000007C03E000007C01F00000F801F80001F800F80001F000FC0003F0007E0007E0003F801FC 0001FE07F80000FFFFF000003FFFC000001FFF80000003FC0000242C7DAA2B>79 D86 D<01FC000FFF003FFF803FFFC03C07E0 3003E00001F00001F00001F00001F00001F00003F001FFF007FFF01FFFF03FC1F07E01F0FC01F0 F801F0F801F0F801F0FC03F07E0FF07FFFF03FFFF01FF9F00FE1F0141B7E9A1C>97 D<007F8001FFE007FFF80FFFF81FC0783F00103E00007C00007C00007C0000F80000F80000F800 00F80000F80000F80000F800007C00007C00007C00003E00003F00181FC0780FFFF807FFF801FF E0007F80151B7E9A1A>99 D<00007C00007C00007C00007C00007C00007C00007C00007C00007C 00007C00007C00007C00007C00007C00007C00FC7C03FF7C07FFFC0FFFFC1FC1FC3F00FC3E007C 7C007C7C007CFC007CF8007CF8007CF8007CF8007CF8007CF8007CF8007CF8007C7C007C7C007C 7E00FC3F00FC1F83FC1FFFFC0FFF7C03FE7C01F87C162A7EA91E>I<00FC0003FF0007FF800FFF C01F87E03F01E03E00F07C00F07C0070F80078FFFFF8FFFFF8FFFFF8FFFFF8F80000F80000F800 007800007C00007C00003E00003F00181FC0F80FFFF807FFF801FFE0007F00151B7E9A1A>I<00 1FC0007FC000FFC001FFC003F00003E00007C00007C00007C00007C00007C00007C00007C00007 C00007C000FFFE00FFFE00FFFE00FFFE0007C00007C00007C00007C00007C00007C00007C00007 C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007 C00007C00007C000122A7FA912>I<00F80F0003FE7F800FFFFF801FFFFF801F07C0003E03E000 3E03E0007C01F0007C01F0007C01F0007C01F0007C01F0003E03E0003E03E0001F07C0001FFFC0 003FFF80003BFE000038F8000078000000780000007C0000003FFF80003FFFF0001FFFF8001FFF FC003FFFFE007C00FE007C003F00F8001F00F8001F00F8001F00F8001F00FC003F007E007E003F 81FC001FFFF8000FFFF00007FFE00000FF000019287E9A1D>III108 DII<007E0001FF8007FFE00FFFF01F81F83F00FC 3E007C7C003E7C003E78001EF8001FF8001FF8001FF8001FF8001FF8001FF8001F7C003E7C003E 7C003E3E007C3F00FC1FC3F80FFFF007FFE003FFC0007E00181B7E9A1D>II114 D<07F8001FFF003FFF807FFF807C0780F8 0100F80000F80000F80000FC00007F00007FF0003FFC001FFE0007FF0000FF80001FC0000FC000 07C00007C04007C0E007C0FC1F80FFFF807FFF001FFE0007F800121B7E9A16>I<07C00007C000 07C00007C00007C00007C00007C000FFFF80FFFF80FFFF80FFFF8007C00007C00007C00007C000 07C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C000 07C08007E38003FFC003FFC001FF0000FC0012227FA115>III E /Fv 13 122 df66 D73 D<00000003FFF0000000000000007FFFFF80000000000003FFFFFF F000000000000FFFFFFFFC00000000003FFF003FFF0000000000FFF80007FFC000000003FFE000 01FFF000000007FF8000007FF80000000FFF0000003FFC0000001FFE0000001FFE0000003FFC00 00000FFF0000007FF800000007FF800000FFF000000003FFC00001FFF000000003FFE00003FFE0 00000001FFF00003FFC000000000FFF00007FFC000000000FFF80007FFC000000000FFF8000FFF 80000000007FFC000FFF80000000007FFC001FFF80000000007FFE001FFF00000000003FFE003F FF00000000003FFF003FFF00000000003FFF003FFF00000000003FFF007FFF00000000003FFF80 7FFE00000000001FFF807FFE00000000001FFF807FFE00000000001FFF807FFE00000000001FFF 80FFFE00000000001FFFC0FFFE00000000001FFFC0FFFE00000000001FFFC0FFFE00000000001F FFC0FFFE00000000001FFFC0FFFE00000000001FFFC0FFFE00000000001FFFC0FFFE0000000000 1FFFC0FFFE00000000001FFFC0FFFE00000000001FFFC0FFFE00000000001FFFC0FFFE00000000 001FFFC0FFFE00000000001FFFC0FFFE00000000001FFFC07FFE00000000001FFF807FFF000000 00003FFF807FFF00000000003FFF807FFF00000000003FFF803FFF00000000003FFF003FFF0000 0000003FFF003FFF80000000007FFF001FFF80000000007FFE001FFF80000000007FFE001FFF80 000000007FFE000FFFC000000000FFFC000FFFC000000000FFFC0007FFE000000001FFF80003FF E000000001FFF00003FFF000000003FFF00001FFF000000003FFE00000FFF800000007FFC00000 7FFC0000000FFF8000007FFE0000001FFF8000003FFF0000003FFF0000000FFFC00000FFFC0000 0007FFE00001FFF800000003FFF80007FFF000000000FFFF003FFFC0000000007FFFFFFFFF8000 0000000FFFFFFFFC000000000003FFFFFFF00000000000007FFFFF8000000000000003FFF00000 00004A497AC757>79 D<0003FFF0000000003FFFFF00000000FFFFFFE0000001FFFFFFF8000003 FF003FFC000007FF800FFE000007FF8007FF000007FF8003FF800007FF8003FFC00007FF8001FF C00007FF8001FFC00003FF0000FFE00001FE0000FFE00000FC0000FFE00000000000FFE0000000 0000FFE00000000000FFE00000000000FFE00000000000FFE000000000FFFFE00000007FFFFFE0 000007FFFFFFE000003FFFFCFFE00000FFFE00FFE00003FFF000FFE00007FFC000FFE0000FFF00 00FFE0001FFE0000FFE0003FFC0000FFE0007FFC0000FFE0007FF80000FFE000FFF80000FFE000 FFF00000FFE000FFF00000FFE000FFF00000FFE000FFF00001FFE000FFF00001FFE000FFF80003 FFE0007FFC0007FFE0007FFC000FFFE0003FFE001F7FF8001FFF80FE7FFFF007FFFFFC3FFFF003 FFFFF01FFFF0007FFFC00FFFF00007FE0003FFF0342E7DAD38>97 D<007FC000000000FFFFC000 000000FFFFC000000000FFFFC000000000FFFFC000000000FFFFC00000000003FFC00000000001 FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000 000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FF C00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC0000000 0001FFC00000000001FFC00000000001FFC00FFE000001FFC0FFFFC00001FFC3FFFFF00001FFCF FFFFFC0001FFFFE01FFF0001FFFF0007FF8001FFFC0001FFC001FFF80000FFE001FFF00000FFF0 01FFE000007FF001FFC000007FF801FFC000003FFC01FFC000003FFC01FFC000003FFC01FFC000 001FFE01FFC000001FFE01FFC000001FFE01FFC000001FFF01FFC000001FFF01FFC000001FFF01 FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC00000 1FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFE01FFC000001FFE01FF C000003FFE01FFC000003FFC01FFC000003FFC01FFC000003FF801FFC000007FF801FFE000007F F001FFF00000FFE001FFF80001FFE001FFFC0003FFC001FF7F0007FF0001FE3FC03FFE0001FC0F FFFFF80001FC03FFFFF00001F800FFFF800001F0001FF8000038487CC741>I<00000000003F00 0000FFF000FFC00007FFFE03FFE0003FFFFFCFFFF0007FFFFFFFCFF001FFE07FFE0FF003FF801F FC1FF007FF000FFE0FF00FFE0007FF0FF00FFE0007FF07E01FFC0003FF81801FFC0003FF80003F FC0003FFC0003FFC0003FFC0003FFC0003FFC0003FFC0003FFC0003FFC0003FFC0003FFC0003FF C0003FFC0003FFC0003FFC0003FFC0003FFC0003FFC0001FFC0003FF80001FFC0003FF80000FFE 0007FF00000FFE0007FF000007FF000FFE000003FF801FFC000001FFE07FF8000003FFFFFFE000 0003FFFFFFC000000787FFFE0000000780FFF0000000078000000000000F8000000000000F8000 000000000FC000000000000FC000000000000FE000000000000FF8000000000007FFFFFFF00000 07FFFFFFFF000007FFFFFFFFE00003FFFFFFFFF80003FFFFFFFFFC0001FFFFFFFFFE0000FFFFFF FFFF0003FFFFFFFFFF800FFFFFFFFFFF801FF000001FFFC03FC0000001FFC07F800000007FE07F 800000003FE0FF000000003FE0FF000000001FE0FF000000001FE0FF000000001FE0FF00000000 1FE0FF800000003FE07F800000003FC07FC00000007FC03FE0000000FF801FF8000003FF000FFE 00000FFE0007FFE000FFFC0001FFFFFFFFF000003FFFFFFF8000000FFFFFFE000000007FFFC000 0034447DAE3A>103 D<007FC000000000FFFFC000000000FFFFC000000000FFFFC000000000FF FFC000000000FFFFC00000000003FFC00000000001FFC00000000001FFC00000000001FFC00000 000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FF C00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC0000000 0001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC0 07FE000001FFC03FFFC00001FFC0FFFFF00001FFC1FFFFFC0001FFC7F01FFC0001FFCFC00FFE00 01FFDF0007FF0001FFDE0007FF0001FFFC0007FF0001FFF80003FF8001FFF80003FF8001FFF000 03FF8001FFE00003FF8001FFE00003FF8001FFE00003FF8001FFC00003FF8001FFC00003FF8001 FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003 FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FF C00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF 8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC0 0003FF8001FFC00003FF80FFFFFF81FFFFFFFFFFFF81FFFFFFFFFFFF81FFFFFFFFFFFF81FFFFFF FFFFFF81FFFFFF38487BC741>I<00FC0001FF0003FF8007FF800FFFC00FFFC00FFFE00FFFE00F FFE00FFFC00FFFC007FF8003FF8001FF0000FC0000000000000000000000000000000000000000 0000000000000000000000000000000000007FC0FFFFC0FFFFC0FFFFC0FFFFC0FFFFC003FFC001 FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001 FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001 FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC0FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF18 497CC820>I<007FC000FFFFC000FFFFC000FFFFC000FFFFC000FFFFC00003FFC00001FFC00001 FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC000 01FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FF C00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001 FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC000 01FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 00FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF8019487CC720>108 D<00000FFE0000000001 FFFFF000000007FFFFFC0000001FFFFFFF0000007FF803FFC00000FFE000FFE00001FF80003FF0 0003FF00001FF80007FE00000FFC000FFE00000FFE000FFC000007FE001FFC000007FF003FFC00 0007FF803FF8000003FF803FF8000003FF807FF8000003FFC07FF8000003FFC07FF8000003FFC0 FFF8000003FFE0FFF8000003FFE0FFF8000003FFE0FFF8000003FFE0FFF8000003FFE0FFF80000 03FFE0FFF8000003FFE0FFF8000003FFE0FFF8000003FFE0FFF8000003FFE07FF8000003FFC07F F8000003FFC07FF8000003FFC07FF8000003FFC03FFC000007FF803FFC000007FF801FFC000007 FF001FFC000007FF000FFE00000FFE0007FF00001FFC0003FF00001FF80001FFC0007FF00000FF E000FFE000007FFC07FFC000001FFFFFFF00000007FFFFFC00000001FFFFF0000000001FFF0000 00332E7DAD3A>111 D<007FC00FFE0000FFFFC0FFFFC000FFFFC3FFFFF000FFFFCFFFFFFC00FF FFFFE03FFF00FFFFFF000FFF8003FFFC0007FFC001FFF80003FFE001FFF00001FFF001FFE00000 FFF001FFC00000FFF801FFC000007FFC01FFC000007FFC01FFC000003FFC01FFC000003FFE01FF C000003FFE01FFC000003FFE01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001F FF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC0 00001FFF01FFC000001FFF01FFC000001FFF01FFC000003FFE01FFC000003FFE01FFC000003FFE 01FFC000007FFC01FFC000007FFC01FFC000007FF801FFC00000FFF801FFE00000FFF001FFF000 01FFE001FFF80003FFE001FFFC0007FFC001FFFF000FFF0001FFFFC07FFE0001FFCFFFFFF80001 FFC3FFFFF00001FFC0FFFF800001FFC01FF8000001FFC00000000001FFC00000000001FFC00000 000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FF C00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC0000000 0001FFC000000000FFFFFF80000000FFFFFF80000000FFFFFF80000000FFFFFF80000000FFFFFF 8000000038427CAD41>I<00FF807F00FFFF81FFC0FFFF87FFF0FFFF8FFFF8FFFF9F8FFCFFFF9F 1FFE03FFBE1FFE01FFBC1FFE01FFF81FFE01FFF81FFE01FFF01FFE01FFF00FFC01FFE007F801FF E003F001FFE0000001FFE0000001FFE0000001FFC0000001FFC0000001FFC0000001FFC0000001 FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC00000 01FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC000 0001FFC0000001FFC0000001FFC0000001FFC00000FFFFFFE000FFFFFFE000FFFFFFE000FFFFFF E000FFFFFFE000272E7CAD2F>114 D121 D E /Fw 8 117 df<00001E000000003E00000000FE 00000003FE0000003FFE0000FFFFFE0000FFFFFE0000FFFFFE0000FFCFFE0000000FFE0000000F FE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE000000 0FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000 000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE00 00000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE 0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000F FE0000000FFE0000000FFE00007FFFFFFF807FFFFFFF807FFFFFFF807FFFFFFF80213879B730> 49 D<0000001FFE0000E0000003FFFFE001E000001FFFFFF803E000007FFFFFFE07E00001FFFC 00FF0FE00007FFC0001FDFE0000FFF000007FFE0003FFC000001FFE0007FF0000000FFE000FFE0 0000007FE001FFC00000003FE003FF800000001FE007FF800000001FE007FF000000000FE00FFE 0000000007E00FFE0000000007E01FFC0000000007E01FFC0000000003E03FFC0000000003E03F F80000000003E07FF80000000001E07FF80000000001E07FF80000000001E07FF0000000000000 FFF0000000000000FFF0000000000000FFF0000000000000FFF0000000000000FFF00000000000 00FFF0000000000000FFF0000000000000FFF0000000000000FFF0000000000000FFF000000000 0000FFF0000000000000FFF0000000000000FFF00000000000007FF00000000000007FF8000000 0000007FF80000000001E07FF80000000001E03FF80000000001E03FFC0000000001E01FFC0000 000001E01FFC0000000003E00FFE0000000003C00FFE0000000007C007FF0000000007C007FF80 0000000F8003FF800000000F8001FFC00000001F0000FFE00000003E00007FF00000007C00003F FC000000F800000FFF000003F0000007FFC0000FE0000001FFFC007FC00000007FFFFFFF000000 001FFFFFFC0000000003FFFFE000000000001FFE0000003B3D7BBB46>67 D<003FFF00000001FFFFE0000007FFFFFC00000FF007FE00001FF801FF80001FF800FFC0001FF8 007FE0001FF8007FE0001FF8007FF0000FF0003FF00007E0003FF00003C0003FF0000000003FF0 000000003FF0000000003FF0000000003FF0000000FFFFF000000FFFFFF000007FF83FF00003FF 803FF00007FE003FF0001FFC003FF0003FF8003FF0007FF0003FF0007FE0003FF000FFE0003FF0 00FFC0003FF000FFC0003FF000FFC0003FF000FFC0007FF000FFC0007FF000FFE000FFF0007FE0 01DFF0003FF003DFFC001FFC0F9FFFE00FFFFE0FFFE001FFF807FFE0003FE001FFE02B267DA52F >97 D<0003FF8000001FFFF000007FFFFC0000FF83FF0003FE00FF8007FC003F800FF8003FC01F F8001FE01FF0001FE03FF0001FF03FF0000FF07FE0000FF07FE0000FF87FE0000FF8FFE0000FF8 FFE0000FF8FFFFFFFFF8FFFFFFFFF8FFFFFFFFF8FFE0000000FFE0000000FFE0000000FFE00000 00FFE00000007FE00000007FE00000007FF00000003FF00000783FF00000781FF80000F80FF800 00F007FC0001F003FE0003E001FF000FC000FFC07F80003FFFFE00000FFFFC000001FFC0002526 7DA52C>101 D<00FF0000000000FFFF0000000000FFFF0000000000FFFF0000000000FFFF0000 00000007FF000000000003FF000000000003FF000000000003FF000000000003FF000000000003 FF000000000003FF000000000003FF000000000003FF000000000003FF000000000003FF000000 000003FF000000000003FF000000000003FF000000000003FF000000000003FF000000000003FF 000000000003FF003FE0000003FF01FFFC000003FF07FFFE000003FF0F81FF800003FF3C00FF80 0003FF3800FFC00003FF7000FFC00003FFE0007FE00003FFC0007FE00003FFC0007FE00003FF80 007FE00003FF80007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE000 03FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF0000 7FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003 FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007F E00003FF00007FE000FFFFFC1FFFFF80FFFFFC1FFFFF80FFFFFC1FFFFF80FFFFFC1FFFFF80313C 7DBB36>104 D<00FF00FF8000FFFF0FFFF800FFFF3FFFFE00FFFFFE03FF00FFFFF000FFC007FF E0007FE003FFC0003FF003FF80003FF803FF00001FF803FF00001FFC03FF00000FFC03FF00000F FE03FF00000FFE03FF00000FFE03FF000007FF03FF000007FF03FF000007FF03FF000007FF03FF 000007FF03FF000007FF03FF000007FF03FF000007FF03FF000007FF03FF000007FF03FF00000F FE03FF00000FFE03FF00000FFE03FF00000FFC03FF00001FFC03FF00001FF803FF80003FF003FF C0007FF003FFE000FFE003FFF001FF8003FFFE07FF0003FF3FFFFC0003FF0FFFF00003FF01FF00 0003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF 0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF000000 00FFFFFC000000FFFFFC000000FFFFFC000000FFFFFC00000030377EA536>112 D<00FE01F800FFFE07FF00FFFE1FFF80FFFE3E3FC0FFFE787FE007FE707FE003FEE07FE003FEE0 7FE003FFC07FE003FFC03FC003FF801F8003FF800F0003FF80000003FF80000003FF00000003FF 00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003 FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF000000 03FF00000003FF00000003FF000000FFFFFE0000FFFFFE0000FFFFFE0000FFFFFE000023267EA5 28>114 D<000F0000000F0000000F0000000F0000000F0000001F0000001F0000001F0000001F 0000003F0000003F0000007F0000007F000000FF000001FF000003FF000007FF00001FFFFFF0FF FFFFF0FFFFFFF0FFFFFFF003FF000003FF000003FF000003FF000003FF000003FF000003FF0000 03FF000003FF000003FF000003FF000003FF000003FF000003FF000003FF000003FF000003FF00 0003FF000003FF000003FF003C03FF003C03FF003C03FF003C03FF003C03FF003C03FF003C03FF 003C01FF807801FF807800FFC0F000FFE1F0003FFFE0000FFF800001FE001E377EB626>116 D E /Fx 25 122 df<70F8FCFC7C0C0C0C0C181830306040060F7C840E>44 DI<018003800F80FF80F380038003800380038003800380038003 80038003800380038003800380038003800380038003800380038003800380038003800380FFFE FFFE0F217CA018>49 D<000E00000E00001E00001E00003E00003E00006E0000EE0000CE0001CE 00018E00030E00070E00060E000E0E000C0E00180E00180E00300E00700E00600E00E00E00FFFF F8FFFFF8000E00000E00000E00000E00000E00000E00000E0001FFF001FFF015217FA018>52 D<01F00007FC000E0E001C0700380380780380700380F001C0F001C0F001C0F001E0F001E0F001 E0F001E0F001E07001E07003E07803E03807E01C0DE00FF9E007F1E00081C00001C00003C00003 80000380300700780700780E00701C003838001FF0000FC00013227EA018>57 D<000FE010003FF83000F81C7001E0067003C003F0078001F00F0000F01E0000F03E0000703C00 00707C0000707C0000307800003078000030F8000030F8000000F8000000F8000000F8000000F8 000000F8000000F800000078000030780000307C0000307C0000303C0000603E0000601E000060 0F0000C0078000C003C0018001E0030000F80E00003FF800000FE0001C247DA223>67 DI70 D<03FFF003FFF0000F00000F00000F00000F00000F00000F00 000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00 000F00000F00000F00000F00000F00700F00F80F00F80F00F80F00F01E00601C003878001FF000 07C00014237EA119>74 D77 D80 D<03F0200FFC601C0EE03803 E07001E07001E0E000E0E000E0E00060E00060E00060F00000F000007800007F00003FF0001FFE 000FFF0003FF80003FC00007E00001E00000F00000F0000070C00070C00070C00070C00070E000 60E000E0F000C0F801C0EF0380C7FF0081FC0014247DA21B>83 D<0FE0001FF8003C1C003C0E00 180700000700000700000F0003FF000FFF003F07007C0700780700F00700F00718F00718F00F18 780F187C3FB83FF3F00FC3C015157E9418>97 D<0E0000FE0000FE00001E00000E00000E00000E 00000E00000E00000E00000E00000E00000E00000E00000E1F800E7FE00FC0F00F00780E00380E 003C0E001C0E001E0E001E0E001E0E001E0E001E0E001E0E001E0E001C0E003C0F00380F80700F C1F00C7FC00C1F0017237FA21B>I<01FE0007FF000F07801C0780380300780000700000F00000 F00000F00000F00000F00000F00000F000007800007800C03C00C01E01800F030007FE0001F800 12157E9416>I<01FC0007FF000F07801C03C03801C07801E07000E0FFFFE0FFFFE0F00000F000 00F00000F00000F000007800007800603C00601E00C00F838007FF0000FC0013157F9416>101 D<1C001E003E001E001C00000000000000000000000000000000000E00FE00FE001E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E00FFC0FFC00A227FA10E>105 D<0E00FE00FE001E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E000E00FFE0FFE00B237FA20E> 108 D<0E1F80FE7FC0FFC1E01F80F00F00700E00700E00700E00700E00700E00700E00700E0070 0E00700E00700E00700E00700E00700E00700E0070FFE7FFFFE7FF18157F941B>110 D<01FC0007FF000F07801C01C03800E07800F0700070F00078F00078F00078F00078F00078F000 78F000787000707800F03800E01C01C00F078007FF0001FC0015157F9418>I<0E3CFEFEFFCF1F 8F0F060F000E000E000E000E000E000E000E000E000E000E000E000E000E00FFF0FFF010157F94 13>114 D<0F883FF87078E038E018E018E018F0007F003FE01FF001F8003CC01CC01CE01CE01C F018F878DFF08FC00E157E9413>I<060006000600060006000E000E000E001E003E00FFF8FFF8 0E000E000E000E000E000E000E000E000E000E000E0C0E0C0E0C0E0C0E0C0E08071803F001E00E 1F7F9E13>I<0E0070FE07F0FE07F01E00F00E00700E00700E00700E00700E00700E00700E0070 0E00700E00700E00700E00700E00700E00F00E01F007037803FE7F01F87F18157F941B>I121 D E /Fy 20 122 df68 D73 D76 DI< 00003FC000000001FFF800000007C03E0000001F000F8000003C0003C00000F80001F00001F000 00F80003E000007C0003C000003C0007C000003E000F8000001F000F8000001F001F0000000F80 1F0000000F803E00000007C03E00000007C07E00000007E07E00000007E07C00000003E07C0000 0003E0FC00000003F0FC00000003F0FC00000003F0FC00000003F0FC00000003F0FC00000003F0 FC00000003F0FC00000003F0FC00000003F0FC00000003F0FC00000003F07E00000007E07E0000 0007E07E00000007E07E00000007E03E00000007C03F0000000FC01F0000000F801F0000000F80 0F8000001F000F8000001F0007C000003E0003E000007C0003E000007C0001F00000F80000F800 01F000003E0007C000001F000F80000007E07E00000001FFF8000000003FC000002C337CB134> 79 DI<01FE00000FFFC0001C03E0003C00F0003E00 78003E003C003E001C0008001E0000001E0000001E0000001E0000001E000001FE00003FFE0001 FF1E0007F01E000F801E001F001E003E001E007C001E007C001E00F8001E0CF8001E0CF8001E0C F8003E0CF8003E0C7C007E0C3E00DF1C1F038FB80FFF07F003F803C01E1F7D9E21>97 D<07800000FF800000FF8000000F80000007800000078000000780000007800000078000000780 000007800000078000000780000007800000078000000780000007800000078000000780000007 80FC000787FF80079E07C007B801E007E000F007C0007807C0007C0780003C0780003E0780003E 0780001E0780001F0780001F0780001F0780001F0780001F0780001F0780001F0780001F078000 1F0780001E0780003E0780003E0780003C07C0007807C00078076000F0073801E0071E07C00607 FF000601FC0020327EB125>I<003FC000FFF803E01C07801E0F003E1E003E1E003E3C00087C00 007C0000780000F80000F80000F80000F80000F80000F80000F80000F80000F800007C00007C00 007C00003E00031E00031F00060F800E07C01C03F03800FFE0003F80181F7D9E1D>I<003F8000 00FFE00003E0F80007803C000F001E001E001E001E000F003C000F007C000F007C000F80780007 80F8000780FFFFFF80FFFFFF80F8000000F8000000F8000000F8000000F8000000F80000007800 00007C0000003C0000003C0001801E0001800F0003000F80060007C00E0001F03C0000FFF00000 1FC000191F7E9E1D>101 D<0003E0001FF8003C3800707C00F07C01E07C01E03803C00003C000 03C00003C00003C00003C00003C00003C00003C00003C00003C00003C000FFFFC0FFFFC003C000 03C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C000 03C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00007E000 7FFF007FFF0016327FB114>I<07000F801F801F800F8007000000000000000000000000000000 00000000000000000780FF80FF800F800780078007800780078007800780078007800780078007 800780078007800780078007800780078007800780078007800FC0FFF8FFF80D307EAF12>105 D<0780FF80FF800F80078007800780078007800780078007800780078007800780078007800780 078007800780078007800780078007800780078007800780078007800780078007800780078007 80078007800780078007800780078007800FC0FFFCFFFC0E327EB112>108 D<0781FE003FC000FF87FF80FFF000FF8E07C1C0F8000F9803E3007C0007B001E6003C0007E000 FC001E0007C000F8001E0007C000F8001E0007C000F8001E00078000F0001E00078000F0001E00 078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0 001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E0007 8000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F000 1E000FC001F8003F00FFFC1FFF83FFF0FFFC1FFF83FFF0341F7E9E38>I<0781FE0000FF87FF80 00FF8E07C0000F9803E00007B001E00007E000F00007C000F00007C000F00007C000F000078000 F000078000F000078000F000078000F000078000F000078000F000078000F000078000F0000780 00F000078000F000078000F000078000F000078000F000078000F000078000F000078000F00007 8000F000078000F000078000F0000FC001F800FFFC1FFF80FFFC1FFF80211F7E9E25>I<001FC0 0000FFF80001E03C0007800F000F0007801E0003C01E0003C03C0001E03C0001E0780000F07800 00F0780000F0F80000F8F80000F8F80000F8F80000F8F80000F8F80000F8F80000F8F80000F878 0000F07C0001F03C0001E03C0001E01E0003C01E0003C00F00078007C01F0001F07C0000FFF800 001FC0001D1F7E9E21>I<0783E0FF8FF8FF9C7C0FB07C07F07C07E03807C00007C00007C00007 800007800007800007800007800007800007800007800007800007800007800007800007800007 80000780000780000780000780000780000FC000FFFE00FFFE00161F7E9E19>114 D<00C00000C00000C00000C00000C00001C00001C00001C00003C00003C00007C0000FC0001FC0 00FFFFE0FFFFE003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C0 0003C00003C00003C00003C00003C00003C03003C03003C03003C03003C03003C03003C03003C0 3001E06001E06000F0C0007F80001F00142C7FAB19>116 D<078000F000FF801FF000FF801FF0 000F8001F000078000F000078000F000078000F000078000F000078000F000078000F000078000 F000078000F000078000F000078000F000078000F000078000F000078000F000078000F0000780 00F000078000F000078000F000078000F000078000F000078001F000078001F000078003F00003 8007F00003C00EF00001F03CF80000FFF0FF80003FC0FF80211F7E9E25>I121 D E /Fz 5 85 df<00000000600000000000E00000000000E00000000001E00000000001F00000 000003F00000000003F00000000007F0000000000FF0000000000FF0000000001BF0000000001B F00000000033F00000000033F00000000063F00000000063F000000000C3F000000001C3F00000 000183F00000000303F00000000303F00000000603F80000000603F80000000C01F80000000C01 F80000001801F80000003801F80000003001F80000006001F80000006001F8000000C001F80000 00C001F80000018001F8000001FFFFF8000003FFFFF80000070001F80000060001F800000C0001 F800000C0001F80000180001FC0000180001FC0000300000FC0000300000FC0000600000FC0000 E00000FC0000C00000FC0001C00000FC0003C00000FC000FE00003FC00FFFC003FFFE0FFF8003F FFE02B337CB234>65 D<000FFFFFFC00000FFFFFFF8000007F000FC000007E0003E000007E0001 F000007E0000F800007E00007C0000FC00007C0000FC00003E0000FC00003E0000FC00003E0001 F800001E0001F800001F0001F800001F0001F800001F0003F000001F0003F000001F0003F00000 1F0003F000001F0007E000003F0007E000003F0007E000003F0007E000003F000FC000003E000F C000007E000FC000007E000FC000007E001F8000007C001F800000FC001F800000FC001F800000 F8003F000001F8003F000001F0003F000003F0003F000003E0007E000007E0007E000007C0007E 00000F80007E00000F0000FC00001F0000FC00003E0000FC00007C0000FC0000F80001F80001F0 0001F80007C00001F8000F800003F8007E0000FFFFFFF80000FFFFFFC0000030317BB035>68 D<000FFFFFFFF8000FFFFFFFF800007F0003F800007E0000F800007E00007800007E0000780000 7E0000300000FC0000300000FC0000300000FC0000300000FC0000300001F80000300001F80000 300001F80000300001F80000600003F00180600003F00180000003F00180000003F00180000007 E00300000007E00300000007E00700000007E01F0000000FFFFE0000000FFFFE0000000FC01E00 00000FC00E0000001F800C0000001F800C0000001F800C0000001F800C0000003F00180000003F 00180000003F00000000003F00000000007E00000000007E00000000007E00000000007E000000 0000FC0000000000FC0000000000FC0000000000FC0000000001F80000000001F80000000001F8 0000000003F800000000FFFFF0000000FFFFF00000002D317BB02E>70 D<000FFFFFE000000FFF FFFC0000007F003F0000007E000F8000007E0007C000007E0003E000007E0003F00000FC0001F0 0000FC0001F00000FC0001F00000FC0001F00001F80003F00001F80003F00001F80003F00001F8 0003E00003F00007E00003F00007C00003F0000FC00003F0001F800007E0003F000007E0007C00 0007E000F8000007E007E000000FFFFF8000000FFFFF0000000FC00FC000000FC007E000001F80 03F000001F8001F000001F8001F800001F8001F800003F0001F800003F0001F800003F0001F800 003F0001F800007E0003F000007E0003F000007E0003F000007E0003F00000FC0007E00000FC00 07E00000FC0007E00000FC0007E01801F80007E03001F80007E03001F80007E06003F80003E060 FFFFE001F1C0FFFFC000FF80000000003E002D327BB033>82 D<07FFFFFFFFF007FFFFFFFFF00F E007F007F00F8007E001F00E0007E000E00C0007E000E01C0007E000E018000FC000E018000FC0 00E030000FC000C030000FC000C030001F8000C060001F8000C060001F8000C060001F8000C0C0 003F00018000003F00000000003F00000000003F00000000007E00000000007E00000000007E00 000000007E0000000000FC0000000000FC0000000000FC0000000000FC0000000001F800000000 01F80000000001F80000000001F80000000003F00000000003F00000000003F00000000003F000 00000007E00000000007E00000000007E00000000007E0000000000FC0000000000FC000000000 0FC0000000000FC0000000001F80000000001F80000000001F80000000003F800000007FFFFF80 00007FFFFF8000002C3174B032>84 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin %%EndSetup %%Page: 0 1 bop 799 988 a Fz(D)25 b(R)g(A)g(F)g(T)388 1079 y Fy(Do)r(cumen)n(t)c(for)h(a) f(P)n(arallel)h(MPI)g(IO)g(Library)207 1274 y Fx(Marc)16 b(Snir)116 b(P)o(eter)16 b(Corb)q(ett)117 b(Dror)17 b(F)l(eitelson)116 b(Jean-Pierre)15 b(Prost)793 1400 y(Jan)o(uary)i(14,)f(1994)p eop %%Page: 1 2 bop 75 362 a Fw(Chapter)31 b(1)75 570 y Fv(IO)75 812 y Fu(1.1)59 b(Intro)r(duction)75 916 y Ft(This)16 b(do)q(cumen)o(t)f(is)g(a)g(prop)q (osal)g(for)g(an)g(extension)g(of)g(MPI)g(in)h(supp)q(ort)f(of)f(\014le)i (I/O)g(\(MPI-IO\).)f(The)75 972 y(need)g(for)e(suc)o(h)i(extension)g(arises)f (from)f(the)i(fact)e(that)g(not)h(all)h(parallel)h(mac)o(hines)f(supp)q(ort)f (Unix)h(\014le)75 1029 y(access)h(at)e(eac)o(h)i(no)q(de;)g(when)f(it)h(is)g (supp)q(orted,)g(the)f(sharing)h(seman)o(tics)f(of)g(Unix)i(\014les)f(are)f (ill-suited)75 1085 y(to)c(parallel)j(computing,)e(and)h(dep)q(end)g(on)f (the)g(t)o(yp)q(e)g(of)f(\014le)i(system)f(accessed)g(and)g(on)g(the)g(mec)o (hanism)75 1141 y(used)18 b(to)e(share)h(access)g(across)f(a)g(parallel)j (application.)27 b(It)16 b(is)i(hard)f(to)f(co)q(ordinate)h(accesses)h(to)e (the)75 1198 y(same)f(\014le)h(b)o(y)f(all)h(pro)q(cesses)g(in)f(a)g (parallel)i(computation,)e(and)g(imp)q(ossible)i(to)e(guaran)o(tee)f (atomicit)o(y)75 1254 y(and)j(serializabili)q(t)o(y)i(of)d(accesses)i(with)f (lev)o(els)h(of)e(p)q(erformance)h(required)h(b)o(y)f(parallel)i (applications.)75 1311 y(The)i(b)q(eha)o(vior)g(of)f(F)l(ortran)f(I/O)i (libraries)h(further)e(complicate)i(the)f(picture.)36 b(Therefore,)22 b(use)e(of)75 1367 y(regular)11 b(I/O)h(within)h(parallel)f(applications)h (should)g(b)q(e)f(exercised)g(with)g(m)o(uc)o(h)f(caution,)h(in)g(particular) 75 1424 y(when)k(using)g(standard)e(input,)i(standard)f(output)g(and)g (standard)g(error.)166 1481 y(MPI-IO)k(pro)o(vides)h(a)e(paradigm)h(for)f (I/O)h(that)f(matc)o(hes)g(the)h(general)g(computing)g(paradigm)75 1538 y(supp)q(orted)c(b)o(y)g(MPI.)f(In)h(particular,)g(it)g(supp)q(orts)f(a) g(lo)q(osely)i(sync)o(hronous)f(collectiv)o(e)h(mo)q(de)f(of)f(I/O,)75 1594 y(analogous)i(to)g(the)h(lo)q(osely)g(sync)o(hronous)g(mo)q(de)g(of)f (comm)o(unication)h(pro)o(vided)g(b)o(y)g(collectiv)o(e)h(com-)75 1651 y(m)o(unication.)166 1708 y(The)11 b(general)g(approac)o(h)f(tak)o(en)g (b)o(y)h(MPI-IO)g(is)g(to)f(mo)q(del)h(\014le)h(I/O)f(as)f(a)g(comm)o (unication)i(b)q(et)o(w)o(een)75 1765 y(computing)19 b(pro)q(cesses)f (\(readers)g(or)f(writers\))h(and)g(a)f(\014le)i(serv)o(er)f(pro)q(cess.)29 b(A)18 b(simple,)h(nonscalable)75 1821 y(implemen)o(tation)e(of)e(MPI-IO)i (will)h(ha)o(v)o(e)d(one)h(\014le)h(serv)o(er)e(pro)q(cess)h(that)f(accesses) h(the)g(\014le)h(on)f(b)q(ehalf)75 1877 y(of)j(the)g(remaining)i(pro)q (cesses;)g(those)e(pro)q(cesses)h(comm)o(unicate)g(with)f(the)h(\014le)g (serv)o(er)f(pro)q(cess)h(via)75 1934 y(messages.)e(More)11 b(e\016cien)o(t)h(implemen)o(tations)g(are)f(p)q(ossible;)j(in)f(particular,) f(one)f(can)h(tak)o(e)f(adv)m(an)o(tage)75 1990 y(of)h(the)g(concurrency)h(a) o(v)m(ailable)h(in)f(a)f(parallel)i(\014le)f(system)f(suc)o(h)g(as)g(V)l (esta)g([2)o(],)g(in)h(order)f(to)g(replace)h(the)75 2047 y(sequen)o(tial)j (\014le)h(serv)o(er)d(b)o(y)i(m)o(ultiple)h(parallel)f(\014le)h(serv)o(ers,)d (th)o(us)h(a)o(v)o(oiding)g(sequen)o(tial)i(b)q(ottlenec)o(ks.)166 2104 y(The)g(I/O)h(calls)g(use)f(the)g(same)g(syn)o(tax)f(and)h(seman)o(tics) h(as)e(message)h(passing)g(comm)o(unication)75 2161 y(in)h(MPI.)e(A)h(\014le) h(is)g(op)q(ened)g(b)o(y)e(a)h(group)g(of)f(pro)q(cesses,)h(sp)q(eci\014ed)i (b)o(y)e(a)g Fs(c)n(ommunic)n(ator)p Ft(.)25 b(The)17 b(op)q(en)75 2217 y(op)q(eration)k(creates)f(a)g(new)h(comm)o(unicator,)g(whic)o(h)g(is)g (used)h(for)d(accessing)j(the)e(\014le.)37 b(File)22 b(access)75 2274 y(can)e(b)q(e)h(individual,)j(analogous)c(to)g(a)g(p)q(oin)o(t-to-p)q (oin)o(t)h(comm)o(unication,)g(or)f(it)g(can)h(b)q(e)g(collectiv)o(e,)75 2330 y(in)o(v)o(olving)f(all)f(pro)q(cesses)g(in)g(the)f(group.)30 b(Both)18 b(individual)j(and)e(collectiv)o(e)h(accesses)f(ha)o(v)o(e)f(sev)o (eral)75 2387 y(v)m(arian)o(ts,)d(according)g(to)g(the)g(mo)q(de)h(of)e(in)o (teraction)i(b)q(et)o(w)o(een)g(accesses)f(to)g(the)g(same)g(\014le.)166 2444 y(MPI-IO)i(supp)q(orts)g(\014les)h(that)e(are)g(structured)h(as)f(C)g (\014les)i(\(a)e(sequence)i(of)e(b)o(ytes\),)g(as)g(F)l(ortran)75 2501 y(direct-access)h(\014les)g(\(a)f(sequence)h(of)f(\014xed-size)i (records\))e(and)g(as)g(F)l(ortran)f(sequen)o(tial)j(unformatted)75 2557 y(\014les)e(\(a)f(sequence)h(of)f(v)m(ariable)h(size)h(records\).)166 2615 y(Regular)22 b(F)o(OR)l(TRAN)f(or)g(C)g(I/O)g(library)h(routines)f(ma)o (y)f(co)q(exist)i(in)g(a)e(co)q(de)i(with)f(MPI-IO)p 75 2661 720 2 v 127 2688 a Fr(1)144 2704 y Fq(V)m(ersion)14 b(as)f(of)g(Jan)g(13,)g (1994)-32 46 y Fr(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 2 3 bop 75 -100 a Ft(2)1410 b Fp(CHAPTER)16 b(1.)34 b(IO)75 49 y Ft(pro)q(cedures.)39 b(Ho)o(w)o(ev)o(er,)22 b(implemen)o(tations)g(ma)o(y)f (prohibit)i(concurren)o(t)e(access)h(to)e(the)i(same)f(\014le)75 106 y(using)16 b(b)q(oth)f(MPI-IO)h(and)g(regular)f(\014le)h(I/O.)166 165 y(MPI-IO)g(inherits)g(from)e(the)h(design)g(of)g(MPI)f(comm)o(unication)i (calls)g(sev)o(eral)f(features)f(that)g(are)75 222 y(imp)q(ortan)o(t)j(for)h (high)g(p)q(erformance)g(I/O.)g(Data)e(can)i(b)q(e)h(mo)o(v)o(ed)e(directly)i (to)e(user)h(space,)h(a)o(v)o(oiding)75 278 y(extra)12 b(bu\013ering)i(at)e (the)h(computation)g(no)q(de.)19 b(Deriv)o(ed)14 b(datat)o(yp)q(es)e(pro)o (vide)h(a)g(facilit)o(y)h(for)e(scattering)75 335 y(data)k(read)h(or)g (gathering)g(data)f(to)g(b)q(e)i(written)f(as)g(part)f(of)g(the)i(I/O)f (call.)26 b(Async)o(hronous)17 b(I/O)h(can)75 391 y(b)q(e)e(executed)g(in)g (a)f(manner)g(similar)i(to)d(non)o(blo)q(c)o(king)j(comm)o(unication.)166 450 y(The)e(\014rst)f(sections)h(of)f(this)h(do)q(cumen)o(t)g(describ)q(e)i (the)d(MPI-IO)i(functions)f(in)g(terms)g(of)f(accesses)75 507 y(to)g(a)h(regular)g(\(F)l(ortran)f(or)g(C\))h(\014le.)21 b(The)15 b(last)g(section)g(of)g(this)g(do)q(cumen)o(t)h(presen)o(ts)f(extensions)h (that)75 563 y(are)f(sp)q(eci\014cally)j(targeted)c(to)g(a)h(parallel)i (\014le)f(system)f(suc)o(h)g(as)g(V)l(esta)g([2)o(],)f(and)i(allo)o(w)f(the)g (user)g(more)75 620 y(con)o(trol)h(on)h(the)f(ph)o(ysical)i(\014le)f(la)o(y)o (out,)f(the)h(logical)g(view)g(of)f(the)h(\014le)g(and)g(on)f(\014le)i (sharing.)23 b(W)l(e)17 b(also)75 676 y(sp)q(ecify)g(in)f(the)f(last)g (section)h(the)f(precise)h(seman)o(tics)g(of)e(MPI-IO)j(accesses)e(to)g (parallel)h(\014les.)75 815 y Fo(1.1.1)49 b(Ackno)o(wledgements)75 907 y Ft(The)19 b(prop)q(osal)h(is)f(based)h(on)f(an)g(earlier)h(do)q(cumen)o (t)g(of)e(Jean-Pierre)j(Prost)d([7)o(],)h(and)h(incorp)q(orates)75 964 y(ideas)14 b(dev)o(elop)q(ed)g(b)o(y)f(the)h(V)l(esta)e(team)h([3)o(].)19 b(The)13 b(o)o(v)o(erall)g(design)h(has)f(b)q(een)h(in\015uenced)i(b)o(y)d (the)g(design)75 1020 y(of)i(the)g(Express)g(I/O)h(library)g([6)o(].)75 1180 y Fu(1.2)59 b(File)20 b(structure)75 1288 y Ft(MPI-IO)c(supp)q(orts)f (three)h(t)o(yp)q(es)f(of)g(\014les:)75 1394 y Fn(stream)i(\014le)23 b Ft(whic)o(h)16 b(consists)g(of)e(a)h(sequence)i(of)d(b)o(ytes.)75 1500 y Fn(\014xed)j(record)g(\014le)23 b Ft(whic)o(h)16 b(consists)g(of)e(a)h (sequence)i(of)d(\014xed)i(size)g(records.)75 1606 y Fn(v)m(ariable)i(record) f(\014le)24 b Ft(whic)o(h)16 b(consists)f(of)g(a)g(sequence)h(of)f(v)m (ariable)h(size)h(records.)166 1712 y(Stream)10 b(\014les)i(ha)o(v)o(e)f(the) g(same)g(structure)f(as)h(C)g(\(binary\))g(\014les.)19 b(Fixed)12 b(record)f(\014les)h(ha)o(v)o(e)e(the)h(same)75 1768 y(structure)18 b(as)g(F)l(ortran)g(direct)h(access,)g(unformatted)f(\014les.)30 b(V)l(ariable)20 b(record)e(\014les)i(ha)o(v)o(e)e(the)g(same)75 1824 y(structure)g(as)g(F)l(ortran)g(sequen)o(tial,)i(unformatted)e(\014les.) 30 b(Th)o(us,)19 b(\014les)h(can)e(b)q(e)i(exc)o(hanged)f(b)q(et)o(w)o(een)75 1881 y(parallel)f(MPI)e(programs)f(and)i(sequen)o(tial)g(C)f(or)g(F)l(ortran) f(programs.)22 b(MPI-IO)17 b(supp)q(orts)f(access)h(to)75 1937 y(all)f(three)f(t)o(yp)q(es)h(of)e(\014les)j(b)q(oth)e(from)f(F)l(ortran)g (and)i(C.)166 1997 y(Bytes)k(in)h(a)f(stream)g(\014le)h(and)g(records)f(in)h (a)f(\(\014xed)g(or)g(v)m(ariable\))i(record)e(\014le)h(are)f(n)o(um)o(b)q (ered)75 2053 y(consecutiv)o(ely)l(,)25 b(starting)c(from)g(zero)h(\(this)g (is)g(di\013eren)o(t)g(from)f(F)l(ortran)g(record)h(n)o(um)o(b)q(ering,)i (that)75 2110 y(starts)17 b(from)h(one\).)31 b(A)19 b(lo)q(cation)g(in)h(a)e (\014le)i(is)f(recorded)g(b)o(y)g(a)f Fn(\014le)k(p)q(oin)o(ter)p Ft(,)e(whic)o(h)g(stores)e(a)g(b)o(yte)75 2166 y(n)o(um)o(b)q(er,)g(for)f (stream)f(\014les,)j(and)e(a)g(record)h(n)o(um)o(b)q(er,)f(for)g(records)g (\014le.)28 b(The)18 b(\014le)g(p)q(oin)o(ter)g(is)g(alw)o(a)o(ys)75 2223 y(lo)q(cated)d(at)f(the)h(b)q(eginning)i(of)d(a)g(record)h(after)f(eac)o (h)g(\014le)i(access)f(or)f(\014le)i(p)q(ositioning)g(op)q(eration.)k(The)75 2279 y(\014le)14 b(p)q(oin)o(ter)g(is)g(at)f Fn(end-of-\014le)i Ft(if)f(it)g(p)q(oin)o(ts)g(to)e(the)i(\014rst)f(b)o(yte)g(after)g(the)h (last)f(v)m(alid)i(b)o(yte)e(in)i(a)e(stream)75 2336 y(\014le,)k(or)f(the)h (\014rst)f(record)g(after)g(the)h(last)f(v)m(alid)i(record)f(in)g(a)f(record) g(\014le)i(\(i.e.,)e(if)h(the)f(\014le)i(con)o(tains)f Fm(n)75 2392 y Ft(b)o(ytes)e(\(records\),)f(then)i(the)f(\014le)h(p)q(oin)o(ter)g(is) f(at)g(end-of-\014le)i(when)f(it)f(has)g(v)m(alue)h Fm(n)p Ft(\).)166 2534 y Fl(Discussion:)39 b Fk(F)m(ortran)15 b(assumes)g(that)h(a) 30 b Fj(SEQUENTIAL)14 b Fk(\014le)h(is)g(terminated)g(b)o(y)g(a)g (zero-length)h(end-)75 2591 y(of-\014le)f(record.)26 b(This)16 b(is)g(somewhat)f(di\013eren)o(t)i(from)d(the)j(de\014nition)e(ab)q(o)o(v)o (e.)25 b(The)16 b(di\013erence)i(seems)e(to)g(ha)o(v)o(e)75 2647 y(only)d(one)g(visible)g(consequence:)20 b(a)26 b Fj(BA)o(CKSP)m(A)o(CE) 10 b Fk(is)j(not)h(required)g(b)q(efore)g(one)f(start)h(writing)f(after)h(a)f (call)f(to)75 2704 y Fj(ENDFILE)p Fk(.)1967 46 y Fr(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 3 4 bop 75 -100 a Fp(1.3.)34 b(FILE)16 b(SHARING)g(AND)f(A)o(CCESS)g(MODES)841 b Ft(3)166 49 y(A)15 b(read)h(or)e(write)i(starts)e(at)g(the)i(curren)o(t)f (p)q(osition)h(of)f(the)h(\014le)g(p)q(oin)o(ter.)21 b(If)15 b(the)h(\014le)g(is)g(a)f(stream)75 106 y(\014le)f(then)f(the)g(\(priv)m(ate) g(or)g(shared\))g(\014le)g(p)q(oin)o(ter)h(is)f(p)q(ositioned)i(at)d(the)h (end)g(of)g(the)g(access)g(at)f(the)h(b)o(yte)75 162 y(follo)o(wing)i(the)g (last)g(accessed)g(b)o(yte.)k(If)c(the)g(\014le)h(is)f(a)f(record)h(\014le)h (then)f(the)f(\014le)i(p)q(oin)o(ter)f(is)g(p)q(ositioned)75 219 y(at)g(the)g(end)h(of)e(the)i(access)f(at)g(the)g(start)f(of)h(the)g (\014rst)g(record)g(follo)o(wing)h(the)f(last)g(accessed)h(record.)166 275 y(The)11 b(end-of-\014le)i(p)q(osition)f(is)f(mo)q(di\014ed)i(b)o(y)d(a)h (write)g(op)q(eration)g(that)g(writes)g(b)q(ey)o(ond)g(the)g(previous)75 332 y(end-of-\014le)17 b(p)q(osition,)f(th)o(us)f(increasing)i(the)e(\014le)h (size.)21 b(The)16 b(new)g(\014le)g(p)q(oin)o(ter)g(p)q(osition)g(b)q(ecomes) g(the)75 388 y(curren)o(t)k(end-of-\014le.)35 b(The)20 b(\014le)h(size)g(can) f(b)q(e)h(reduced)g(\(and)e(the)h(end-of-\014le)i(b)q(e)e(mo)o(v)o(ed)g(bac)o (k\))f(b)o(y)75 445 y(calls)f(to)e Fm(MPI)p 320 445 14 2 v 16 w(ENDFILE)p Ft(.)f(A)i(\014le)h(p)q(oin)o(ter)f(can)g(b)q(e)g(mo)o(v)o(ed) g(b)q(ey)o(ond)g(the)g(end-of-\014le)h(\(using)f(a)g(call)h(to)75 501 y Fm(MPI)p 160 501 V 16 w(SEEK)p Ft(\).)c(This)i(do)q(es)g(not)e (a\013ect)h(the)g(\014le)h(length)g(and)g(the)f(end-of-\014le)i(p)q(osition.) 166 558 y(It)e(is)g(erroneous)f(for)g(a)g(read)h(op)q(eration)g(to)f(access)g (b)o(ytes)h(that)f(w)o(ere)g(not)g(set)g(b)o(y)h(some)f(previous)75 614 y(write)h(op)q(eration.)k(E.g.,)13 b(if)i(a)g(\014le)g(is)g(extended)g(b) o(y)g(mo)o(ving)f(the)h(\014le)g(p)q(oin)o(ter)g(b)q(ey)o(ond)g(the)g (end-of-\014le,)75 671 y(then)e(the)g(\014le)g(con)o(tains)g(no)o(w)f(b)o (ytes)h(or)f(records)g(that)g(w)o(ere)g(nev)o(er)h(written;)g(these)g(need)h (to)e(b)q(e)h(written)75 727 y(b)q(efore)j(they)g(are)f(read.)21 b(Also,)16 b(one)g(cannot)f(read)h(from)f(a)g(\014xed)i(size)f(record)g(more) f(b)o(ytes)h(than)f(w)o(ere)75 784 y(written)g(b)o(y)g(the)h(last)f(write)g (op)q(eration)g(that)g(up)q(dated)h(that)e(record.)166 923 y Fl(Discussion:)32 b Fk(Implemen)o(tations)10 b(ma)o(y)g(supp)q(ort)j(in)f (some)f(cases)j(reads)f(that)g(access)h(data)e(that)g(w)o(as)g(not)75 980 y(set)17 b(b)o(y)e(a)h(previous)g(write:)22 b(In)16 b(a)f(P)o(osix)h(en)o (vironmen)o(t,)f(a)g(read)h(to)g(a)f(p)q(osition)h(that)f(w)o(as)h(not)g (written)g(b)q(efore)75 1036 y(will)d(return)j(zero)g(b)o(ytes,)f(if)f(the)h (read)h(is)e(b)q(ey)o(ond)h(the)g(end-of-\014le,)g(or)f(will)g(return)i(an)e (all)g(zero)h(record,)h(if)e(it)g(is)75 1093 y(b)q(elo)o(w)f(the)h (end-of-\014le.)j(Suc)o(h)d(read)g(in)f(V)m(esta)g(will)f(also)h(return)h (either)g(zero)h(b)o(ytes)f(or)f(an)g(all-zero)g(record,)h(but)75 1149 y(the)k(conditions)f(under)h(whic)o(h)f(either)h(o)q(ccur)h(are)f (somewhat)e(less)i(in)o(tuitiv)o(e.)27 b(Also,)18 b(implemen)o(tati)o(ons)d (ma)o(y)75 1206 y(supp)q(ort)g(zero)f(\014lling)f(of)g(\014xed)h(size)h (records)g(that)f(are)h(only)e(partially)f(up)q(dated.)75 1433 y Fu(1.3)59 b(File)20 b(sha)n(ring)g(and)g(access)g(mo)r(des)75 1535 y Ft(A)15 b(\014le)g(is)g(op)q(ened)h(for)e(parallel)i(I/O)f(b)o(y)g(a)f (group)g(of)h(pro)q(cesses)g(that)e(is)j(de\014ned)g(b)o(y)e(an)h(MPI)f Fs(c)n(ommu-)75 1592 y(nic)n(ator)p Ft(.)19 b(MPI-IO)d(supp)q(orts)e(three)g (distinct)i Fn(sharing)g(mo)q(des)e Ft(for)g(\014les.)21 b(The)14 b(\014le)h(sharing)g(mo)q(de)f(is)75 1648 y(sp)q(eci\014ed)j(when)f(the)f (\014le)i(is)e(op)q(ened.)75 1756 y Fn(Priv)m(ate:)22 b Ft(Eac)o(h)15 b(pro)q(cess)h(accesses)f(a)g(distinct)i(\014le.)75 1851 y Fn(Shared:)22 b Ft(All)i(pro)q(cesses)e(in)h(the)g(group)f(access)g(the)g (same)g(\014le.)42 b(The)23 b(pro)q(cesses)f(share)h(one)f(\014le)189 1908 y(p)q(oin)o(ter.)e(Eac)o(h)15 b(pro)q(cess)g(can)h(access)f(the)g (\014le)i(individuall)q(y)l(.)75 2003 y Fn(Collectiv)o(e:)23 b Ft(All)f(pro)q(cesses)e(in)h(the)f(group)g(access)g(the)g(same)g(\014le.)36 b(File)21 b(accesses)f(are)g(collectiv)o(e)189 2059 y(op)q(erations)15 b(that)f(in)o(v)o(olv)o(e)i(all)g(pro)q(cesses)g(in)g(the)f(group.)166 2167 y(Accesses)i(to)e(record)h(\014les)i(are)e(record-orien)o(ted.)23 b(MPI-IO)17 b(supp)q(orts)f(t)o(w)o(o)f(distinct)j(\014le)f Fn(access)75 2223 y(metho)q(ds)e Ft(for)g(record)g(\014les.)21 b(The)15 b(\014le)i(access)e(metho)q(d)g(is)h(sp)q(eci\014ed)h(when)f(the)f (\014le)h(is)g(op)q(ened.)75 2331 y Fn(Uni:)23 b Ft(Eac)o(h)15 b(\(individual)j(or)d(collectiv)o(e\))h(\014le)h(access)e(is)h(con)o(tained)g (within)g(one)f(record.)75 2426 y Fn(Multi:)23 b Ft(A)15 b(\014le)g(access)g (ma)o(y)e(span)i(m)o(ultiple)h(successiv)o(e)g(records;)e(eac)o(h)g (individual)k(item)d(in)g(the)g(I/O)189 2483 y(list)h(is)f(con)o(tained)h (within)h(one)e(distinct)h(record.)166 2591 y(MPI-IO)h(calls)g(b)q(eha)o(v)o (e)g(as)f(if)g(the)h(\014le)g(serv)o(er)f(initiates)h(one)g(I/O)f(op)q (eration)h(on)f(b)q(ehalf)h(of)f(eac)o(h)75 2647 y(individual)e(call)f(b)o(y) e(one)h(pro)q(cess)f(in)h(priv)m(ate)g(or)f(shared)h(mo)q(des,)f(and)h(one)f (I/O)h(op)q(eration)g(on)f(b)q(ehalf)h(of)75 2704 y(the)i(en)o(tire)g(group,) g(for)f(eac)o(h)h(collectiv)o(e)i(I/O)e(call)h(in)g(collectiv)o(e)g(mo)q(de.) 20 b(The)14 b(outcome)f(of)h(concurren)o(t)-32 46 y Fr(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 4 5 bop 75 -100 a Ft(4)1410 b Fp(CHAPTER)16 b(1.)34 b(IO)75 49 y Ft(accesses)16 b(to)f(the)h(same)f(\014le)i(b)o(y)f(sev)o(eral)f(pro)q (cesses)i(is)f(as)f(if)h(the)g(accesses)g(w)o(ere)f(executed)i(serially)l(,)g (in)75 106 y(some)e(unsp)q(eci\014ed)j(order)d(that)g(is)h(consisten)o(t)f (with)h(program)e(execution)j(order:)i(i.e.)i(\(individual)d(or)75 162 y(collectiv)o(e\))f(I/O)e(op)q(erations)h(are)f(atomic,)f(serializable)k (and)d(causal.)166 1091 y(The)f(same)f(\014le)h(can)g(b)q(e)g(sim)o (ultaneously)h(op)q(ened)g(b)o(y)e(di\013eren)o(t)h(comm)o(unication)g (groups.)19 b(Th)o(us,)75 1147 y(if)h(pro)q(cesses)g(access)g(a)f(\014le)h (in)h(a)e(totally)g(unco)q(ordinated)i(manner,)f(then)g(eac)o(h)g(pro)q(cess) g(can)f(op)q(en)75 1204 y(the)c(\014le)g(separately)l(.)20 b(The)15 b(prede\014ned)h(comm)o(unicator)e Fj(MPI)p 1140 1204 13 2 v 14 w(COMM)p 1284 1204 V 15 w(I)g Ft(whic)o(h)h(has)g(a)f(singleton)h (group)75 1260 y(con)o(taining)i(only)f(the)g(lo)q(cal)h(pro)q(cess,)f(can)h (b)q(e)f(used)h(b)o(y)f(a)f(pro)q(cess)i(in)f(order)g(to)f(op)q(en)i(a)f (\014le)h(with)f(no)75 1317 y(co)q(ordination)j(with)g(an)o(y)f(other)f(pro)q (cess.)30 b(\()p Fj(MPI)p 940 1317 V 14 w(COMM)p 1084 1317 V 14 w(I)18 b Ft(is)h(not)f(part)g(of)f(the)i(MPI)f(de\014nition;)j(w)o(e)75 1373 y(prop)q(ose)15 b(to)g(add)g(it.\))166 2385 y Fl(Discussion:)49 b Fk(The)18 b(main)e(raison)i(d')o(^)-20 b(etre)18 b(of)f(the)i Fj(p)o(rivate)f Fk(mo)q(de)f(is)h(for)f(allo)o(wing)f(pro)q(cesses)k(to)e(op) q(en)75 2441 y(priv)n(ate)e(\014les)h(without)g(w)o(orrying)f(ab)q(out)g(sp)q (ecifying)h(a)f(distinct)h(\014le)g(name)e(for)i(eac)o(h)g(pro)q(cess.)28 b(In)17 b(a)f(system)75 2498 y(where)j(all)e(\014les)h(are)h(in)e(a)h(shared) h(directory)m(,)f(a)g(con)o(v)o(en)o(tion)g(migh)o(t)d(b)q(e)k(used)g(whereb) o(y)g(the)f(\014le)g(op)q(ened)h(b)o(y)75 2554 y(pro)q(cess)14 b Fj(id)f Fk(is)f(named)f Fj(\014lename)p Fi(<)p Fj(id)p Fi(>)p Fk(.)17 b(In)12 b(a)g(system)g(where)h(eac)o(h)g(pro)q(cess)h(has)e(it)g(o)o (wn)g(priv)n(ate)g Fj(/tmp)d Fk(directory)75 2611 y(the)14 b(con)o(v)o(en)o(tion)g(ma)o(y)e(b)q(e)i(that)g(eac)o(h)h(pro)q(cess)h(op)q (ens)e(a)g(\014le)g Fj(\014lename)e Fk(in)h(its)h(o)o(wn)g Fj(/tmp)d Fk(directory)m(.)1967 46 y Fr(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 5 6 bop 75 -100 a Fp(1.4.)34 b(FILE)16 b(MANIPULA)l(TION)g(FUNCTIONS)893 b Ft(5)75 49 y Fu(1.4)59 b(File)20 b(manipulation)d(functions)75 198 y Fm(MPI)p 160 198 14 2 v 16 w(OPEN\(\014lename,)9 b(status,)k(action,)e (p)q(osition,)h(t)o(yp)q(e,)g(rec)p 1106 198 V 16 w(size,)f(mo)q(de,)e(metho) q(d,)g(concurrency)l(,)j(comm)m(,)75 254 y(new)o(comm)n(\))117 332 y Fk(IN)155 b Fm(\014lename)428 b Fk(name)13 b(of)g(op)q(ened)i(\014le)f (\(string\))117 404 y(IN)155 b Fm(status)476 b Fk(one)15 b(of)e Fj(MPI)p 1106 404 13 2 v 14 w(OLD)h Fk(\(\014le)h(exists\),)f Fj(MPI)p 1518 404 V 15 w(NEW)g Fk(\(\014le)g(do)q(es)h(not)905 460 y(exist\))k(,)h Fj(MPI)p 1133 460 V 14 w(REPLA)o(CE)d Fk(\(\014le)i(is)g (created)h(if)e(it)h(do)q(es)g(not)905 517 y(exist,)g(replaced)f(if)f(it)h (exists\))g(,)g Fj(MPI)p 1516 517 V 14 w(UNKNO)o(WN)g Fk(\(\014le)g(is)905 573 y(op)q(ened)d(if)e(it)h(exists,)g(created)h(if)e(it)h(do)q(es)g(not)g (exist\))905 629 y Fj(MPI)p 982 629 V 15 w(SCRA)m(TCH)8 b Fk(\(new)h(unnamed) g(\014le)g(is)g(created;)j(it)d(is)g(deleted)905 686 y(when)k(it)f(is)g (closed)h(or)g(when)g(program)e(terminates\))h(\(state\))117 758 y(IN)155 b Fm(action)472 b Fk(one)14 b(of)f Fj(MPI)p 1105 758 V 14 w(READ,)g(MPI)p 1329 758 V 14 w(WRITE)p Fk(,)f(or)i Fj(MPI)p 1624 758 V 14 w(READ)o(WRITE)905 814 y Fk(\(state\))117 886 y(IN)155 b Fm(p)q(osition)439 b Fk(one)19 b(of)e Fj(MPI)p 1114 886 V 15 w(REWIND)h Fk(\(start)h(of)e(\014le\))i(or)f Fj(MPI)p 1698 886 V 14 w(APPEND)905 943 y Fk(\(end)d(of)e(\014le\))h (\(state\))117 1015 y(IN)155 b Fm(t)o(yp)q(e)507 b Fk(one)16 b(of)e Fj(MPI)p 1108 1015 V 15 w(STREAM)p Fk(,)g Fj(MPI)p 1391 1015 V 15 w(FIXED)f Fk(\(\014xed)j(size)g(record\),)905 1071 y(or)e Fj(MPI)p 1033 1071 V 14 w(V)m(ARIABLE)e Fk(\(v)n(ariable)h(size)i (record\))g(\(state\))117 1143 y(IN)155 b Fm(rec)p 377 1143 14 2 v 17 w(size)446 b Fk(record)19 b(length)f(for)f(\014xed)h(size)h(record) g(\014le,)f(and)f(maxima)o(l)905 1200 y(record)e(length)f(for)g(v)n(ariable)e (size)j(record)g(\014le)f(\(in)o(teger\))117 1272 y(IN)155 b Fm(mo)q(de)481 b Fk(one)14 b(of)g Fj(MPI)p 1106 1272 13 2 v 14 w(PRIV)m(A)m(TE,)d(MPI)p 1387 1272 V 14 w(SHARED)905 1328 y Fk(or)j Fj(MPI)p 1033 1328 V 14 w(COLLECTIVE)f Fk(\(state\))117 1400 y(IN)155 b Fm(metho)q(d)442 b Fk(one)14 b(of)g Fj(MPI)p 1106 1400 V 14 w(UNI)f Fk(or)h Fj(MPI)p 1329 1400 V 14 w(MUL)m(TI)g Fk(\(state\))117 1472 y(IN)155 b Fm(concurrency)362 b Fk(one)10 b(of)e Fj(MPI)p 1096 1472 V 15 w(EX)o(CLUSIVE)f Fk(or)i Fj(MPI)p 1457 1472 V 14 w(CONCURRENT)f Fk(\(state\))117 1544 y(IN)155 b Fm(comm)466 b Fk(comm)o(unicator)11 b(of)j(group)f(that)h(op)q(ens)h(the)f (\014le)g(\(handle\))117 1616 y(OUT)108 b Fm(new)o(comm)393 b Fk(comm)o(unicator)13 b(asso)q(ciated)j(with)f(\014le)h(b)o(y)f(op)q(en)h (op)q(eration)905 1672 y(\(handle\))75 1797 y Fh(int)47 b(MPI)p 269 1797 15 2 v 17 w(Open\(char)23 b(*filename,)f(MPI)p 859 1797 V 17 w(IO)i(status,)e(MPI)p 1210 1797 V 17 w(IO)i(action,)f(MPI)p 1562 1797 V 17 w(IO)g(position,)393 1853 y(MPI)p 468 1853 V 17 w(IO)h(type,)f(int)g(rec)p 867 1853 V 17 w(size,)g(MPI)p 1099 1853 V 17 w(IO)g(mode,)h(MPI)p 1403 1853 V 16 w(IO)g(method,)393 1910 y(MPI)p 468 1910 V 17 w(IO)g(concurrency,)e(MPI)p 939 1910 V 17 w(Comm)h(comm,)g(MPI)p 1290 1910 V 17 w(Comm)g(*newcomm\))75 1996 y(MPI)p 150 1996 V 17 w(OPEN\()g(FILENAME,)g(STATUS,)f(ACTION,)h (POSITION,)g(TYPE,)g(REC)p 1384 1996 V 17 w(SIZE,)g(MODE,)g(METHOD,)393 2053 y(CONCURRENCY,)f(COMM,)i(NEWCOMM,)e(IERROR\))170 2109 y(CHARACTER\(*\))h(FILENAME)170 2165 y(INTEGER)g(STATUS,)g(ACTION,)g (POSITION,)g(TYPE,)g(REC)p 1200 2165 V 17 w(SIZE,)g(MODE,)g(METHOD,)170 2222 y(CONCURRENCY,)g(COMM,)g(NEWCOMM,)g(IERROR)166 2308 y Ft(Op)q(en)d(the)e(\014le)i(iden)o(ti\014ed)g(b)o(y)f(\014le)g(name)g (\(path\))f Fm(\014lename)e Ft(in)j(the)g(access)g(mo)q(de)f(sp)q(eci\014ed)j (b)o(y)75 2365 y Fm(mo)q(de)p Ft(.)c(This)d(is)g(a)g(collectiv)o(e)h(call)g (that)e(is)h(executed)g(b)o(y)g(all)g(pro)q(cesses)g(in)h(the)e(group)h(asso) q(ciated)g(with)75 2421 y Fm(comm)m Ft(;)k(all)j(should)g(pro)o(vide)f(the)f (same)h(argumen)o(t)e(v)m(alues.)34 b Fm(new)o(comm)15 b Ft(returns)k(a)g (comm)o(unicator)75 2478 y(handle)g(that)f(can)g(b)q(e)h(subsequen)o(tly)g (used)g(to)e(access)h(the)h(op)q(ened)g(\014le.)29 b(This)19 b(new)f(comm)o(unicator)75 2534 y(should)e(not)f(b)q(e)h(used)g(for)e(an)o(y) h(other)g(purp)q(ose.)166 2591 y(The)f(argumen)o(ts)f Fm(status,)i(action)g Ft(and)e Fm(p)q(osition)i Ft(ha)o(v)o(e)f(the)g(same)f(meaning)h(as)g(for)f (F)l(ortran)f(90)h([1].)75 2647 y(The)19 b Fm(p)q(osition)h Ft(argumen)o(t)f(is)g(ignored)h(if)f(the)g(\014le)h(is)g(new)f(or)g (replaced.)32 b(The)20 b Fm(rec)p 1539 2647 14 2 v 16 w(size)f Ft(and)g Fm(metho)q(d)75 2704 y Ft(argumen)o(ts)14 b(are)h(ignored)h(if)g Fm(t)o(yp)q(e)g Ft(=)f Fj(MPI)p 799 2704 13 2 v 15 w(STREAM)p Ft(.)-32 46 y Fr(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 6 7 bop 75 -100 a Ft(6)1410 b Fp(CHAPTER)16 b(1.)34 b(IO)166 49 y Ft(The)23 b Fm(\014lename)e Ft(argumen)o(t)h(is)i(alw)o(a)o(ys)e (signi\014can)o(t.)44 b(If)23 b(the)g(\014le)h(is)g(a)f(scratc)o(h)f(\014le)i (\()p Fm(status)h Ft(=)75 106 y Fj(MPI)p 152 106 13 2 v 14 w(SCRA)m(TCH)p Ft(\))19 b(then)i(the)f Fm(\014lename)e Ft(is)j(signi\014can)o (t)g(only)g(during)g(the)g(duration)f(of)g(the)g(program.)75 162 y(It)f(can)h(b)q(e)g(used)g(in)g(order)f(to)g(op)q(en)h(the)f(same)h (scratc)o(h)e(\014le)j(sev)o(eral)e(times)h(during)g(program)e(exe-)75 219 y(cution,)j(p)q(ossibly)g(with)f(di\013eren)o(t)g(parameters)f(and)g(a)h (di\013eren)o(t)g(group)f(of)g(pro)q(cesses.)34 b(Th)o(us,)20 b(the)75 275 y(\014rst)15 b(call)h(to)e Fm(MPI)p 392 275 14 2 v 16 w(OPEN)i Ft(with)f Fm(status)j Ft(=)d Fj(MPI)p 901 275 13 2 v 14 w(SCRA)m(TCH)f Ft(and)h(a)g(particular)h Fm(\014lename)d Ft(v)m(alue)j(creates)75 332 y(a)k(new)h(\014le;)k(subsequen)o(t)c(calls)h (with)f(the)g(same)f Fm(\014lename)f Ft(v)m(alue)j(op)q(en)f(the)g(existing)h (scratc)o(h)e(\014le.)75 388 y(Implemen)o(tations)c(ma)o(y)f(restrict)g(the)g (set)g(of)g(allo)o(w)o(able)h(names)f(for)f(scratc)o(h)h(\014les.)166 534 y Fl(Implemen)o(tati)o(on)g(note:)44 b Fk(A)16 b(p)q(ossible)g(implemen)o (tation)d(is)j(to)f(map)g(a)g(scratc)o(h)j(\014le)e Fj(\014lename)e Fk(in)o(to)h(a)75 590 y(Unix)e(\014le)h Fj(/tmp/\014lenam)n(e)p Fk(.)166 736 y Ft(The)20 b(corresp)q(ondence)h(b)q(et)o(w)o(een)g(the)f(v)m (alues)h(of)e(the)h Fm(MPI)p 1214 736 14 2 v 16 w(OPEN)h Ft(argumen)o(ts)e (and)h(the)g Fm(mo)q(de)75 792 y Ft(parameter)14 b(of)h(the)g(P)o(osix)h Fm(fop)q(en)g Ft(function)g(is)g(giv)o(en)f(b)o(y)g(the)h(table)f(b)q(elo)o (w)p 75 855 1927 2 v 74 911 2 57 v 100 894 a(mo)q(de)50 b(op)q(eration)545 b Fm(ST)l(A)l(TUS)230 b(A)o(CTION)180 b(POSITION)p 2000 911 V 75 913 1927 2 v 74 969 2 57 v 100 952 a(r)141 b Ft(op)q(en)16 b(for)e(reading)397 b Fm(MPI)p 1064 952 14 2 v 15 w(OLD)193 b(MPI)p 1445 952 V 16 w(READ)123 b(MPI)p 1785 952 V 16 w(REWIND)p 2000 969 2 57 v 74 1082 2 113 v 100 1009 a(w)j Ft(create)13 b(for)f(writing;)i(if)f(\014le)h(exists,)257 1065 y(o)o(v)o(erwrite)926 1009 y Fm(MPI)p 1011 1009 14 2 v 16 w(REPLA)o(CE)129 b(MPI)p 1433 1009 V 16 w(WRITE)111 b(MPI)p 1785 1009 V 16 w(REWIND)p 2000 1082 2 113 v 74 1195 V 100 1122 a(a)135 b Ft(op)q(en)16 b(for)e(writing)i(at)f(end)h(of)e(\014le;)257 1178 y(create)h(if)h(\014le)g (do)q(es)f(not)g(exist)907 1122 y Fm(MPI)p 992 1122 14 2 v 16 w(UNKNO)o(WN)109 b(MPI)p 1433 1122 V 16 w(WRITE)g(MPI)p 1783 1122 V 16 w(APPEND)p 2000 1195 2 113 v 74 1308 V 100 1235 a(r+)d Ft(up)q(date)19 b(existing)h(\014le)g(\(read)e(and)257 1291 y(write\))979 1235 y Fm(MPI)p 1064 1235 14 2 v 15 w(OLD)122 b(MPI)p 1374 1235 V 16 w(READ)o(WRITE)52 b(MPI)p 1785 1235 V 16 w(REWIND)p 2000 1308 2 113 v 74 1421 V 100 1348 a(w+)91 b Ft(create)22 b(new)g(\014le)h(for)f(up)q(date;)k(if)257 1404 y(\014le)16 b(exists,)f(o)o(v)o(erwrite)926 1348 y Fm(MPI)p 1011 1348 14 2 v 16 w(REPLA)o(CE)70 b(MPI)p 1374 1348 V 16 w(READ)o(WRITE)52 b(MPI)p 1785 1348 V 16 w(REWIND)p 2000 1421 2 113 v 74 1534 V 100 1460 a(a+)100 b Ft(op)q(en)17 b(for)e(up)q(date)h(at)f (end)i(of)e(\014le;)257 1517 y(create)g(if)h(\014le)g(do)q(es)f(not)g(exist) 907 1460 y Fm(MPI)p 992 1460 14 2 v 16 w(UNKNO)o(WN)50 b(MPI)p 1374 1460 V 16 w(READ)o(WRITE)g(MPI)p 1783 1460 V 16 w(APPEND)p 2000 1534 2 113 v 75 1535 1927 2 v 166 1659 a Ft(A)18 b(\014le)h(ma)o(y)f(b)q (e)g(op)q(ened)i(at)d(the)h(same)g(time)g(with)h(sev)o(eral)f(di\013eren)o(t) g(comm)o(unicators.)29 b(If)18 b(the)75 1716 y(\014le)13 b(is)g(op)q(ened)h (with)f Fm(concurrency)g(=)g(MPI)p 813 1716 14 2 v 16 w(CONCURRENT)h Ft(then)f(MPI-IO)g(ensures)g(that)f(concurren)o(t)75 1772 y(accesses)18 b(to)g(the)g(\014le)h(are)f(atomic,)g(serializable)j(and)d(causal:)26 b(the)18 b(outcome)g(is)h(as)e(if)i(MPI-IO)g(calls)75 1829 y(w)o(ere)f(executed)g(in)h(some)f(serial)g(order,)g(compatible)h(with)f(the) g(program)f(order.)27 b(\(This)19 b(result)f(can)75 1885 y(b)q(e)f(ac)o(hiev) o(ed,)g(e.g.,)e(b)o(y)h(serializing)i(all)f(accesses)g(to)e(the)h(\014le)i (at)d(a)h(unique)i(\014le)f(serv)o(er.\))22 b(If)16 b(the)g(\014le)h(is)75 1941 y(op)q(ened)e(with)f Fm(concurrency)g Ft(=)g Fj(MPI)p 696 1941 13 2 v 15 w(EX)o(CLUSIVE)d Ft(then)j(no)g(suc)o(h)g(concurrency)g (proto)q(col)g(is)g(used.)20 b(The)75 1998 y Fj(MPI)p 152 1998 V 14 w(EX)o(CLUSIVE)d Ft(mo)q(de)j(should)g(b)q(e)g(used)g(only)g(when)f(the) h(\014le)g(is)g(exclusiv)o(ely)h(accessed)f(with)g(one)75 2054 y(comm)o(unicator,)14 b(or)h(is)h(only)g(accessed)f(in)h(read)g(mo)q(de.)166 2117 y(When)g(a)f(\014le)h(is)g(op)q(ened)g(with)g Fm(mo)q(de)d Ft(=)j Fj(MPI)p 962 2117 V 14 w(PRIV)m(A)m(TE)d Ft(then)i Fm(n)h Ft(\014les)h(are)e(actually)h(op)q(ened,)g(one)75 2174 y(for)g(eac)o(h)h(pro) q(cess)g(in)g(the)g(group.)24 b(Tw)o(o)16 b(\014les)h(op)q(ened)h(with)f(the) g Fm(p)o(rivate)f Ft(mo)q(de)h(are)g(the)f(same)h(\014le)g(if)75 2230 y(they)e(w)o(ere)g(op)q(ened)i(with)e(the)g(same)g(name)g(and)h(b)o(y)f (the)g(same)g(pro)q(cess.)166 2293 y(Implemen)o(tations)f(ma)o(y)e(restrict)h (the)g(supp)q(orted)h(com)o(binations)f(of)g(argumen)o(ts)f(in)i Fm(MPI)p 1728 2293 14 2 v 16 w(OPEN)p Ft(.)75 2350 y(F)l(or)h(example:)143 2470 y Fg(\017)23 b Ft(The)13 b(t)o(yp)q(e)h(and)g(record)g(length)g (parameters)f(of)g(a)g(preexisting)i(\014le)g(ma)o(y)e(b)q(e)h(required)h(to) e(matc)o(h)189 2527 y(t)o(yp)q(e)i(and)g(record)g(length)h(at)f(\014le)h (creation.)143 2647 y Fg(\017)23 b Ft(Priv)m(ate)15 b(mo)q(de)g(\()p Fm(m)o(o)q(de)e Ft(=)i Fj(MPI)p 732 2647 13 2 v 14 w(PRIV)m(A)m(TE)p Ft(\))d(ma)o(y)i(b)q(e)h(supp)q(orted)g(only)h(for)e(scratc)o(h)g(\014les)i (\()e Fm(status)189 2704 y Ft(=)h Fj(SCRA)m(TCH)p Ft(\),)e(whic)o(h)j(disapp) q(ear)h(at)d(the)h(end)h(of)f(the)g(program)f(execution.)1967 46 y Fr(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 7 8 bop 75 -100 a Fp(1.4.)34 b(FILE)16 b(MANIPULA)l(TION)g(FUNCTIONS)893 b Ft(7)166 49 y Fl(Discussion:)39 b Fk(It)16 b(is)f(usually)f(feasible)h(to)g (ha)o(v)o(e)g(consisten)o(t)i(organizations)d(for)h(stream)g(\014les)g(and)h (\014xed)75 106 y(record)f(\014les;)f(v)n(ariable)f(record)i(\014les)f (require)h(additional)d(structure.)166 255 y Ft(The)j(new)h(comm)o(unicator)f (has)g(the)g(same)g(comm)o(unication)h(group)f(as)f(the)i(old)g(comm)o (unicator.)166 405 y Fl(Implemen)o(tati)o(on)h(note:)53 b Fk(The)19 b(new)g(comm)o(unicator)c(ma)o(y)h(ha)o(v)o(e)i(additional)f(cac)o(hed)i (information)75 462 y(\(suc)o(h)c(as)f(a)f(\014le)h(descriptor)h(or)f(a)g (comm)o(unicator)d(to)j(comm)o(unicate)d(with)j(the)g(\014le)g(serv)o (er\(s\)\).)166 694 y Fl(Discussion:)44 b Fk(The)17 b(requiremen)o(t)f(that)h (the)g(new)f(comm)o(unicator)e(ha)o(v)o(e)i(the)h(same)f(group)g(as)h(the)g (old)75 750 y(one)i(imp)q(oses)f(some)g(o)o(v)o(erhead)i(\(one)f(could,)h (otherwise,)g(put)f(the)h(\014le)f(serv)o(er\(s\))i(as)e(mem)o(b)q(ers)f(of)g (the)i(new)75 807 y(comm)o(unication)15 b(group\).)32 b(Ho)o(w)o(ev)o(er,)20 b(it)e(seems)h(more)f(elegan)o(t)g(that)h(the)g(ob)r(ject)h(visible)e(to)g (the)h(user)h(\(the)75 863 y(new)d(comm)o(unicator\))d(has)i(no)h(visible)e (information)f(on)i(\014le)h(serv)o(ers,)h(and)e(that)h(it)f(represen)o(ts)j (the)e(group)g(of)75 920 y(pro)q(cesses)g(that)c(are)i(in)o(v)o(olv)o(ed)d (in)i(collectiv)o(e)g(I/O)g(op)q(erations.)75 1117 y Fm(MPI)p 160 1117 14 2 v 16 w(REOPEN\()i(action,)f(p)q(osition,)h(mo)q(de,)d(metho)q (d,)g(concurrency)l(,)j(comm)m(\))117 1205 y Fk(IN)155 b Fm(action)472 b Fk(one)14 b(of)f Fj(MPI)p 1105 1205 13 2 v 14 w(READ,)g(MPI)p 1329 1205 V 14 w(WRITE)p Fk(,)f(or)i Fj(MPI)p 1624 1205 V 14 w(READ)o(WRITE)905 1261 y Fk(\(state\))117 1357 y(IN)155 b Fm(p)q(osition)439 b Fk(one)10 b(of)e Fj(MPI)p 1096 1357 V 15 w(ASIS)p Fk(,)g Fj(MPI)p 1292 1357 V 14 w(REWIND)h Fk(or)g Fj(MPI)p 1598 1357 V 15 w(APPEND)f Fk(\(state\))117 1454 y(IN)155 b Fm(mo)q(de)481 b Fk(one)119 b(of)f Fj(MPI)p 1315 1454 V 14 w(PRIV)m(A)m(TE,)e(MPI)p 1701 1454 V 14 w(SHARED)905 1510 y Fk(or)14 b Fj(MPI)p 1033 1510 V 14 w(COLLECTIVE)f Fk(\(state\))117 1607 y(IN)155 b Fm(metho)q(d)442 b Fk(one)14 b(of)g Fj(MPI)p 1106 1607 V 14 w(UNI)f Fk(or)h Fj(MPI)p 1329 1607 V 14 w(MUL)m(TI)g Fk(\(state\))117 1703 y(IN)155 b Fm(concurrency)362 b Fk(one)10 b(of)e Fj(MPI)p 1096 1703 V 15 w(EX)o(CLUSIVE)f Fk(or)i Fj(MPI)p 1457 1703 V 14 w(CONCURRENT)f Fk(\(state\))117 1799 y(INOUT)62 b Fm(comm)466 b Fk(comm)o(unicator)11 b(of)j(group)f(that)h(op)q(ens)h(the)f (\014le)g(\(handle\))75 1934 y Fh(int)47 b(MPI)p 269 1934 15 2 v 17 w(reopen\()23 b(MPI)p 549 1934 V 17 w(IO)g(action,)g(MPI)p 900 1934 V 17 w(IO)h(position,)e(MPI)p 1299 1934 V 17 w(IO)i(mode,)f(MPI)p 1603 1934 V 17 w(IO)g(method,)393 1991 y(MPI)p 468 1991 V 17 w(IO)h(concurrency,)e(MPI)p 939 1991 V 17 w(Comm)h(*comm\))75 2088 y(MPI)p 150 2088 V 17 w(REOPEN\()g(ACTION,)g(POSITION,)f(MODE,)h (METHOD,)g(CONCURRENCY,)g(COMM,)g(IERROR\))170 2144 y(INTEGER)g(ACTION,)g (POSITION,)g(MODE,)g(METHOD,)g(CONCURRENCY,)f(COMM,)h(IERROR)166 2241 y Ft(Reop)q(ens)16 b(an)f(already)h(op)q(ened)g(\014le,)g(p)q(ossibly)g (c)o(hanging)g(the)f(action,)g(p)q(osition,)h(mo)q(de,)f(metho)q(d)75 2298 y(and)21 b(concurrency)g(parameters.)36 b(This)21 b(op)q(eration)g(is)g (equiv)m(alen)o(t)h(to)e(closing)i(and)f(op)q(ening)h(a)e(\014le)75 2354 y(anew,)14 b(but)h(it)g(allo)o(ws)g(to)f(preserv)o(e)h(the)g(use)g(of)g (the)f(same)h(comm)o(unicator,)f(preserv)o(e)h(the)g(\014le)g(p)q(oin)o(ter) 75 2411 y(p)q(osition)g(\(using)27 b Fm(p)q(osition)15 b Ft(=)e Fj(MPI)p 683 2411 13 2 v 15 w(ASIS)p Ft(\))f(and)i(a)o(v)o(oid)f(sp)q (ecifying)i(anew)f(the)f(parameters)g(that)g(do)g(not)75 2467 y(c)o(hange)k(\(\014le)h(name,)f(t)o(yp)q(e)g(and)g(record)h(length\).)25 b(The)18 b Fm(metho)q(d)d Ft(argumen)o(ts)i(is)g(the)g(\014le)h(is)g(a)f (stream)75 2524 y(\014le.)166 2591 y(MPI-IO)d(do)q(es)g(not)f(pro)o(vide)h (an)g(explicit)h Fm(CLOSE)f Ft(function.)20 b(A)14 b(\014le)g(is)g(closed)h (when)f(all)g(comm)o(u-)75 2647 y(nicators)h(that)g(w)o(ere)g(asso)q(ciated)h (to)e(the)i(\014le)g(b)o(y)f(calls)i(to)e Fm(MPI)p 1181 2647 14 2 v 15 w(OPEN)i Ft(or)d Fm(MPI)p 1471 2647 V 16 w(REOPEN)j Ft(ha)o(v)o(e)e(b)q(een)75 2704 y(freed)h(\(b)o(y)e(calling)j Fm(MPI)p 497 2704 V 16 w(COMM)p 655 2704 V 16 w(FREE)p Ft(\).)-32 46 y Fr(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 8 9 bop 75 -100 a Ft(8)1410 b Fp(CHAPTER)16 b(1.)34 b(IO)75 49 y Fm(MPI)p 160 49 14 2 v 16 w(GET)p 264 49 V 17 w(FILE\(com)o(m)m(,)22 b(exist,)k(\014lename,)d(action,)j(p)q(osition,)h(t)o(yp)q(e,)f(rec)p 1378 49 V 17 w(size,)f(concurrency)l(,)i(mode,)75 106 y(metho)q(d\))117 185 y Fk(IN)155 b Fm(comm)466 b Fk(comm)o(unicator)11 b(attac)o(hed)k(to)f (\014le)f(\(handle\))117 264 y(OUT)108 b Fm(exist)502 b Fj(true)11 b Fk(if)d(\014le)i(is)f(attac)o(hed)h(to)f(comm)o(unicator,)18 b Fj(false)9 b Fk(otherwise)117 344 y(OUT)108 b Fm(\014lename)428 b Fk(name)13 b(of)g(op)q(ened)i(\014le)f(\(string\))117 423 y(OUT)108 b Fm(action)472 b Fk(one)14 b(of)f Fj(MPI)p 1105 423 13 2 v 14 w(READ,)g(MPI)p 1329 423 V 14 w(WRITE)p Fk(,)f(or)i Fj(MPI)p 1624 423 V 14 w(READ)o(WRITE)905 480 y Fk(\(state\))117 559 y(OUT)108 b Fm(p)q(osition)439 b Fk(one)11 b(of)e Fj(MPI)p 1098 559 V 15 w(REWIND)h Fk(\(start)h(of)e(\014le\),)i Fj(MPI)p 1607 559 V 14 w(APPEND)e Fk(\(end)905 615 y(of)k(\014le\))i(or)e Fj(MPI)p 1163 615 V 15 w(ASIS)g Fk(\(none)h(of)f(the)i(ab)q(o)o(v)o(e\))f (\(state\))117 695 y(OUT)108 b Fm(t)o(yp)q(e)507 b Fk(one)16 b(of)e Fj(MPI)p 1108 695 V 15 w(STREAM)p Fk(,)g Fj(MPI)p 1391 695 V 15 w(FIXED)f Fk(\(\014xed)j(size)g(record\),)905 751 y(or)e Fj(MPI)p 1033 751 V 14 w(V)m(ARIABLE)e Fk(\(v)n(ariable)h(size)i (record\))g(\(state\))117 831 y(OUT)108 b Fm(rec)p 377 831 14 2 v 17 w(size)446 b Fk(record)19 b(length)f(for)f(\014xed)h(size)h(record) g(\014le,)f(and)f(maxima)o(l)905 887 y(record)e(length)f(for)g(v)n(ariable)e (size)j(record)g(\014le)f(\(in)o(teger\))117 967 y(OUT)108 b Fm(mo)q(de)481 b Fk(one)119 b(of)f Fj(MPI)p 1315 967 13 2 v 14 w(PRIV)m(A)m(TE,)e(MPI)p 1701 967 V 14 w(SHARED)905 1023 y Fk(or)14 b Fj(MPI)p 1033 1023 V 14 w(COLLECTIVE)f Fk(\(state\))117 1102 y(OUT)108 b Fm(metho)q(d)442 b Fk(one)14 b(of)g Fj(MPI)p 1106 1102 V 14 w(UNI)f Fk(or)h Fj(MPI)p 1329 1102 V 14 w(MUL)m(TI)g Fk(\(state\))117 1182 y(OUT)108 b Fm(concurrency)362 b Fk(one)10 b(of)e Fj(MPI)p 1096 1182 V 15 w(EX)o(CLUSIVE)f Fk(or)i Fj(MPI)p 1457 1182 V 14 w(CONCURRENT)f Fk(\(state\))75 1308 y Fh(int)47 b(MPI)p 269 1308 15 2 v 17 w(Get)p 358 1308 V 17 w(file\()23 b(MPI)p 590 1308 V 17 w(Comm)g(comm,)g(int)h(*exist,)f(char)g(*filename,)393 1365 y(MPI)p 468 1365 V 17 w(IO)h(*action,)e(MPI)p 843 1365 V 17 w(IO)i(*position,)e(MPI)p 1266 1365 V 17 w(IO)i(*type,)f(int)g(*rec)p 1713 1365 V 17 w(size,)393 1421 y(MPI)p 468 1421 V 17 w(IO)h(*mode,)f(MPI)p 796 1421 V 16 w(IO)h(*method)f(,)h(MPI)p 1195 1421 V 16 w(IO)g (*concurrency\))75 1510 y(MPI)p 150 1510 V 17 w(GET)p 239 1510 V 17 w(FILE\()f(COMM,)g(EXIST,)g(FILENAME,)g(ACTION,)f(POSITION,)h(TYPE,)g (REC)p 1592 1510 V 17 w(SIZE,)g(MODE,)393 1566 y(METHOD,)g(CONCURRENCY,)f (IERROR\))170 1623 y(INTEGER)h(COMM,)h(ACTION,)e(POSITION,)h(TYPE,)g(REC)p 1152 1623 V 17 w(SIZE,)g(MODE,)g(METHOD,)170 1679 y(CONCURRENCY,)g(IERROR)170 1736 y(LOGICAL)g(EXIST)170 1792 y(CHARACTER\(*\))g(FILENAME)166 1881 y Ft(Query)d(whether)f(an)g(op)q(en)h(\014le)h(is)f(curren)o(tly)f (attac)o(hed)g(with)h(comm)o(unicator)f Fm(comm)m Ft(.)29 b(If)20 b(y)o(es,)75 1937 y(returns)14 b Fm(exist)i(=)f(true)g Ft(and)g(returns)f(in) i(the)e(remaining)i(parameters)e(information)h(on)f(the)h(\014le.)21 b(If)14 b(no,)75 1994 y Fm(exist)i(=)g(false)f Ft(is)h(returned,)f(and)g(the) g(v)m(alue)i(of)e(the)g(remaining)h(argumen)o(ts)f(is)g(unde\014ned.)166 2052 y(A)23 b(\014le)g(is)h(p)q(ositioned)g(at)e(start)g(of)g(\014le)i(\()e Fm(p)q(osition)i Ft(=)f Fj(MPI)p 1250 2052 13 2 v 14 w(REWIND)p Ft(\))f(if)i(the)e(\014le)i(p)q(oin)o(ter)f(is)75 2109 y(p)q(ositioned)18 b(at)e(the)h(\014rst)g(b)o(yte)f(in)i(a)e(stream)g(\014le)i(or)e(\014rst)g (record)h(in)h(a)e(record)h(\014le.)25 b(It)17 b(is)g(p)q(ositioned)75 2165 y(at)f(the)h(end)g(of)g(\014le)g(\()g Fm(p)q(osition)h Ft(=)f Fj(MPI)p 757 2165 V 14 w(APPEND)p Ft(\))e(if)i(it)g(is)h(at)e(the)g (\014rst)h(record)f(\(\014rst)g(b)o(yte\))h(past)f(the)75 2221 y(signi\014can)o(t)g(records)f(\(b)o(ytes\))g(in)h(the)f(\014le.)21 b(Otherwise,)15 b(the)h(p)q(osition)g(returned)g(is)f Fj(MPI)p 1611 2221 V 15 w(ASIS)p Ft(.)166 2280 y(MPI-IO)d(do)q(es)f(not)f(pro)o(vide)h (a)g(mec)o(hanism)g(for)g(allo)o(wing)g(a)g(pro)q(cess)g(to)f(inquire)i (whether)f(another)75 2337 y(pro)q(cess)k(has)h(op)q(ened)g(a)f(named)g (\014le.)166 2478 y Fl(Discussion:)32 b Fk(The)12 b(curren)o(t)h (de\014nition)e(of)g(MPI-IO)h(allo)o(ws)e(for)h(a)h(distributed)g(implemen)o (tati)o(on,)d(where)75 2534 y(only)14 b(those)i(pro)q(cesses)i(that)e(op)q (en)f(a)g(\014le)g(are)h(in)o(v)o(olv)o(ed)e(in)h(the)g(op)q(en)h(op)q (eration.)22 b(An)15 b(implemen)o(tation,)d(that)75 2591 y(uses)18 b(a)f(globally)e(kno)o(wn)i(serv)o(er)i(for)e(eac)o(h)g(\014le,)h(will)e (allo)o(w)f(to)i(supp)q(ort)h(additional)e(queries)i(\(e.g.,)f(c)o(hec)o (king)75 2647 y(whether)j(a)e(\014le)h(has)g(b)q(een)g(op)q(ened)h(b)o(y)e (another)i(pro)q(cess\),)h(and)d(will)f(allo)o(w)g(to)i(test)h(that)e(the)i Fj(concurrency)75 2704 y Fk(argumen)o(t)13 b(is)h(used)g(correctly)m(,)g(but) h(it)e(ma)o(y)f(cause)j(p)q(erformance)f(b)q(ottlenec)o(ks.)1967 46 y Fr(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 9 10 bop 75 -100 a Fp(1.5.)34 b(INDIVIDUAL)17 b(I/O)1298 b Ft(9)75 49 y Fu(1.5)59 b(Individual)19 b(I/O)75 151 y Ft(The)13 b(functions)h(in)g (this)g(section)f(apply)h(to)f(\014les)h(op)q(ened)g(in)g(the)f Fj(PRIV)m(A)m(TE)e Ft(or)h Fj(SHARED)h Ft(mo)q(des.)20 b(These)75 207 y(functions)i(are)f(executed)h(individuall)q(y)i(b)o(y)d(pro)q(cesses)h (that)e(ha)o(v)o(e)h(access)g(to)g(the)g(\014le.)39 b(They)21 b(are)75 264 y(\\lo)q(cal",)i(in)f(the)g(sense)g(that)e(their)i(completion)h (do)q(es)e(not)g(dep)q(end)i(on)f(an)o(y)f(activit)o(y)g(of)g(another)75 320 y(computation)15 b(pro)q(cess.)75 439 y Fo(1.5.1)49 b(Data)17 b(access)75 572 y Fm(MPI)p 160 572 14 2 v 16 w(READ\(bu\013er,)e(count,)h(t)o (yp)q(e,)g(comm)m(,)c(status\))117 649 y Fk(OUT)124 b Fm(bu\013er)462 b Fk(initial)12 b(address)j(of)f(bu\013er)g(where)i(data)d(is)h(put)g(\(c)o (hoice\))117 722 y(IN)171 b Fm(count)466 b Fk(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(read)i(\(in)o(teger\))117 794 y(IN)171 b Fm(t)o(yp)q(e)491 b Fk(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(elemen)o(t)f (read)i(\(handle\))117 866 y(IN)171 b Fm(comm)450 b Fk(comm)o(unicator)7 b(for)i(\014le)g(access)i(that)e(w)o(as)g(returned)i(b)o(y)e Fj(MPI)p 1865 866 13 2 v 15 w(OPEN)905 923 y Fk(\(handle\))117 995 y(OUT)124 b Fm(status)460 b Fk(return)15 b(status)g(of)e(read)i(op)q (eration)e(\(Status\))75 1119 y Fh(int)47 b(MPI)p 269 1119 15 2 v 17 w(Read\(void*)23 b(buffer,)f(int)i(count,)f(MPI)p 1074 1119 V 17 w(Datatype)g(type,)g(MPI)p 1521 1119 V 16 w(Comm)h(comm,)393 1176 y(MPI)p 468 1176 V 17 w(Status)f(*status\))75 1262 y(MPI)p 150 1262 V 17 w(READ\(BUFFER,)f(COUNT,)h(TYPE,)g(COMM,)g(STATUS,)g(IERROR\)) 170 1319 y()g(BUFFER\(*\))170 1375 y(INTEGER)g(COUNT,)g(TYPE,)h(COMM,)f (STATUS\(MPI)p 1058 1375 V 16 w(STATUS)p 1218 1375 V 16 w(SIZE\),)g(IERROR) 166 1462 y Ft(The)16 b(sequence)i(of)e(v)m(alues)h(sp)q(eci\014ed)i(b)o(y)d (the)g(t)o(yp)q(e)g(signature)h(of)f Fm(t)o(yp)q(e,)h(count)g Ft(is)g(read)f(from)g(the)75 1518 y(\014le)k(asso)q(ciated)f(with)g Fm(comm)m Ft(,)d(and)j(is)g(stored)f(in)i(the)f(comm)o(unication)g(bu\013er)g (sp)q(eci\014ed)i(b)o(y)d Fm(bu\013er,)75 1574 y(t)o(yp)q(e,)d(count)p Ft(.)21 b(Arbitrary)15 b(deriv)o(ed)g(datat)o(yp)q(es)f(can)h(b)q(e)g(used.) 20 b(This)15 b(function)h(can)e(b)q(e)i(called)g(for)e(a)g(\014le)75 1631 y(op)q(ened)i(in)g(priv)m(ate)g(or)f(shared)g(mo)q(des.)166 1687 y(Let)f Fm(t)o(yp)q(e)p 329 1687 14 2 v 17 w(size)h Ft(b)q(e)f(the)g(n)o (um)o(b)q(er)g(of)g(b)o(ytes)f(for)h(the)g(t)o(yp)q(e)g(signature)g(of)f Fm(t)o(yp)q(e)p Ft(;)i(let)f Fm(size)f Ft(=)g Fm(t)o(yp)q(e)p 1772 1687 V 18 w(size)7 b Fg(\001)75 1744 y Fm(count)p Ft(.)166 1800 y(On)15 b(a)g(stream)f(\014le,)h(the)g(call)h(causes)f Fm(size)g Ft(b)o(ytes)g(to)f(b)q(e)h(read)g(from)f(the)h(\(priv)m(ate)g(or)g (shared\))f(\014le;)75 1857 y(the)e(\(priv)m(ate)h(or)e(shared\))h(\014le)h (p)q(oin)o(ter)g(is)f(incremen)o(ted)i(b)o(y)e Fm(size)p Ft(.)19 b(If)12 b(less)h(than)e Fm(size)i Ft(b)o(ytes)f(are)f(a)o(v)m(ailable)75 1913 y(to)k(the)g(end)h(of)e(the)i(\014le,)g(then)f(an)g(end-of-\014le)i (error)e(indication)i(is)e(returned.)166 1970 y(On)21 b(a)f(record)h(\014le)h (accessed)f(with)g Fm(uni)g Ft(metho)q(d)g Fm(size)g Ft(b)o(ytes)f(will)j(b)q (e)e(read)f(from)g(the)h(curren)o(t)75 2026 y(record)14 b(and)g(the)g(\014le) h(p)q(oin)o(ter)f(is)h(incremen)o(ted)g(b)o(y)f(one.)19 b(If)14 b(less)h(than)f Fm(size)g Ft(b)o(ytes)f(are)h(a)o(v)m(ailable)i(to)d(the)75 2083 y(end)i(of)f(record)h(then)f(the)h(call)h(returns)e(an)g(end-of-record)i (error)d(indication;)k(if)e(the)f(\014le)i(p)q(oin)o(ter)f(is)g(at)75 2139 y(end-of-\014le)i(then)e(the)h(call)g(returns)f(an)g(end-of-\014le)i (error)e(indication.)166 2195 y(On)i(a)g(record)g(\014le)h(accessed)f(with)g (the)g Fm(multi)e Ft(metho)q(d)i Fm(t)o(yp)q(e)p 1230 2195 V 17 w(size)g Ft(b)o(ytes)g(will)i(b)q(e)e(accessed)h(from)75 2252 y(eac)o(h)g(of)g Fm(count)i Ft(consecutiv)o(e)f(records;)h(the)e(\014le) i(p)q(oin)o(ter)f(is)g(incremen)o(ted)g(b)o(y)g Fm(count)p Ft(.)30 b(If)19 b(an)o(y)f(of)g(the)75 2308 y(records)d(accessed)g(con)o (tains)h(less)f(than)g Fm(t)o(yp)q(e)p 862 2308 V 17 w(size)h Ft(b)o(ytes)e(then)i(an)f(end-of-record)g(error)g(indication)h(is)75 2365 y(returned;)i(if)g(less)f(than)g Fm(count)i Ft(records)e(are)g(a)o(v)m (ailable)h(to)f(the)g(end)h(of)e(the)i(\014le,)g(then)f(an)g(end-of-\014le)75 2421 y(error)d(indication)k(is)d(returned.)166 2478 y(MPI-IO)c(do)q(es)g(not) f(sp)q(ecify)i(the)e(v)m(alues)i(returned)f(b)o(y)f(a)g(call)i(to)d Fm(MPI)p 1341 2478 V 16 w(READ)i Ft(that)f(encoun)o(tered)h(an)75 2534 y(end-of-\014le)j(or)d(end-of-record)i(error;)f(nor)g(do)q(es)g(it)h(sp) q(ecify)g(the)f(lo)q(cation)h(of)f(the)g(\014le)h(p)q(oin)o(ter)g(if)f(an)g (end-)75 2591 y(of-\014le)17 b(error)e(o)q(ccurred.)22 b(The)16 b(status)f(parameter)g(can)h(b)q(e)h(queried)g(b)o(y)f(a)f(call)i(to)f Fm(MPI)p 1601 2591 V 15 w(GET)p 1704 2591 V 17 w(COUNT)75 2647 y Ft(or)f Fm(MPI)p 216 2647 V 16 w(GET)p 320 2647 V 17 w(ELEMENTS)h Ft(in)h(order)e(to)h(\014nd)g(the)g(n)o(um)o(b)q(er)h(of)e(en)o(tries)i(\(of) e(t)o(yp)q(e)h Fm(t)o(yp)q(e)p Ft(\))g(or)g(the)g(n)o(um-)75 2704 y(b)q(er)i(of)f(basic)h(elemen)o(ts)h(actually)f(read)g(\(see)f([4)o(,)h (3.13.5]\),)d(when)j(end-of-\014le)i(or)d(end-of-record)h(w)o(as)-32 46 y Fr(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 10 11 bop 75 -100 a Ft(10)1387 b Fp(CHAPTER)16 b(1.)34 b(IO)75 49 y Ft(encoun)o(tered.)d(If)19 b Fm(MPI)p 491 49 14 2 v 16 w(GET)p 595 49 V 17 w(COUNT)g Ft(\(resp.)g Fm(MPI)p 1000 49 V 15 w(GET)p 1103 49 V 17 w(ELEMENT)p Ft(\))f(returns)h Fm(n)g Ft(then)g(the)g(\014rst)g Fm(n)75 106 y Ft(en)o(tries)i(\(resp.)e(the)i(\014rst)f Fm(n)g Ft(basic)h(elemen)o(ts\))g(of)f(the)g(receiv)o(e)h(bu\013er)f(w)o(ere)g (correctly)h(up)q(dated)g(b)o(y)75 162 y(the)d(I/O)f(op)q(eration.)27 b(The)18 b(remaining)g(en)o(tries)g(\(elemen)o(ts\))g(of)f(the)g(receiv)o(e)i (bu\013er)e(ma)o(y)g(ha)o(v)o(e)g(b)q(een)75 219 y(partially)f(or)f(totally)g (mo)q(di\014ed,)h(to)q(o,)e(b)o(y)i(the)f(read.)166 276 y(The)f(error)f (handler)h(attac)o(hed)f(with)h(the)g(comm)o(unicator)f(is)h(in)o(v)o(ok)o (ed)g(if)g(an)f(error)g(o)q(ccurs)h(during)75 332 y(the)k(call.)30 b(The)19 b(standard)e(MPI)i(error)e(handler)i(considers)g(end-of-\014le)h(or) e(end-of-record)h(errors)e(on)75 389 y(read)i(to)g(b)q(e)h(non-fatal)g(and)f (has)g(no)h(e\013ect.)32 b(The)20 b(user)f(can)h(c)o(hange)f(this)h(b)q(eha)o (vior)g(b)o(y)f(attac)o(hing)75 445 y(another)c(error)f(handler.)166 502 y(The)20 b(use)f(of)g(deriv)o(ed)i(datat)o(yp)q(es)d(allo)o(ws)i (scattering)f(of)g(con)o(tiguous)h(data)e(from)h(the)h(\014le)g(in)o(to)75 559 y(non-con)o(tiguous)c(lo)q(cations)g(in)g(the)f(reader's)g(memory)l(.)75 663 y Fm(MPI)p 160 663 V 16 w(WRITE\(bu\013er,)f(count,)j(t)o(yp)q(e,)e(comm) n(,)c(status\))117 741 y Fk(OUT)124 b Fm(bu\013er)462 b Fk(initial)12 b(address)j(of)f(bu\013er)g(to)g(write)g(from)f(\(c)o(hoice\))117 817 y(IN)171 b Fm(count)466 b Fk(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f (written)i(\(in)o(teger\))117 893 y(IN)171 b Fm(t)o(yp)q(e)491 b Fk(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(elemen)o(t)f(written)i(\(handle\))117 970 y(IN)171 b Fm(comm)450 b Fk(comm)o(unicator)7 b(for)i(\014le)g(access)i (that)e(w)o(as)g(returned)i(b)o(y)e Fj(MPI)p 1865 970 13 2 v 15 w(OPEN)905 1026 y Fk(\(handle\))117 1102 y(OUT)124 b Fm(status)460 b Fk(return)15 b(status)g(of)e(write)h(op)q(eration)g(\(Status\))75 1227 y Fh(int)47 b(MPI)p 269 1227 15 2 v 17 w(Write\(void*)22 b(buffer,)h(int)h(count,)f(MPI)p 1098 1227 V 17 w(Datatype)f(type,)i(MPI)p 1545 1227 V 16 w(Comm)g(comm,)393 1284 y(MPI)p 468 1284 V 17 w(Status)f(*status\))75 1371 y(MPI)p 150 1371 V 17 w(WRITE\(BUFFER,)f(COUNT,) h(TYPE,)g(COMM,)g(STATUS,)g(IERROR\))170 1427 y()g(BUFFER\(*\))170 1484 y(INTEGER)g(COUNT,)g(TYPE,)h(COMM,)f(STATUS\(MPI)p 1058 1484 V 16 w(STATUS)p 1218 1484 V 16 w(SIZE\),)g(IERROR)166 1571 y Ft(The)f(data)f(con)o(tained)i(in)f(the)g(comm)o(unication)h(bu\013er) f(sp)q(eci\014ed)i(b)o(y)e Fm(bu\013er,)h(t)o(yp)q(e,)h(count)f Ft(is)75 1627 y(written)c(on)o(to)f(the)h(\014le)h(sp)q(eci\014ed)h(b)o(y)e Fm(comm)m Ft(.)29 b(Arbitrary)18 b(deriv)o(ed)i(datat)o(yp)q(es)f(can)g(b)q (e)h(used.)31 b(This)75 1684 y(function)16 b(can)f(b)q(e)h(called)h(for)e(a)f (\014le)j(op)q(ened)f(in)g(priv)m(ate)g(or)f(shared)g(mo)q(des.)166 1741 y(Let)f Fm(t)o(yp)q(e)p 329 1741 14 2 v 17 w(size)h Ft(b)q(e)f(the)g(n)o (um)o(b)q(er)g(of)g(b)o(ytes)f(for)h(the)g(t)o(yp)q(e)g(signature)g(of)f Fm(t)o(yp)q(e)p Ft(;)i(let)f Fm(size)f Ft(=)g Fm(t)o(yp)q(e)p 1772 1741 V 18 w(size)7 b Fg(\001)75 1797 y Fm(count)p Ft(.)166 1854 y(On)23 b(a)f(stream)f(\014le,)k Fm(size)d Ft(b)o(ytes)g(are)g(written)g (and)h(the)f(\(priv)m(ate)h(or)e(shared\))h(\014le)i(p)q(oin)o(ter)e(is)75 1911 y(incremen)o(ted)16 b(b)o(y)g Fm(size)p Ft(.)k(An)15 b(end-of-\014le)i (error)d(is)i(rep)q(orted)g(if)f(the)g(maxim)o(um)h(\014le)g(size)g(is)g (exceeded.)166 1968 y(On)f(a)f(record)g(\014le)i(accessed)f(with)f Fm(uni)h Ft(metho)q(d,)g Fm(size)f Ft(b)o(ytes)h(are)f(written)g(on)g(the)h (curren)o(t)f(record)75 2024 y(and)c(the)h(\014le)g(p)q(oin)o(ter)g(is)f (incremen)o(ted)i(b)o(y)e(one.)18 b(An)11 b(end-of-record)g(error)e(is)i(rep) q(orted)f(if)h(the)f(maxim)o(um)75 2081 y(record)k(size)h(is)g(exceeded;)h (an)e(end-of-\014le)i(error)d(is)i(rep)q(orted)f(if)h(the)g(maxim)o(um)f (\014le)h(size)g(is)g(exceeded.)166 2138 y(On)g(a)g(record)g(\014le)h (accessed)g(with)f(the)g Fm(multi)e Ft(metho)q(d,)i Fm(t)o(yp)q(e)p 1226 2138 V 17 w(size)g Ft(b)o(ytes)g(are)g(written)g(in)o(to)g Fm(count)75 2194 y Ft(successiv)o(e)j(records,)f(and)g(the)g(\014le)h(p)q (oin)o(ter)f(is)h(incremen)o(ted)g(b)o(y)e Fm(count)p Ft(.)27 b(An)17 b(end-of-record)g(error)f(is)75 2251 y(rep)q(orted)g(if)f(the)h (maxim)o(um)f(record)g(size)i(is)f(exceeded;)g(and)g(an)f(end-of-\014le)i (error)e(is)h(rep)q(orted)f(if)h(the)75 2307 y(maxim)o(um)f(\014le)h(size)g (is)g(exceeded.)166 2364 y(If)i(a)f(v)m(ariable)i(size)g(record)e(is)h (written,)g(then)g(the)g(record)f(size)i(is)f(adjusted)g(to)f(the)g(size)i (of)e(the)75 2421 y(data)f(written.)23 b(If)17 b(a)f(\014xed)h(size)g(record) f(is)h(written)f(and)h(only)g(partially)g(\014lled,)h(then)f(the)f(remainder) 75 2477 y(of)f(the)g(record)g(is)h(unde\014ned.)166 2534 y(MPI-IO)h(do)q(es)f (not)f(sp)q(ecify)j(whic)o(h)e(v)m(alues)h(are)f(stored)g(on)f(\014le)i(b)o (y)f(a)g(call)h(to)e Fm(MPI)p 1620 2534 V 16 w(WRITE)h Ft(that)75 2591 y(encoun)o(tered)g(an)f(end-of-\014le)h(or)f(end-of-record)g(error;)f (nor)h(do)q(es)g(it)g(sp)q(ecify)i(the)e(lo)q(cation)g(of)g(the)g(\014le)75 2647 y(p)q(oin)o(ter)j(if)g(an)f(end-of-\014le)i(error)e(o)q(ccurred.)27 b(The)17 b(status)g(parameter)g(can)g(b)q(e)h(queried)h(b)o(y)e(a)g(call)i (to)75 2704 y Fm(MPI)p 160 2704 V 16 w(GET)p 264 2704 V 17 w(COUNT)d Ft(or)f Fm(MPI)p 591 2704 V 15 w(GET)p 694 2704 V 17 w(ELEMENTS)h Ft(in)g(order)f(to)g(\014nd)h(the)f(n)o(um)o(b)q(er)h(of)f (en)o(tries)g(\(of)g(t)o(yp)q(e)1967 46 y Fr(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 11 12 bop 75 -100 a Fp(1.5.)34 b(INDIVIDUAL)17 b(I/O)1276 b Ft(11)75 49 y Fm(t)o(yp)q(e)p Ft(\))16 b(or)f(the)h(n)o(um)o(b)q(er)g(of)f(basic)h (elemen)o(ts)g(actually)g(written)g(\(see)f([4,)g(3.13.5]\),)d(when)k (end-of-\014le)h(or)75 106 y(end-of-record)e(w)o(as)e(encoun)o(tered.)20 b(If)15 b Fm(MPI)p 838 106 14 2 v 16 w(GET)p 942 106 V 16 w(COUNT)g Ft(\(resp.)f Fm(MPI)p 1337 106 V 16 w(GET)p 1441 106 V 17 w(ELEMENT)p Ft(\))f(returns)i Fm(n)75 162 y Ft(then)j(the)g(\014rst)g Fm(n)g Ft(en)o(tries)g(\(resp.)g(the)g(\014rst)f Fm(n)i Ft(basic)f(elemen)o(ts\))g (of)g(the)g(send)g(bu\013er)g(w)o(ere)g(correctly)75 219 y(written)g(on)g (\014le)h(b)o(y)g(the)f(I/O)g(op)q(eration.)29 b(F)l(urther)18 b(data)g(ma)o(y)f(ha)o(v)o(e)h(b)q(een)h(\(partially\))g(written)f(on)75 275 y(the)d(\014le.)166 332 y(The)f(error)f(handler)h(attac)o(hed)f(with)h (the)g(comm)o(unicator)f(is)h(in)o(v)o(ok)o(ed)g(if)g(an)f(error)g(o)q(ccurs) h(during)75 388 y(the)k(call.)30 b(The)19 b(standard)e(MPI)i(error)e(handler) i(considers)g(end-of-\014le)h(or)e(end-of-record)h(errors)e(on)75 444 y(write)f(to)f(b)q(e)h(fatal)f(and)h(ab)q(orts)f(the)h(program.)k(The)c (user)g(can)g(c)o(hange)f(this)i(b)q(eha)o(vior)f(b)o(y)f(attac)o(hing)75 501 y(another)g(error)f(handler.)166 557 y(The)k(use)g(of)f(deriv)o(ed)i (datat)o(yp)q(es)e(allo)o(ws)g(gathering)h(of)f(data)g(from)g(non-con)o (tiguous)h(lo)q(cations)75 614 y(in)e(the)f(reader's)g(memory)g(and)g(write)g (them)g(on)o(to)g(a)g(con)o(tiguous)g(blo)q(c)o(k)h(on)f(the)g(\014le.)166 670 y(If)g(a)f(\014le)i(is)f(op)q(en)g(b)q(oth)g(for)f(reads)h(and)f(writes)h (\()p Fm(action)g(=)g(MPI)p 1274 670 V 16 w(READ)o(WRITE)p Ft(\))f(then)h(reads)g(and)75 727 y(writes)f(can)h(b)q(e)f(mixed)h(in)g(an)g (arbitrary)e(manner)h({)g(there)g(is)h(no)f(need)h(to)f(rep)q(osition)h(the)f (\014le)h(p)q(oin)o(ter)75 783 y(b)q(et)o(w)o(een)h(reads)f(and)g(writes.)75 887 y Fm(MPI)p 160 887 V 16 w(IREAD\(bu\013er,)g(count,)h(t)o(yp)q(e,)g(comm) m(,)c(request\))117 964 y Fk(OUT)124 b Fm(bu\013er)462 b Fk(initial)12 b(address)j(of)f(bu\013er)g(where)i(data)d(is)h(put)g(\(c)o(hoice\))117 1039 y(IN)171 b Fm(count)466 b Fk(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f (read)i(\(in)o(teger\))117 1114 y(IN)171 b Fm(t)o(yp)q(e)491 b Fk(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(elemen)o(t)f(read)i(\(handle\))117 1189 y(IN)171 b Fm(comm)450 b Fk(comm)o(unicator)11 b(for)j(\014le)g(access)h (\(handle\))117 1263 y(OUT)124 b Fm(request)436 b Fk(comm)o(unication)11 b(request)k(\(handle\))75 1388 y Fh(int)47 b(MPI)p 269 1388 15 2 v 17 w(Iread\(void*)22 b(buffer,)h(int)h(count,)f(MPI)p 1098 1388 V 17 w(Datatype)f(type,)i(MPI)p 1545 1388 V 16 w(Comm)g(comm,)393 1444 y(MPI)p 468 1444 V 17 w(Comm)p 581 1444 V 17 w(request)e(*request\))75 1531 y(MPI)p 150 1531 V 17 w(IREAD\(BUFFER,)g(COUNT,)h(TYPE,)g(COMM,)g (REQUEST,)g(IERROR\))170 1587 y()g(BUFFER\(*\))170 1644 y(INTEGER)g(COUNT,)g(TYPE,)h(COMM,)f(REQUEST,)g(IERROR)166 1730 y Ft(Initiates)e(a)f(non)o(blo)q(c)o(king)i(\(async)o(hronous\))e(read)g (op)q(eration.)36 b(The)20 b(op)q(eration)h(is)g(completed)75 1787 y(in)d(the)g(same)f(manner)g(as)g(a)g(non)o(blo)q(c)o(king)i(message)e (passing)g(op)q(eration,)h(b)o(y)f(a)g(call)i(to)e Fm(MPI)p 1737 1787 14 2 v 15 w(W)l(AIT)p Ft(,)75 1843 y Fm(MPI)p 160 1843 V 16 w(TEST)p Ft(,)h(or)f(one)h(of)g(the)g(deriv)o(ed)h(functions.)29 b(Calls)18 b(to)g Fm(MPI)p 1242 1843 V 16 w(W)l(AIT)l(ALL)p Ft(,)f Fm(MPI)p 1559 1843 V 16 w(W)l(AIT)l(ANY)p Ft(,)h(etc.)75 1900 y(can)d(b)q(e)h(used)g(to)e(w)o(ait)h(for)f(the)h(completion)i(of)d(a)h (set)g(of)f(I/O)i(op)q(erations)f(\(or,)f(more)h(generally)l(,)h(for)e(a)75 1956 y(mixed)i(set)f(of)g(comm)o(unication)h(and)f(I/O)h(op)q(erations\).)75 2060 y Fm(MPI)p 160 2060 V 16 w(IWRITE\(bu\013er,)e(count,)i(t)o(yp)q(e,)g (comm)m(,)c(request\))117 2137 y Fk(OUT)124 b Fm(bu\013er)462 b Fk(initial)12 b(address)j(of)f(bu\013er)g(to)g(write)g(from)f(\(c)o (hoice\))117 2212 y(IN)171 b Fm(count)466 b Fk(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(written)i(\(in)o(teger\))117 2287 y(IN)171 b Fm(t)o(yp)q(e)491 b Fk(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(elemen)o(t)f (written)i(\(handle\))117 2361 y(IN)171 b Fm(comm)450 b Fk(comm)o(unicator)11 b(for)j(\014le)g(access)h(\(handle\))117 2436 y(OUT)124 b Fm(request)436 b Fk(comm)o(unication)11 b(request)k(\(handle\))75 2561 y Fh(int)47 b(MPI)p 269 2561 15 2 v 17 w(Iwrite\(void*)22 b(buffer,)h(int)h(count,)f(MPI) p 1122 2561 V 16 w(Datatype)g(type,)g(MPI)p 1568 2561 V 17 w(Comm)h(comm,)393 2617 y(MPI)p 468 2617 V 17 w(Comm)p 581 2617 V 17 w(request)e(*request\))75 2704 y(MPI)p 150 2704 V 17 w(IWRITE\(BUFFER,)g(COUNT,)h(TYPE,)g(COMM,)g(REQUEST,)g(IERROR\))-32 46 y Fr(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 12 13 bop 75 -100 a Ft(12)1387 b Fp(CHAPTER)16 b(1.)34 b(IO)170 49 y Fh()23 b(BUFFER\(*\))170 106 y(INTEGER)g(COUNT,)g(TYPE,)h(COMM,)f (REQUEST,)g(IERROR)166 193 y Ft(Initiates)16 b(a)f(non)o(blo)q(c)o(king)i (\(async)o(hronous\))d(write)h(op)q(eration.)75 319 y Fo(1.5.2)49 b(File)17 b(p)q(ositioning)75 406 y Ft(The)d(\014le)g(p)q(ositioning)i (functions)e(allo)o(w)g(to)f(rep)q(osition)h(the)g(\014le)h(p)q(oin)o(ter)f (at)e(a)i(particular)g(place)g(in)h(the)75 462 y(\014le)d(with)f(the)g (function)g Fm(MPI)p 575 462 14 2 v 16 w(SEEK)p Ft(,)f(at)h(the)f(b)q (eginning)j(of)d(the)h(\014le)h(with)f(the)g(function)g Fm(MPI)p 1672 462 V 16 w(REWIND)p Ft(,)75 519 y(or)k(one)h(lo)q(cation)h(bac)o(k)e (with)i(the)e(function)i Fm(MPI)p 942 519 V 16 w(BA)o(CKSP)l(A)o(CE)p Ft(.)f(In)g(addition,)h(the)f(user)g(ma)o(y)f(query)75 575 y(the)j(\014le)i(p)q(oin)o(ter)f(lo)q(cation)g(with)f(a)g(call)i(to)d Fm(MPI)p 946 575 V 16 w(GETPOS)p Ft(,)i(and)g(truncate)f(the)g(\014le)i(with) e(a)g(call)i(to)75 632 y Fm(MPI)p 160 632 V 16 w(ENDFILE)p Ft(.)166 689 y(The)f(\014le)h(p)q(oin)o(ter)g(p)q(osition)g(is)g(a)f(b)o(yte) g(n)o(um)o(b)q(er,)h(for)e(stream)g(\014les,)j(or)e(a)f(record)i(n)o(um)o(b)q (er,)g(for)75 745 y(record)e(\014les.)29 b(The)19 b(p)q(osition)g(is)f(a)g(v) m(ariable)h(of)f(t)o(yp)q(e)34 b Fj(INTEGER)18 b Ft(in)h(F)l(ortran,)e(and)h (of)g(in)o(tegral)g(t)o(yp)q(e)75 802 y Fj(MPI)p 152 802 13 2 v 14 w(Fint)d Ft(in)h(C.)189 912 y Fs(R)n(ationale.)38 b Ft(The)14 b(use)f(of)g(t)o(yp)q(e)25 b Fj(MPI)p 836 912 V 14 w(Fint)14 b Ft(allo)o(w)f(implemen)o(tors)h(to)e(pro)o(vide)i(64)f(bit)g (addressing)189 968 y(in)o(to)i(\014les)h(in)g(a)f(transparen)o(t)f(manner.) 20 b(\()p Fs(End)c(of)g(r)n(ationale.)p Ft(\))75 1125 y Fm(MPI)p 160 1125 14 2 v 16 w(SEEK\(comm)m(,)c(o\013set,)j(whence\))117 1203 y Fk(IN)155 b Fm(comm)466 b Fk(comm)o(unicator)11 b(asso)q(ciated)k (with)f(\014le)f(\(handle\))117 1280 y(IN)155 b Fm(o\013set)484 b Fk(o\013set)15 b(in)o(to)e(\014le)h(\(in)o(teger\))117 1356 y(IN)155 b Fm(whence)450 b Fk(one)20 b(of)f Fj(MPI)p 1117 1356 13 2 v 15 w(SET)g Fk(\(indicating)g(start)i(of)e(\014le\),)i Fj(MPI)p 1779 1356 V 14 w(END)905 1413 y Fk(\(indicating)16 b(end)i(of)e(\014le\),)h(or)g Fj(MPI)p 1479 1413 V 15 w(CUR)e Fk(\(indicating)h(cur-)905 1469 y(ren)o(t)f(p)q(osition\))e(\(state\))p Ft(.)75 1594 y Fh(int)47 b(MPI)p 269 1594 15 2 v 17 w(Seek\(MPI)p 478 1594 V 16 w(Comm)23 b(comm,)h(MPI)p 829 1594 V 16 w(Fint)g(offset,)f(MPI) p 1228 1594 V 16 w(IO)h(whence\))75 1681 y(MPI)p 150 1681 V 17 w(SEEK\(COMM,)e(OFFSET,)h(WHENCE,)g(IERROR\))170 1738 y(INTEGER)g(COMM,)h (OFFSET,)e(WHENCE,)h(IERROR)166 1825 y Ft(Mo)o(v)o(es)15 b(the)h(\(priv)m (ate)h(or)f(shared\))g(\014le)h(p)q(oin)o(ter)g(to)e(the)i(p)q(osition)g (determined)h(b)o(y)e(adding)h Fm(o\013set)75 1882 y Ft(to)e(the)g(p)q (osition)h(indicated)h(b)o(y)e Fm(whence)p Ft(.)166 1939 y(The)20 b(\014le)i(p)q(oin)o(ter)e(cannot)g(b)q(e)h(set)f(to)f(b)q(e)i(negativ)o(e.) 35 b(The)20 b(\014le)i(p)q(oin)o(ter)e(can)h(b)q(e)f(rep)q(ositioned)75 1995 y(b)q(ey)o(ond)c(the)f(end-of-\014le,)i(e.g.)i(using)d(a)f(p)q(ositiv)o (e)h(displacemen)o(t)h(with)e Fm(whence)i Ft(=)f Fj(MPI)p 1605 1995 13 2 v 14 w(END)p Ft(.)166 2052 y(MPI-IO)g(do)q(es)g(not)e(require)i Fm(MPI)p 758 2052 14 2 v 16 w(SEEK)f Ft(to)g(b)q(e)h(supp)q(orted)g(for)e(v)m (ariable)j(size)f(record)f(\014les.)75 2157 y Fm(MPI)p 160 2157 V 16 w(REWIND\(comm)m(\))117 2235 y Fk(IN)155 b Fm(comm)466 b Fk(comm)o(unicator)11 b(asso)q(ciated)k(with)f(\014le)f(\(handle\))75 2360 y Fh(int)47 b(MPI)p 269 2360 15 2 v 17 w(Rewind\(MPI)p 526 2360 V 16 w(Comm)23 b(comm\))75 2447 y(MPI)p 150 2447 V 17 w(REWIND\(COMM,)f(IERROR\))170 2504 y(INTEGER)h(COMM,)h(IERROR)166 2591 y Ft(Mo)o(v)o(es)9 b(the)h(\014le)h(p)q(oin)o(ter)g(to)f(the)g(b)q (eginning)i(of)e(the)g(\014le.)19 b(This)11 b(is)g(equiv)m(alen)o(t)h(to)d Fm(MPI)p 1615 2591 14 2 v 16 w(SEEK\(comm)m(,)75 2647 y(0,)18 b(MPI)p 214 2647 V 16 w(SET\))p Ft(.)g(\()p Fm(MPI)p 462 2647 V 16 w(REWIND)g Ft(is)h(supp)q(orted)g(for)f(all)h(\014le)g(t)o(yp)q(es,)g (including)i(v)m(ariable)f(record)e(size)75 2704 y(\014les.\))j(The)15 b(call)h(has)f(no)g(e\013ect)g(if)h(the)f(\014le)i(is)e(already)h(p)q (ositioned)g(at)f(its)g(start.)1967 46 y Fr(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 13 14 bop 75 -100 a Fp(1.6.)34 b(COLLECTIVE)16 b(A)o(CCESSES)1095 b Ft(13)75 49 y Fm(MPI)p 160 49 14 2 v 16 w(BA)o(CKSP)l(A)o(CE\(comm)m(\))117 147 y Fk(IN)155 b Fm(comm)466 b Fk(comm)o(unicator)11 b(asso)q(ciated)k(with) f(\014le)f(\(handle\))75 291 y Fh(int)47 b(MPI)p 269 291 15 2 v 17 w(Backspace\(MPI)p 598 291 V 15 w(Comm)24 b(comm\))75 398 y(MPI)p 150 398 V 17 w(BACKSPACE\(COMM,)e(IERROR\))170 454 y(INTEGER)h(COMM,)h(IERROR)166 561 y Ft(Mo)o(v)o(es)17 b(the)h(\014le)h(p)q(oin)o(ter)g(one)f(lo)q(cation)h(bac)o(k.)28 b(This)19 b(is)f(equiv)m(alen)o(t)i(to)d Fm(MPI)p 1536 561 14 2 v 16 w(SEEK\(comm)m(,)e(-1,)75 617 y(MPI)p 160 617 V 16 w(CUR\))p Ft(.)i(\()p Fm(MPI)p 413 617 V 15 w(BA)o(CKSP)l(A)o(CE)h Ft(is)f(supp)q(orted)h(for)e(all)i(\014le)g(t)o(yp)q(es.\))26 b(The)17 b(call)h(is)f(erroneous)g(if)h(the)75 674 y(\014le)e(is)g(p)q (ositioned)h(at)d(its)i(start.)75 797 y Fm(MPI)p 160 797 V 16 w(GETPOS\(comm)n(,)c(p)q(os\))117 895 y Fk(IN)155 b Fm(comm)466 b Fk(comm)o(unicator)11 b(asso)q(ciated)k(with)f(\014le)f(\(handle\))117 1010 y(OUT)108 b Fm(p)q(os)523 b Fk(\014le)14 b(p)q(oin)o(ter)g(p)q(osition) 75 1154 y Fh(int)47 b(MPI)p 269 1154 15 2 v 17 w(Getpos\(MPI)p 526 1154 V 16 w(Comm)23 b(comm,)g(MPI)p 876 1154 V 17 w(Fint)g(*pos\))75 1261 y(MPI)p 150 1261 V 17 w(GETPOS\(COMM,)f(POS,)h(IERROR\))170 1317 y(INTEGER)g(COMM,)h(POS,)f(IERROR)166 1424 y Ft(Returns)16 b(in)g Fm(p)q(os)g Ft(the)f(curren)o(t)g(p)q(osition)h(of)f(the)g(\(priv)m (ate)h(or)f(shared\))g(\014le)h(p)q(oin)o(ter.)166 1501 y(MPI-IO)g(do)q(es)g (not)e(require)i Fm(MPI)p 758 1501 14 2 v 16 w(GETPOS)g Ft(to)f(b)q(e)h(supp) q(orted)g(for)e(v)m(ariable)j(size)f(record)f(\014les.)75 1624 y Fm(MPI)p 160 1624 V 16 w(ENDFILE\(com)o(m)m(\))117 1722 y Fk(IN)155 b Fm(comm)466 b Fk(comm)o(unicator)11 b(asso)q(ciated)k(with)f (\014le)75 1866 y Fh(int)47 b(MPI)p 269 1866 15 2 v 17 w(Endfile\(MPI)p 550 1866 V 16 w(Comm)23 b(comm\))75 1973 y(MPI)p 150 1973 V 17 w(ENDFILE\(COMM,)f(IERROR\))170 2029 y(INTEGER)h(COMM,)h(IERROR)166 2136 y Ft(T)l(runcate)18 b(the)h(\014le)g(at)f(the)g(curren)o(t)g(p)q (osition.)31 b(I.e.)e(if)19 b(the)f(curren)o(t)g(v)m(alue)i(of)e(the)g (\014le)h(p)q(oin)o(ter)75 2192 y(is)g Fm(n)g Ft(then)g(the)f(\014le)h(is)g (left)g(with)g Fm(n)g Ft(b)o(ytes)f(\(records\),)g(and)h(the)f(\014le)h(p)q (oin)o(ter)g(p)q(osition)h(b)q(ecomes)f(the)75 2249 y(end-of-\014le.)75 2507 y Fu(1.6)59 b(Collective)20 b(accesses)75 2647 y Ft(The)11 b(functions)g(in)g(this)g(section)g(apply)h(to)e(\014les)h(op)q(ened)h(in)20 b Fj(COLLECTIVE)10 b Ft(mo)q(de.)18 b(These)11 b(are)f(collectiv)o(e)75 2704 y(calls)16 b(executed)g(b)o(y)g(all)g(pro)q(cesses)f(in)h(the)g(group.) -32 46 y Fr(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 14 15 bop 75 -100 a Ft(14)1387 b Fp(CHAPTER)16 b(1.)34 b(IO)75 49 y Fo(1.6.1)49 b(Data)17 b(access)75 198 y Fm(MPI)p 160 198 14 2 v 16 w(READ)p 295 198 V 17 w(ALL\(bu\013er,)d(count,)i(t)o(yp)q(e,)g (comm)m(,)c(status\))117 284 y Fk(OUT)124 b Fm(bu\013er)462 b Fk(initial)12 b(address)j(of)f(bu\013er)g(where)i(data)d(is)h(put)g(\(c)o (hoice\))117 375 y(IN)171 b Fm(count)466 b Fk(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(read)i(\(in)o(teger\))117 467 y(IN)171 b Fm(t)o(yp)q(e)491 b Fk(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(elemen)o(t)f (read)i(\(handle\))117 558 y(IN)171 b Fm(comm)450 b Fk(comm)o(unicator)11 b(asso)q(ciated)k(with)f(\014le)f(\(handle\))117 650 y(OUT)124 b Fm(status)460 b Fk(return)15 b(status)g(of)e(read)i(op)q(eration)e (\(Status\))75 782 y Fh(int)47 b(MPI)p 269 782 15 2 v 17 w(Read)p 382 782 V 17 w(all\()23 b(void*)g(buffer,)g(int)h(count,)f(MPI)p 1187 782 V 16 w(Datatype)g(type,)393 839 y(MPI)p 468 839 V 17 w(Comm)g(comm,)g(MPI)p 819 839 V 17 w(Status)g(*status\))75 933 y(MPI)p 150 933 V 17 w(READ)p 263 933 V 16 w(ALL\()h(BUFFER,)f(COUNT,)g (TYPE,)g(COMM,)g(STATUS,)g(IERROR\))170 990 y()g(BUFFER\(*\))170 1046 y(INTEGER)g(COUNT,)g(TYPE,)h(COMM,)f(STATUS\(MPI)p 1058 1046 V 16 w(STATUS)p 1218 1046 V 16 w(SIZE\),)g(IERROR)166 1141 y Ft(This)14 b(function)g(b)q(eha)o(v)o(es)f(lik)o(e)h Fm(MPI)p 777 1141 14 2 v 16 w(READ)p Ft(,)g(except)f(that)g(the)g(data)f (read)i(from)e(the)h(\014le)h(is)g(broad-)75 1197 y(cast)21 b(to)h(all)h(pro)q(cesses)f(in)h(the)f(group)f(of)h Fm(comm)m Ft(;)g(eac)o(h)g(pro)q(cess)g(receiv)o(es)h(the)f(same)g(data)f(in)i(its)75 1254 y(comm)o(unication)17 b(bu\013er)g(that)f(is)h(sp)q(eci\014ed)i(b)o(y)e (the)g Fm(bu\013er,)f(t)o(yp)q(e,)i(count)g Ft(argumen)o(ts.)24 b(All)18 b(pro)q(cesses)75 1310 y(supply)d(consisten)o(t)e(v)m(alues)h(for)f Fm(t)o(yp)q(e,)h(count)h Ft(\(i.e.,)e(the)g(t)o(yp)q(e)h(signature)f(sp)q (eci\014ed)j(b)o(y)d(these)g(t)o(w)o(o)f(argu-)75 1367 y(men)o(ts)k(should)g (b)q(e)h(iden)o(tical)h(at)d(all)i(no)q(des\).)22 b(The)16 b Fm(status)i Ft(argumen)o(t)d(can)h(b)q(e)h(queried)g(after)e(the)h(call)75 1423 y(to)f(\014nd)g(out)g(the)h(n)o(um)o(b)q(er)f(of)g(elemen)o(ts)h (actually)g(read.)75 1535 y Fm(MPI)p 160 1535 V 16 w(WRITE)p 319 1535 V 16 w(ALL\(bu\013er,)e(count,)i(t)o(yp)q(e,)g(comm)m(,)c(status\)) 117 1620 y Fk(IN)171 b Fm(bu\013er)462 b Fk(initial)12 b(address)j(of)f (bu\013er)g(where)i(data)d(is)h(tak)o(en)g(\(c)o(hoice\))117 1712 y(IN)171 b Fm(count)466 b Fk(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f (written)i(\(in)o(teger\))117 1803 y(IN)171 b Fm(t)o(yp)q(e)491 b Fk(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(elemen)o(t)f(written)i(\(handle\))117 1895 y(IN)171 b Fm(comm)450 b Fk(comm)o(unicator)11 b(asso)q(ciated)k(with)f (\014le)f(\(handle\))117 1986 y(OUT)124 b Fm(status)460 b Fk(return)15 b(status)g(of)e(write)h(op)q(eration)g(\(Status\))75 2119 y Fh(int)47 b(MPI)p 269 2119 15 2 v 17 w(Write)p 406 2119 V 17 w(all\()23 b(void*)g(buffer,)g(int)g(count,)g(MPI)p 1210 2119 V 17 w(Datatype)g(type,)393 2176 y(MPI)p 468 2176 V 17 w(Comm)g(comm,)g(MPI)p 819 2176 V 17 w(Status)g(*status\))75 2270 y(MPI)p 150 2270 V 17 w(WRITE)p 287 2270 V 16 w(ALL\()h(BUFFER,)e(COUNT,)i(TYPE,)f(COMM,)g (STATUS,)g(IERROR\))170 2327 y()g(BUFFER\(*\))170 2383 y(INTEGER)g(COUNT,)g(TYPE,)h(COMM,)f(STATUS\(MPI)p 1058 2383 V 16 w(STATUS)p 1218 2383 V 16 w(SIZE\),)g(IERROR)166 2478 y Ft(This)18 b(function)g(b)q(eha)o(v)o(es)f(lik)o(e)h Fm(MPI)p 793 2478 14 2 v 16 w(WRITE)p Ft(,)f(except)g(that)g(it)g(is)h(executed)g(b)o (y)f(all)h(pro)q(cesses)f(in)75 2534 y(the)f(group;)f(all)i(pro)q(cesses)f (pro)o(vide)g(the)g(same)g(data.)21 b(The)16 b(pro)q(cesses)g(supply)h (consisten)o(t)f(v)m(alues)h(for)75 2591 y Fm(t)o(yp)q(e,)j(count)g Ft(\(i.e.,)e(the)h(t)o(yp)q(e)f(signature)h(sp)q(eci\014ed)i(b)o(y)d(these)h (t)o(w)o(o)e(argumen)o(ts)g(should)j(b)q(e)f(iden)o(tical)75 2647 y(at)c(all)i(no)q(des\).)22 b(The)16 b Fm(status)h Ft(argumen)o(t)e(can) h(b)q(e)h(queried)g(after)e(the)h(call)g(to)g(\014nd)g(out)f(the)h(n)o(um)o (b)q(er)g(of)75 2704 y(elemen)o(ts)g(actually)g(written.)1967 46 y Fr(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 15 16 bop 75 -100 a Fp(1.6.)34 b(COLLECTIVE)16 b(A)o(CCESSES)1095 b Ft(15)75 49 y Fm(MPI)p 160 49 14 2 v 16 w(READ)p 295 49 V 17 w(SCA)l(TTER\()16 b(bu\013er,)f(count,)h(t)o(yp)q(e,)g(comm)m(,)c (status\))117 126 y Fk(OUT)124 b Fm(bu\013er)462 b Fk(initial)12 b(address)j(of)f(bu\013er)g(where)i(data)d(is)h(put)g(\(c)o(hoice\))117 200 y(IN)171 b Fm(count)466 b Fk(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f (read)i(\(in)o(teger\))117 274 y(IN)171 b Fm(t)o(yp)q(e)491 b Fk(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(elemen)o(t)f(read)i(\(handle\))117 348 y(IN)171 b Fm(comm)450 b Fk(comm)o(unicator)11 b(asso)q(ciated)k(with)f (\014le)f(\(handle\))117 421 y(OUT)124 b Fm(status)460 b Fk(return)15 b(status)g(of)e(read)i(op)q(eration)e(\(Status\))75 546 y Fh(int)47 b(MPI)p 269 546 15 2 v 17 w(read)p 382 546 V 17 w(scatter\()22 b(void*)i(buffer,)f(int)g(count,)g(MPI)p 1282 546 V 17 w(Datatype)g(type,)393 602 y(MPI)p 468 602 V 17 w(Comm)g(comm,)g(MPI)p 819 602 V 17 w(Status)g(*status\))75 689 y(MPI)p 150 689 V 17 w(READ)p 263 689 V 16 w(SCATTER\()g(BUFFER,)g(COUNT,)g(TYPE,)g(COMM,)g(STATUS,)g(IERROR\)) 170 745 y()g(BUFFER\(*\))170 801 y(INTEGER)g(COUNT,)g(TYPE,)h(COMM,)f (STATUS\(MPI)p 1058 801 V 16 w(STATUS)p 1218 801 V 16 w(SIZE\),)g(IERROR)166 888 y Ft(This)14 b(is)g(a)f(collectiv)o(e)j(read)d(function)h(that)f(is)h(in) o(v)o(ok)o(ed)g(b)o(y)f(all)i(pro)q(cesses)f(in)g(the)g(group)f(of)g Fm(comm)m Ft(.)75 944 y(Its)20 b(seman)o(tics)g(is)g(similar)h(to)e(the)h (seman)o(tics)g(of)f(the)h(collectiv)o(e)i(comm)o(unication)e Fm(MPI)p 1663 944 14 2 v 16 w(SCA)l(TTER)75 1001 y Ft(function:)28 b(a)18 b(blo)q(c)o(k)h(of)f(data)g(is)h(read)g(from)f(the)g(\014le)i(and)f (scattered)f(among)g(the)h(group)f(pro)q(cesses;)75 1057 y(eac)o(h)h(pro)q (cess)f(receiv)o(es)i(the)e(same)h(coun)o(t)f(of)g(data.)29 b(The)19 b(t)o(yp)q(e)g(signature)f(asso)q(ciated)h(with)g Fm(count,)75 1114 y(t)o(yp)q(e)d Ft(should)g(b)q(e)g(iden)o(tical)h(at)e(all) h(calling)h(no)q(des.)166 1170 y(Let)f Fm(n)g Ft(b)q(e)g(the)g(size)g(of)g (the)f(comm)o(unication)i(group)e(and)h(let)g Fm(t)o(yp)q(e)p 1330 1170 V 17 w(size)g Ft(b)q(e)g(the)g(size)h(of)e(the)h(data)75 1227 y(describ)q(ed)f(b)o(y)f Fm(t)o(yp)q(e)p Ft(.)20 b(The)14 b(data)e(transfer)h(e\013ected)g(b)o(y)h(this)g(call)g(is)g(as)f(if)h Fm(n)6 b Fg(\001)g Fm(count)16 b Ft(copies)e(of)f Fm(t)o(yp)q(e)h Ft(w)o(ere)75 1283 y(read)g(from)f(the)h(\014le;)h(the)f(resulting)h(message) e(w)o(as)g(split)i(in)o(to)f Fm(n)g Ft(equal)h(sized)g(segmen)o(ts;)e(and)h (segmen)o(t)75 1340 y Fm(i)h Ft(w)o(as)f(sen)o(t)h(to)g(the)g Fm(i)p Ft(-th)g(pro)q(cessor)g(in)h(the)f(comm)o(unication)h(group.)k(Eac)o (h)15 b(pro)q(cess)g(stores)g(incoming)75 1396 y(data)g(in)h(the)f(receiv)o (e)h(bu\013er)f(sp)q(eci\014ed)i(b)o(y)f Fm(bu\013er,)f(count,)h(t)o(yp)q(e)p Ft(.)166 1452 y(On)c(a)g(stream)f(\014le,)i(the)f(\014le)h(p)q(oin)o(ter)g (is)f(incremen)o(ted)h(b)o(y)f Fm(n)t Fg(\001)t Fm(count)t Fg(\001)t Fm(t)o(yp)q(e)p 1409 1452 V 17 w(size)p Ft(;)h(on)e(a)h(record)g (\014le)h(ac-)75 1509 y(cessed)f(with)g Fm(uni)g Ft(metho)q(d,)g Fm(n)s Fg(\001)s Fm(count)s Fg(\001)s Fm(t)o(yp)q(e)p 801 1509 V 18 w(size)f Ft(b)o(ytes)h(are)f(accessed)h(from)f(the)g(curren)o(t)h (record)f(and)h(the)75 1565 y(record)j(p)q(oin)o(ter)h(is)g(incremen)o(ted)h (b)o(y)e(one;)g(on)g(a)g(record)h(\014le)g(accessed)g(with)g Fm(multi)d Ft(metho)q(d)i Fm(t)o(yp)q(e)p 1792 1565 V 18 w(size)75 1622 y Ft(b)o(ytes)i(are)f(accessed)i(from)e Fm(n)c Fg(\001)f Fm(count)18 b Ft(successiv)o(e)h(records)d(and)i(the)f(\014le)h(p)q(oin)o (ter)f(is)h(incremen)o(ted)g(b)o(y)75 1678 y Fm(n)11 b Fg(\001)e Fm(count)p Ft(.)166 1735 y(If)15 b(an)h(end-of-\014le)h(or)d(end-of-record)i (exception)h(has)e(o)q(ccurred,)g(then)h(all)g(pro)q(cesses)g(will)h(return) 75 1791 y(the)e(corresp)q(onding)i(error)d(co)q(de.)21 b(Eac)o(h)15 b(pro)q(cess)h(is)g(returning)g(in)g Fm(status)h Ft(a)e(coun)o(t)g(of)g(the)h (n)o(um)o(b)q(er)f(of)75 1848 y(items)g(and)g(basic)g(elemen)o(ts)h(it)e(had) h(successfully)i(read)e(from)f(the)g(\014le.)21 b(The)15 b(coun)o(t)f(ma)o(y) g(b)q(e)i(di\013eren)o(t)75 1904 y(at)h(di\013eren)o(t)h(pro)q(cesses)g(if)g (an)f(end-of-record)h(or)f(end-of-\014le)j(error)c(o)q(ccurred.)28 b(The)18 b(standard)f(MPI)75 1961 y(error)d(handler)j(considers)f(these)f (errors)g(to)f(b)q(e)i(non-fatal)f(and)h(tak)o(es)e(no)h(further)g(action.)75 2064 y Fm(MPI)p 160 2064 V 16 w(READ)p 295 2064 V 17 w(SCA)l(TTERV\()h (bu\013er,)f(count,)h(t)o(yp)q(e,)g(comm)m(,)c(status\))117 2141 y Fk(OUT)124 b Fm(bu\013er)462 b Fk(initial)12 b(address)j(of)f (bu\013er)g(where)i(data)d(is)h(put)g(\(c)o(hoice\))117 2215 y(IN)171 b Fm(count)466 b Fk(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(read)i (\(in)o(teger\))117 2289 y(IN)171 b Fm(t)o(yp)q(e)491 b Fk(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(elemen)o(t)f(read)i(\(handle\))117 2363 y(IN)171 b Fm(comm)450 b Fk(comm)o(unicator)11 b(asso)q(ciated)k(with)f(\014le)f (\(handle\))117 2436 y(OUT)124 b Fm(status)460 b Fk(return)15 b(status)g(of)e(read)i(op)q(eration)e(\(Status\))75 2561 y Fh(int)47 b(MPI)p 269 2561 15 2 v 17 w(read)p 382 2561 V 17 w(scatterv\()22 b(void*)i(buffer,)e(int)i(count,)f(MPI)p 1306 2561 V 17 w(Datatype)f(type,)393 2617 y(MPI)p 468 2617 V 17 w(Comm)h(comm,)g(MPI)p 819 2617 V 17 w(Status)g(*status\))75 2704 y(MPI)p 150 2704 V 17 w(READ)p 263 2704 V 16 w(SCATTERV\()g(BUFFER,)g (COUNT,)g(TYPE,)g(COMM,)g(STATUS,)g(IERROR\))-32 46 y Fr(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 16 17 bop 75 -100 a Ft(16)1387 b Fp(CHAPTER)16 b(1.)34 b(IO)170 49 y Fh()23 b(BUFFER\(*\))170 106 y(INTEGER)g(COUNT,)g(TYPE,)h(COMM,)f (STATUS\(MPI)p 1058 106 15 2 v 16 w(STATUS)p 1218 106 V 16 w(SIZE\),)g(IERROR)166 197 y Ft(This)14 b(is)g(a)f(collectiv)o(e)j(read)d (function)h(that)f(is)h(in)o(v)o(ok)o(ed)g(b)o(y)f(all)i(pro)q(cesses)f(in)g (the)g(group)f(of)g Fm(comm)m Ft(.)75 253 y(Its)h(seman)o(tics)h(are)f (similar)h(to)f(the)g(seman)o(tics)g(of)g(the)h(collectiv)o(e)h(comm)o (unication)f Fm(MPI)p 1633 253 14 2 v 16 w(SCA)l(TTERV)75 310 y Ft(function)f(whic)o(h)g(allo)o(ws)g(di\013eren)o(t)f(coun)o(t)g(of)g(data) g(to)f(b)q(e)i(sen)o(t)f(to)g(eac)o(h)g(pro)q(cess.)20 b(The)13 b(t)o(yp)q(e)h(signature)75 366 y(asso)q(ciated)g(with)g Fm(t)o(yp)q(e)h Ft(should)g(b)q(e)f(iden)o(tical)i(at)d(all)i(calling)g(no)q(des;)f(pro)q (cesses)h(ma)o(y)e(sp)q(ecify)i(di\013eren)o(t)75 423 y(\(nonnegativ)o(e\))g (v)m(alues)h(for)f Fm(count)p Ft(.)166 484 y(Let)20 b Fm(n)h Ft(b)q(e)g(the)g(size)g(of)f(the)g(comm)o(unication)h(group,)g(let)g Fm(t)o(yp)q(e)p 1302 484 V 17 w(size)g Ft(b)q(e)g(the)f(size)h(of)f(the)h (data)75 540 y(describ)q(ed)c(b)o(y)e Fm(t)o(yp)q(e)p Ft(,)h(and)f(let)h Fm(count)707 547 y Ff(i)733 540 y Ft(b)q(e)g(the)f(v)m(alue)i(of)d(the)i Fm(count)g Ft(argumen)o(t)f(at)f(pro)q(cess)i Fm(i)p Ft(.)166 602 y(The)h(data)g(transfer)f(e\013ected)h(b)o(y)h(this)f(call)h(is)g(as)f (if)1094 570 y Fe(P)1146 602 y Fm(count)1251 609 y Ff(i)1279 602 y Ft(en)o(tries)g(of)g(t)o(yp)q(e)g Fm(t)o(yp)q(e)h Ft(w)o(ere)f(read)75 658 y(from)e(the)g(\014le)i(and)f(concatenated;)f(the)h(resulting)h(list)f(w) o(as)f(split)h(in)o(to)g Fm(n)g Ft(segmen)o(ts,)f(where)g(segmen)o(t)75 715 y Fm(i)g Ft(con)o(tains)g Fm(count)384 722 y Ff(i)410 715 y Ft(en)o(tries;)g(and)h(the)f Fm(i)p Ft(-th)g(segmen)o(t)g(w)o(as)f(sen)o(t) h(to)g Fm(i)p Ft(-th)g(pro)q(cess)g(in)h(the)g(group.)166 776 y(Eac)o(h)i(pro)q(cess)h(returns)f(in)i Fm(status)g Ft(a)e(coun)o(t)g(of)h (the)f(n)o(um)o(b)q(er)h(of)f(items)h(and)f(basic)i(elemen)o(ts)f(it)75 832 y(had)c(read)h(from)e(the)h(\014le;)h(truncation)f(errors)g(are)g (handled)h(as)f(for)g Fm(MPI)p 1346 832 V 16 w(GA)l(THER)p Ft(.)75 941 y Fm(MPI)p 160 941 V 16 w(WRITE)p 319 941 V 16 w(GA)l(THER\()h(bu\013er,)f(count,)i(t)o(yp)q(e,)e(comm)n(,)c(status\))117 1023 y Fk(OUT)124 b Fm(bu\013er)462 b Fk(initial)12 b(address)j(of)f (bu\013er)g(where)i(data)d(is)h(tak)o(en)g(\(c)o(hoice\))117 1107 y(IN)171 b Fm(count)466 b Fk(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f (tak)o(en)h(\(in)o(teger\))117 1192 y(IN)171 b Fm(t)o(yp)q(e)491 b Fk(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(elemen)o(t)f(\(handle\))117 1277 y(IN)171 b Fm(comm)450 b Fk(comm)o(unicator)11 b(asso)q(ciated)k(with)f (\014le)f(\(handle\))117 1361 y(OUT)124 b Fm(status)460 b Fk(return)15 b(status)g(of)e(write)h(op)q(eration)g(\(Status\))75 1491 y Fh(int)47 b(MPI)p 269 1491 15 2 v 17 w(write)p 406 1491 V 17 w(gather\()22 b(void*)i(buffer,)f(int)g(count,)g(MPI)p 1282 1491 V 17 w(Datatype)g(type,)393 1547 y(MPI)p 468 1547 V 17 w(Comm)g(comm,)g(MPI)p 819 1547 V 17 w(Status)g(*status\))75 1638 y(MPI)p 150 1638 V 17 w(WRITE)p 287 1638 V 16 w(GATHER\()g(BUFFER,)g (COUNT,)g(TYPE,)g(COMM,)g(STATUS,)g(IERROR\))170 1695 y()g(BUFFER\(*\)) 170 1751 y(INTEGER)g(COUNT,)g(TYPE,)h(COMM,)f(STATUS\(MPI)p 1058 1751 V 16 w(STATUS)p 1218 1751 V 16 w(SIZE\),)g(IERROR)166 1842 y Ft(This)13 b(is)g(a)f(collectiv)o(e)j(write)d(function)i(that)e(is)h (in)o(v)o(ok)o(ed)g(b)o(y)f(all)i(pro)q(cesses)f(in)g(the)g(group)f(of)g Fm(comm)m Ft(.)75 1899 y(Its)19 b(seman)o(tics)g(are)g(similar)h(to)f(the)g (seman)o(tics)g(of)g(the)g(collectiv)o(e)i(comm)o(unication)f Fm(MPI)p 1686 1899 14 2 v 16 w(GA)l(THER)75 1955 y Ft(function:)33 b(Eac)o(h)22 b(pro)q(cess)g(sends)g(the)f(same)g(coun)o(t)h(of)f(data.)38 b(The)22 b(messages)f(generated)g(b)o(y)h(the)75 2012 y(pro)q(cesses)15 b(are)f(concatenated)g(and)h(the)g(resulting)g(blo)q(c)o(k)g(is)g(written)g (on)o(to)e(the)i(\014le.)20 b(The)15 b(t)o(yp)q(e)g(signa-)75 2068 y(ture)g(asso)q(ciated)h(with)f Fm(count,)h(t)o(yp)q(e)g Ft(should)h(b)q(e)e(iden)o(tical)i(at)e(all)h(calling)h(no)q(des.)166 2129 y(Let)i Fm(n)g Ft(b)q(e)h(the)f(size)h(of)e(the)h(comm)o(unication)h (group,)f(and)g Fm(t)o(yp)q(e)p 1310 2129 V 18 w(size)g Ft(b)q(e)g(the)g (size)h(of)f(the)g(data)75 2186 y(describ)q(ed)e(b)o(y)e Fm(t)o(yp)q(e)p Ft(.)21 b(The)15 b(outcome)f(is)i(as)e(if)i(eac)o(h)f(pro)q(cess)g(assem)o (bles)g(a)g(message)g(from)f(the)h(data)f(in)75 2242 y(the)g(send)h(bu\013er) f(sp)q(eci\014ed)j(b)o(y)d Fm(bu\013er,)g(count,)i(t)o(yp)q(e)p Ft(.)k(The)15 b(blo)q(c)o(ks)g(are)f(concatenated)g(in)h(the)f(order)g(of)75 2299 y(pro)q(cess)h(ranks,)f(and)i(the)f(resulting)h(list)f(of)g Fm(n)10 b Fg(\001)f Fm(count)16 b Ft(en)o(tries)f(of)g(t)o(yp)q(e)g Fm(t)o(yp)q(e)h Ft(is)f(written)g(on)o(to)f(the)h(\014le.)166 2360 y(On)e(a)f(stream)g(\014le,)i Fm(n)5 b Fg(\001)g Fm(count)g Fg(\001)g Fm(t)o(yp)q(e)p 762 2360 V 18 w(size)13 b Ft(b)o(ytes)f(are)h (written)f(and)h(the)g(\014le)g(p)q(oin)o(ter)h(is)f(incremen)o(ted)75 2417 y(b)o(y)19 b(that)f(n)o(um)o(b)q(er;)i(on)e(a)h(record)f(\014le)i (accessed)f(with)g Fm(uni)h Ft(metho)q(d,)f Fm(n)13 b Fg(\001)f Fm(count)i Fg(\001)e Fm(t)o(yp)q(e)p 1591 2417 V 17 w(size)19 b Ft(b)o(ytes)f(are)75 2473 y(written)g(on)g(the)g(curren)o(t)g(record)g(and) g(the)g(\014le)h(p)q(oin)o(ter)g(is)f(incremen)o(ted)h(b)o(y)f(one;)h(on)f(a) g(record)g(\014le)75 2529 y(accessed)12 b(with)f Fm(multi)e Ft(metho)q(d,)j Fm(t)o(yp)q(e)p 717 2529 V 17 w(size)f Ft(b)o(ytes)g(are)g (written)g(on)g(eac)o(h)g(of)g Fm(n)r Fg(\001)r Fm(count)h Ft(successiv)o(e)g(records,)75 2586 y(and)j(the)h(\014le)g(p)q(oin)o(ter)g (is)f(incremen)o(ted)i(b)o(y)e Fm(n)10 b Fg(\001)g Fm(count)p Ft(.)166 2647 y(Eac)o(h)15 b(pro)q(cess)h(is)g(returning)h(in)f Fm(status)i Ft(a)d(coun)o(t)g(of)g(the)h(n)o(um)o(b)q(er)g(of)f(items)h(and)g (basic)g(elemen)o(ts)75 2704 y(it)j(has)g(written)f(on)h(the)g(\014le.)32 b(Di\013eren)o(t)18 b(pro)q(cesses)h(ma)o(y)f(return)h(di\013eren)o(t)g(v)m (alues)h(if)f(a)g(truncation)1967 46 y Fr(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 17 18 bop 75 -100 a Fp(1.6.)34 b(COLLECTIVE)16 b(A)o(CCESSES)1095 b Ft(17)75 49 y(o)q(ccurred.)27 b(In)18 b(suc)o(h)g(situation,)g(eac)o(h)g (pro)q(cess)g(is)g(returned)g(an)f(error)g(co)q(de)h(and)g(the)f(error)g (handler)75 106 y(attac)o(hed)e(to)f(the)h(comm)o(unicator)g(is)h(in)o(v)o (ok)o(ed)f(at)g(eac)o(h)g(pro)q(cess.)75 213 y Fm(MPI)p 160 213 14 2 v 16 w(WRITE)p 319 213 V 16 w(GA)l(THERV\()i(bu\013er,)e(count,)h(t) o(yp)q(e,)g(comm)m(,)c(status\))117 294 y Fk(OUT)124 b Fm(bu\013er)462 b Fk(initial)12 b(address)j(of)f(bu\013er)g(where)i(data)d(is)h(gotten)g(\(c) o(hoice\))117 377 y(IN)171 b Fm(count)466 b Fk(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(read)i(\(in)o(teger\))117 460 y(IN)171 b Fm(t)o(yp)q(e)491 b Fk(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(elemen)o(t)f (\(handle\))117 543 y(IN)171 b Fm(comm)450 b Fk(comm)o(unicator)11 b(asso)q(ciated)k(with)f(\014le)f(\(handle\))117 625 y(OUT)124 b Fm(status)460 b Fk(return)15 b(status)g(of)e(write)h(op)q(eration)g (\(Status\))75 754 y Fh(int)47 b(MPI)p 269 754 15 2 v 17 w(write)p 406 754 V 17 w(gatherv\(*file)22 b(f,)h(void*)g(buffer,)g(int)h(count,)f(MPI) p 1497 754 V 17 w(Datatype)f(type,)393 810 y(MPI)p 468 810 V 17 w(Comm)h(comm,)g(MPI)p 819 810 V 17 w(Status)g(*status\))75 900 y(MPI)p 150 900 V 17 w(WRITE)p 287 900 V 16 w(GATHERV\()g(BUFFER,)g (COUNT,)g(TYPE,)g(COMM,)g(STATUS,)g(IERROR\))170 957 y()g(BUFFER\(*\)) 170 1013 y(INTEGER)g(COUNT,)g(TYPE,)h(COMM,)f(STATUS\(MPI)p 1058 1013 V 16 w(STATUS)p 1218 1013 V 16 w(SIZE\),)g(IERROR)166 1104 y Ft(This)13 b(is)g(a)f(collectiv)o(e)j(write)d(function)i(that)e(is)h (in)o(v)o(ok)o(ed)g(b)o(y)f(all)i(pro)q(cesses)f(in)g(the)g(group)f(of)g Fm(comm)m Ft(.)75 1160 y(Its)k(seman)o(tics)h(are)f(similar)h(to)f(the)g (seman)o(tics)h(of)f(the)g(collectiv)o(e)i(comm)o(unication)f Fm(MPI)p 1655 1160 14 2 v 16 w(GA)l(THERV)75 1217 y Ft(function)d(whic)o(h)f (allo)o(ws)g(di\013eren)o(t)g(coun)o(t)g(of)f(data)g(to)g(b)q(e)i(sen)o(t)e (b)o(y)h(eac)o(h)g(pro)q(cess.)19 b(The)13 b(t)o(yp)q(e)g(signature)75 1273 y(asso)q(ciated)h(with)g Fm(t)o(yp)q(e)h Ft(should)g(b)q(e)f(iden)o (tical)i(at)d(all)i(calling)g(no)q(des;)f(pro)q(cesses)h(ma)o(y)e(sp)q(ecify) i(di\013eren)o(t)75 1329 y(\(nonnegativ)o(e\))g(v)m(alues)h(for)f Fm(count)p Ft(.)166 1390 y(Let)20 b Fm(n)h Ft(b)q(e)g(the)g(size)g(of)f(the)g (comm)o(unication)h(group,)g(let)g Fm(t)o(yp)q(e)p 1302 1390 V 17 w(size)g Ft(b)q(e)g(the)f(size)h(of)f(the)h(data)75 1446 y(describ)q(ed)i(b)o(y)e Fm(t)o(yp)q(e)h Ft(and)f(let)h Fm(count)724 1453 y Ff(i)756 1446 y Ft(b)q(e)g(the)f(v)m(alue)h(of)f(the)g Fm(count)h Ft(argumen)o(t)f(at)f(pro)q(cess)h Fm(i)p Ft(.)38 b(The)75 1503 y(outcome)14 b(is)i(as)e(if)h(a)f(message)h(is)g(generated)g(b) o(y)f(eac)o(h)h(pro)q(cess)g(b)o(y)g(assem)o(bling)g(the)g(data)f(in)i(the)e (send)75 1559 y(bu\013er)g(sp)q(eci\014ed)j(b)o(y)d Fm(bu\013er,)g(count,)h (t)o(yp)q(e)p Ft(;)h(the)e(messages)g(are)g(concatenated)g(in)h(pro)q(cess)g (rank)f(order;)75 1616 y(and)i(the)f(resulting)i(list)f(of)558 1583 y Fe(P)610 1616 y Fm(count)715 1623 y Ff(i)741 1616 y Ft(elemen)o(ts)g(of)f(t)o(yp)q(e)h Fm(t)o(yp)q(e)h Ft(is)f(written)f(on)h (the)f(\014le.)22 b(Eac)o(h)16 b(pro)q(cess)75 1672 y(is)f(returning)f(in)h Fm(status)h Ft(a)e(coun)o(t)g(of)f(the)h(n)o(um)o(b)q(er)h(of)f(items)g(and)g (basic)h(elemen)o(ts)g(it)f(has)g(written)g(on)o(to)75 1728 y(the)i(\014le.)24 b(An)16 b(end-of-\014le)i(or)e(end-of-record)g(error)g (indication)i(is)f(returned)f(to)g(eac)o(h)g(calling)i(pro)q(cess)75 1785 y(is)e(suc)o(h)f(error)g(o)q(ccurred)h(during)g(the)f(\014le)h(access.) 75 1929 y Fo(1.6.2)49 b(File)17 b(p)q(ositioning)75 2069 y Fm(MPI)p 160 2069 V 16 w(SEEK)p 287 2069 V 16 w(ALL\(comm)m(,)11 b(o\013set,)k(whence\))117 2150 y Fk(IN)155 b Fm(comm)466 b Fk(comm)o(unicator)11 b(asso)q(ciated)k(with)f(\014le)f(\(handle\))117 2233 y(IN)155 b Fm(o\013set)484 b Fk(o\013set)15 b(in)o(to)e(\014le)h(\(in)o (teger\))117 2316 y(IN)155 b Fm(whence)450 b Fk(one)20 b(of)f Fj(MPI)p 1117 2316 13 2 v 15 w(SET)g Fk(\(indicating)g(start)i(of)e (\014le\),)i Fj(MPI)p 1779 2316 V 14 w(END)905 2372 y Fk(\(indicating)16 b(end)i(of)e(\014le\),)h(or)g Fj(MPI)p 1479 2372 V 15 w(CUR)e Fk(\(indicating)h(cur-)905 2429 y(ren)o(t)f(p)q(osition\))e(\(state\))p Ft(.)75 2557 y Fh(int)47 b(MPI)p 269 2557 15 2 v 17 w(Seek)p 382 2557 V 17 w(all\(MPI)p 567 2557 V 16 w(Comm)23 b(comm,)g(MPI)p 917 2557 V 17 w(Fint)h(offset,)e(MPI)p 1316 2557 V 17 w(IO)i(whence\))75 2647 y(MPI)p 150 2647 V 17 w(SEEK)p 263 2647 V 16 w(ALL\(COMM,)f(OFFSET,)g (WHENCE,)g(IERROR\))170 2704 y(INTEGER)g(COMM,)h(OFFSET,)e(WHENCE,)h(IERROR) -32 46 y Fr(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 18 19 bop 75 -100 a Ft(18)1387 b Fp(CHAPTER)16 b(1.)34 b(IO)75 49 y Fm(MPI)p 160 49 14 2 v 16 w(REWIND)p 353 49 V 16 w(ALL\(comm)m(\))117 127 y Fk(IN)155 b Fm(comm)466 b Fk(comm)o(unicator)11 b(asso)q(ciated)k(with) f(\014le)75 252 y Fh(int)47 b(MPI)p 269 252 15 2 v 17 w(Rewind)p 430 252 V 16 w(all\(MPI)p 614 252 V 17 w(Comm)23 b(comm\))75 339 y(MPI)p 150 339 V 17 w(REWIND)p 311 339 V 16 w(ALL\(COMM,)g(IERROR\))170 395 y(INTEGER)g(COMM,)h(IERROR)75 529 y Fm(MPI)p 160 529 14 2 v 16 w(BA)o(CKSP)l(A)o(CE)p 431 529 V 17 w(ALL\(com)o(m)m(\))117 607 y Fk(IN)155 b Fm(comm)466 b Fk(comm)o(unicator)11 b(asso)q(ciated)k(with) f(\014le)75 732 y Fh(int)47 b(MPI)p 269 732 15 2 v 17 w(Backspace)p 502 732 V 16 w(all\(MPI)p 686 732 V 16 w(Comm)24 b(comm\))75 819 y(MPI)p 150 819 V 17 w(BACKSPACE)p 383 819 V 16 w(ALL\(COMM,)e(IERROR\)) 170 875 y(INTEGER)h(COMM,)h(IERROR)75 1009 y Fm(MPI)p 160 1009 14 2 v 16 w(ENDFILE)p 359 1009 V 16 w(ALL\(com)o(m)m(\))117 1087 y Fk(IN)155 b Fm(comm)466 b Fk(comm)o(unicator)11 b(asso)q(ciated)k (with)f(\014le)75 1212 y Fh(int)47 b(MPI)p 269 1212 15 2 v 17 w(Endfile)p 454 1212 V 16 w(all\(MPI)p 638 1212 V 17 w(Comm)23 b(comm\))75 1299 y(MPI)p 150 1299 V 17 w(ENDFILE\(COMM,)f(IERROR\))170 1355 y(INTEGER)h(COMM,)h(IERROR)166 1442 y Ft(These)11 b(functions)h(ha)o(v)o (e)f(the)g(same)g(meaning)g(as)g(the)g(corresp)q(onding)h(functions)g (without)f(the)21 b Fj(ALL)75 1499 y Ft(su\016x,)e(except)f(that)g(they)g (are)g(called)i(collectiv)o(ely)h(b)o(y)d(all)h(pro)q(cesses)g(in)g(the)f (group;)h(all)h(pro)q(cesses)75 1555 y(pro)o(vide)c(the)f(same)g(argumen)o(t) f(v)m(alues.)166 1612 y(The)d(function)i Fm(MPI)p 515 1612 14 2 v 15 w(GETPOS)g Ft(can)e(b)q(e)h(applied)h(to)e(a)g(\014le)h(op)q(ened)h (with)f(the)21 b Fj(COLLECTIVE)10 b Ft(mo)q(de.)75 1669 y(It)15 b(can)h(b)q(e)f(called)i(lo)q(cally)g(b)o(y)e(eac)o(h)h(pro)q(cess)f(in)h (the)f(group.)166 1808 y Fl(Implemen)o(tati)o(on)c(note:)35 b Fk(All)11 b(the)j(functions)e(ma)o(y)f(b)q(e)i(implemen)o(ted)e(lo)q(cally) m(,)f(if)i(the)i(v)n(alue)e(of)g(the)h(\014le)75 1865 y(p)q(oin)o(ter)h(is)g (stored)h(at)e(eac)o(h)i(no)q(de.)75 2093 y Fu(1.7)59 b(Erro)n(r)20 b(handling)75 2195 y Ft(MPI-IO)14 b(uses)g(the)g(general)g(error)f(handling)i (facilities)g(of)f(MPI.)f(MPI-IO)h(calls)h(return)e(an)h(error)e(co)q(de)75 2252 y(that)e(indicate)i(the)f(nature)g(of)f(the)h(error,)g(if)g(one)g(o)q (ccurred.)19 b(The)11 b(function)h Fm(MPI)p 1458 2252 V 15 w(ERROR)p 1620 2252 V 18 w(STRING)g Ft(can)75 2308 y(b)q(e)h(used)g(to)f (prin)o(t)g(the)h(error)e(message)h(asso)q(ciated)h(with)f(an)g(error)g(co)q (de.)20 b(Whenev)o(er)12 b(an)g(error)g(o)q(ccurs,)75 2365 y(the)k(error)g(handler)h(attac)o(hed)e(to)h(the)g(comm)o(unicator)g(used)g (in)h(the)f(call)i(is)e(in)o(v)o(ok)o(ed.)23 b(The)16 b(standard)75 2421 y(error)h(handler)i(considers)f(truncation)g(errors)f(on)h(read)f(to)g (b)q(e)i(non-fatal)e(and)h(truncation)g(error)f(on)75 2478 y(writes)e(to)e(b)q(e)j(fatal.)j(This)c(b)q(eha)o(vior)g(can)g(b)q(e)g(c)o (hanged)g(b)o(y)f(attac)o(hing)g(another)h(error)e(handler)j(to)e(the)75 2534 y(comm)o(unicator)g(used:)20 b(The)15 b(use)g(of)f(the)g(error)g (handler)h Fj(MPI)p 1142 2534 13 2 v 15 w(ERRORSRETURN)e Ft(will)j(cause)e (all)i(errors)75 2591 y(to)e(b)q(e)i(non-fatal:)k(it)15 b(is)g(the)g(user)g (resp)q(onsibilit)o(y)l(,)j(in)d(this)h(case,)e(to)h(test)f(the)h(error)f(co) q(de)i(returned)f(b)o(y)75 2647 y(eac)o(h)i(co)q(de.)24 b(The)17 b(use)g(of)f(the)h(error)e(handler)j Fj(MPI)p 968 2647 V 14 w(ERRORSF)m(A)m(T)m(AL)c Ft(will)k(cause)f(all)h(I/O)f(errors)e(to)h(b)q(e)75 2704 y(treated)f(as)g(fatal)f(and)i(ab)q(ort)e(the)i(computation.)1967 46 y Fr(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 19 20 bop 75 -100 a Fp(1.8.)34 b(VEST)l(A)16 b(FUNCTIONS)1206 b Ft(19)75 49 y Fu(1.8)59 b(V)n(esta)20 b(functions)75 154 y Fo(1.8.1)49 b(File)17 b(Distribution)75 242 y Ft(A)h(V)l(esta)g(\014le)h(consists)f(of)f (a)h(t)o(w)o(o-dimensional)h(arra)o(y)d(of)i(records,)g(un)o(b)q(ounded)i(in) e(the)g(v)o(ertical)h(di-)75 298 y(mension.)h(A)14 b(V)l(esta)g(\014le)g(ma)o (y)f(b)q(e)i(partitioned)f(in)o(to)g Fn(sub\014les)p Ft(,)g(whic)o(h)g(are)g (submatrices)g(of)f(the)h(V)l(esta)75 354 y(\014le)i(matrix.)j(The)d(mec)o (hanism)f(for)f(sp)q(ecifying)j(a)e Fn(partitioning)k(sc)o(heme)14 b Ft(that)g(partitions)i(a)e(V)l(esta)75 411 y(\014le)k(in)o(to)f(sub\014les) h(is)g(similar)g(to)e(that)h(used)g(for)g(mapping)g(a)g(t)o(w)o (o-dimensional)h(arra)o(y)e(in)i(High)f(P)o(er-)75 467 y(formance)e(F)l (ortran)f(\(HPF)g([5)o(]\):)19 b(A)d(sub\014le)g(is)g(a)e(subset)i(of)e(the)h (t)o(w)o(o-dimensional)h(arra)o(y)e(that)h(w)o(ould)75 524 y(b)q(e)h(mapp)q(ed)g(to)e(one)i(\(virtual\))f(no)q(de)h(b)o(y)f(an)g(HPF)g (distribution.)166 581 y(The)j(same)g(\014le)h(ma)o(y)e(b)q(e)i(op)q(ened)g (with)g(di\013eren)o(t)f(partitioning)h(sc)o(hemes.)29 b(Th)o(us,)18 b(the)g(records)75 638 y(in)h(the)g(\014le)g(ma)o(y)e(b)q(e)i(group)q(ed)g (and)g(ordered)f(di\013eren)o(tly)l(,)i(according)f(to)e(the)i(access)f (pattern)g(of)g(an)75 694 y(application,)e(without)g(actually)g(mo)o(ving)f (the)g(data.)166 751 y(When)20 b(a)g(V)l(esta)g(\014le)h(is)f(accessed)h(b)o (y)e(MPI-IO)i(in)g Fm(p)o(rivate)f Ft(mo)q(de,)h(then)f(a)g(distinct)h (sub\014le)g(is)75 808 y(allo)q(cated)14 b(to)f(eac)o(h)g(pro)q(cess)g(in)h (the)f(accessing)h(group.)19 b(When)14 b(a)f(V)l(esta)g(\014le)h(is)f(allo)q (cated)i(in)f(an)o(y)e(other)75 864 y(mo)q(de,)18 b(then)g(the)g(same)f (sub\014le)i(is)f(accessed)g(b)o(y)g(all)g(pro)q(cesses)g(in)g(the)g(group.) 27 b(In)18 b(either)g(case,)g(the)75 921 y(sub\014le)g(is)g(considered)g(to)e (consist)h(of)g(a)f(sequence)i(of)e(records,)h(with)g(records)g(ordered)g(in) h(canonical)75 977 y(order)c(within)h(the)f(sub\014le.)21 b(Suc)o(h)15 b(a)e(sub\014le)j(corresp)q(onds)e(to)g(a)f(\014xed)i(record)f(size)h(MPI-IO) g(\014le.)20 b(This)75 1034 y(\014le)d(can)f(also)f(b)q(e)h(accessed)h(as)e (a)g(stream)g(\014le,)i(b)o(y)e(ignoring)i(the)e(record)h(structure.)21 b(V)l(ariable)c(record)75 1090 y(size)f(\014les)g(are)f(curren)o(tly)h(not)f (supp)q(orted)g(b)o(y)h(V)l(esta.)166 1147 y(W)l(e)f(describ)q(e)i(no)o(w)e (in)h(more)f(detail)h(the)f(structure)g(of)g(V)l(esta)g(\014les)h(and)f(of)g (sub\014les.)166 1205 y(A)i(V)l(esta)g(\014le)h(is)f(stored)f(across)h(a)f(n) o(um)o(b)q(er)i(of)e Fn(storage)k(no)q(des)p Ft(.)25 b(The)17 b(V)l(esta)g(\014le)h(structure)e(is)75 1261 y(de\014ned)e(b)o(y)f(t)o(w)o(o) f(parameters:)18 b(The)13 b(n)o(um)o(b)q(er)g Fd(ncel)q(l)q(s)f Ft(of)h(V)l(esta)f(\014le)i(columns)g(\(also)f(called)h Fn(cells)p Ft(\),)g(and)75 1318 y(the)j(record)h(size)g Fd(r)q(ec)p 451 1318 14 2 v 16 w(siz)r(e)p Ft(.)26 b(W)l(e)17 b(denote)h(b)o(y)f Fd(r)q(ec)941 1325 y Fc(i;j)997 1318 y Ft(record)g Fd(i)g Ft(of)g(cell)i Fd(j)s Ft(.)25 b(Note)17 b(that)g(all)h(n)o(um)o(b)q(erings)75 1374 y(are)d(zero)g(based.)166 1431 y(The)i(default)h(n)o(um)o(b)q(er)g(of)f (cells)h(is)g(one)g(p)q(er)f(storage)f(no)q(de,)i(in)g(whic)o(h)g(case)g(one) f(\014le)h(column)g(is)75 1488 y(stored)c(on)h(eac)o(h)g(storage)e(no)q(de.) 20 b(When)15 b(these)g(n)o(um)o(b)q(ers)g(are)g(not)f(equal,)h(the)g(system)f (allo)q(cates)h(cells)75 1544 y(to)g(storage)f(no)q(des)h(in)h(a)f (round-robin)h(fashion;)g(a)f(cell)h(is)g(alw)o(a)o(ys)e(con)o(tained)i(b)o (y)f(one)h(storage)e(no)q(de.)166 1602 y(A)h(partitioning)h(sc)o(heme)g(is)g (sp)q(eci\014ed)h(b)o(y)e(four)g(parameters:)75 1699 y Fn(Hn:)22 b Ft(n)o(um)o(b)q(er)16 b(of)e(horizon)o(tal)i(partitions.)75 1796 y Fn(Vn:)22 b Ft(n)o(um)o(b)q(er)15 b(of)g(v)o(ertical)h(partitions.)75 1894 y Fn(Hbs:)22 b Ft(horizon)o(tal)15 b(size)h(of)f(blo)q(c)o(k.)75 1991 y Fn(Vbs:)21 b Ft(v)o(ertical)16 b(size)g(of)f(blo)q(c)o(k.)166 2088 y(The)g(inequalit)o(y)i(\()p Fd(H)t(n)10 b Fg(\000)g Ft(1\))g Fg(\001)f Fd(H)t(bs)j(<)h(ncel)q(l)q(s)i Ft(should)h(hold.)166 2145 y(The)g(n)o(um)o(b)q(er)h(of)f(horizon)o(tal)h(or)e(v)o(ertical)i (partitions)g(is)g(equiv)m(alen)o(t)h(to)d(the)i(n)o(um)o(b)q(er)f(of)g (virtual)75 2202 y(pro)q(cessors)i(in)h(a)f(pro)q(cessor)g(grid)h(used)f(as)g (the)h(target)e(of)g(an)h(HPF)35 b Fj(DISTRIBUTE)17 b Ft(statemen)o(t;)h(the) 75 2258 y(horizon)o(tal)d(or)g(v)o(ertical)g(blo)q(c)o(k)h(size)f(is)h(equiv) m(alen)o(t)g(to)f(the)g(parameter)f(that)g(de\014nes)i(blo)q(c)o(k)g(size)f (in)h(an)75 2315 y(HPF)f(blo)q(c)o(k-cyclic)j(distribution.)166 2372 y(The)j(four)f(parameters)g(de\014ne)i(a)e(partitioning)i(sc)o(heme)f (with)g(a)f Fd(H)t(n)14 b Fg(\001)f Fd(V)d(n)21 b Ft(arra)o(y)e(of)i (sub\014les)75 2429 y Fd(par)q(t)160 2436 y Fc(r)o(;s)202 2429 y Ft(.)h(W)l(e)16 b(n)o(um)o(b)q(er)g(these)g(sub\014les)h(in)g(ro)o(w)e(ma)s (jor)f(order.)21 b(In)c(eac)o(h)f(dimension,)h(the)f(partitioning)75 2485 y(sc)o(heme)k(consists)g(of)g(a)f(recurring)i(pattern)e(of)h Fd(H)t(n)f Ft(\()p Fd(V)9 b(n)p Ft(\))20 b(in)o(terlea)o(v)o(ed)h(blo)q(c)o (ks,)g(where)f(eac)o(h)g(blo)q(c)o(k)75 2542 y(con)o(tains)15 b Fd(H)t(bs)g Ft(\()p Fd(V)9 b(bs)p Ft(\))15 b(consecutiv)o(e)h(records.)k (This)15 b(is)h(illustrated)h(in)f(Figure)f(1.1.)166 2599 y(More)f(precisely) l(,)j(assume)e(that)607 2704 y Fd(ncel)q(l)q(s)d Ft(=)h Fd(B)g Fg(\001)c Fd(H)t(n)h Fg(\001)g Fd(H)t(bs)f Ft(+)i Fd(H)i Fg(\001)d Fd(H)t(bs)g Ft(+)g Fd(I)t(;)-32 46 y Fr(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 20 21 bop 75 -100 a Ft(20)1387 b Fp(CHAPTER)16 b(1.)34 b(IO)75 4 y 19608222 14208860 0 0 42626580 30785863 startTexFig 75 4 a %%BeginDocument: vst-part.eps /Freelance_Plus dup 100 dict def load begin [ ] {bind} stopped { (patching the bind operator...) = flush /*bind /bind load def /bind { dup xcheck { *bind } if } *bind def } if pop /bdf {bind def} bind def /ldf {load def} bdf /mt /moveto ldf /rt /rmoveto ldf /l2 /lineto ldf /c2 /curveto ldf /sg /setgray ldf /gs /gsave ldf /ef /eofill ldf /rl2 /rlineto ldf /st /stroke ldf /gr /grestore ldf /np /newpath ldf /sv /save ldf /su /statusdict ldf /rs /restore ldf /sw /setlinewidth ldf /sd /setdash ldf /cp /closepath ldf /ed {exch def } bdf /cfnt {findfont exch makefont setfont} bdf /itr {transform round exch round exch itransform} bdf /fres 72 0 matrix currentmatrix dtransform exch dup mul exch dup mul add sqrt def /res fres def /mcm matrix currentmatrix bdf end Freelance_Plus begin save newpath .1 .1 scale /ecm matrix currentmatrix bdf /sem {ecm setmatrix} bdf -720 -720 translate 0 setlinecap 0 setlinejoin 1.42 setmiterlimit 0.000 sg 6 sw sv np [] 0 sd 1598 3126 itr mt 5402 3126 itr l2 5402 4869 itr l2 1598 4869 itr l2 1598 3126 itr l2 cp st rs /Ich 256 array def StandardEncoding Ich copy pop /bullet/paragraph/section/dieresis/tilde/ring /circumflex/grave/acute/quotedblleft/quotesingle /ellipsis/endash/emdash/guilsinglleft/guilsinglright /quotedblbase/quotesinglbase/quotedblright/OE/oe /Ydieresis/fi/fl/dagger/daggerdbl/Ccedilla/udieresis /eacute/acircumflex/adieresis/agrave/aring/ccedilla /ecircumflex/edieresis/egrave/idieresis/icircumflex /igrave/Adieresis/Aring/Eacute/ae/AE/ocircumflex /odieresis/ograve/ucircumflex/ugrave/ydieresis /Odieresis/Udieresis/oslash/sterling/Oslash/florin /aacute/iacute/oacute/uacute/ntilde/Ntilde/ordfeminine /ordmasculine/questiondown/exclamdown/guillemotleft /guillemotright/Aacute/Acircumflex/Agrave/cent/yen /atilde/Atilde/currency/Ecircumflex/Edieresis/Egrave /dotlessi/Iacute/Icircumflex/Idieresis/Igrave/Oacute /germandbls/Ocircumflex/Ograve/otilde/Otilde/Uacute /Ucircumflex/Ugrave/macron/cedilla/periodcentered Ich 127 97 getinterval astore pop /Ienc { /ncs Ich def /nfn ed /bfn ed /bfd bfn findfont def /nf bfd maxlength dict def bfd{exch dup dup /FID ne exch /Encoding ne and {exch nf 3 1 roll put}{pop pop} ifelse }forall nf/FontName nfn put nf/Encoding ncs put nfn nf definefont pop}bdf /IencO { /ncs Ich def /nfn ed /bfn ed /lnw ed /bfd bfn findfont def /nf bfd maxlength 4 add dict def bfd{exch dup dup /FID ne exch /Encoding ne and {exch nf 3 1 roll put}{pop pop} ifelse }forall nf/FontName nfn put nf/Encoding ncs put nf/PaintType 2 put nf/StrokeWidth lnw put nfn nf definefont pop}bdf /IencSO { /nfn ed /bfn ed /lnw ed /bfd bfn findfont def /nf bfd maxlength 4 add dict def bfd{exch dup /FID ne { exch nf 3 1 roll put}{pop pop} ifelse }forall nf/FontName nfn put nf/PaintType 2 put nf/StrokeWidth lnw put nfn nf definefont pop}bdf /Helvetica /Flfon1 Ienc [191 0 0.00 191 -18 -142] /Flfon1 cfnt 1843 5009 itr mt -53 0 itr rt 0 71 itr rt (0) show 2311 5014 itr mt -53 0 itr rt 0 71 itr rt (1) show 2787 5014 itr mt -53 0 itr rt 0 71 itr rt (2) show 3262 5014 itr mt -53 0 itr rt 0 71 itr rt (3) show 3738 5014 itr mt -53 0 itr rt 0 71 itr rt (4) show 4213 5014 itr mt -53 0 itr rt 0 71 itr rt (5) show 4689 5014 itr mt -53 0 itr rt 0 71 itr rt (6) show 5164 5014 itr mt -53 0 itr rt 0 71 itr rt (7) show 1360 4724 itr mt -53 0 itr rt 0 71 itr rt (0) show 1360 4433 itr mt -53 0 itr rt 0 71 itr rt (1) show 1360 4143 itr mt -53 0 itr rt 0 71 itr rt (2) show 1360 3852 itr mt -53 0 itr rt 0 71 itr rt (3) show 1360 3562 itr mt -53 0 itr rt 0 71 itr rt (4) show 1360 3271 itr mt -53 0 itr rt 0 71 itr rt (5) show 12 sw sv np 2549 4869 itr mt 2549 3126 itr l2 st rs sv np 3500 4869 itr mt 3500 3126 itr l2 st rs sv np 4451 4869 itr mt 4451 3126 itr l2 st rs sv np [] 0 sd 1598 3998 itr mt 2549 3998 itr l2 2549 4869 itr l2 1598 4869 itr l2 1598 3998 itr l2 gs 0.920 sg ef gr rs sv np [] 0 sd 4451 3998 itr mt 5402 3998 itr l2 5402 4869 itr l2 4451 4869 itr l2 4451 3998 itr l2 gs 0.920 sg ef gr rs /fi16 {0 16 2} def /bitval {/yb ed /xb ed bs yb bw mul xb 8 idiv add get 1 7 xb 8 mod sub bitshift and 0 ne} bdf /spotf {/y ed /x ed x 1 add 2 div bps mul cvi y 1 add 2 div bps mul cvi bitval {/onb onb 1 add def 1} {/ofb ofb 1 add def 0} ifelse} def /uset {/uf ed /ua ed /uc ed uc dup matrix scale ua matrix rotate mcm mcm concatmatrix mcm concatmatrix pop 1 0 mcm dtransform /y1 ed /x1 ed /nsc y1 x1 atan def res x1 dup mul y1 dup mul add sqrt div nsc /uf load setscreen} bdf /mkfill {/cz ed /bs ed /bw ed /bps ed /an ed /onb 0 def /ofb 0 def cz an /spotf load uset {} settransfer ofb ofb onb add div sg} bdf sv np [] 0 sd 2549 3998 itr mt 3500 3998 itr l2 3500 4869 itr l2 2549 4869 itr l2 2549 3998 itr l2 gs fi16 < 0001800240042008101008200440028001000280044008201010200840048002 > 03.84 mkfill ef gr rs sv np [] 0 sd 3500 3998 itr mt 4451 3998 itr l2 4451 4869 itr l2 3500 4869 itr l2 3500 3998 itr l2 gs 0.850 sg ef gr rs sv np [] 0 sd 2549 3126 itr mt 3500 3126 itr l2 3500 3998 itr l2 2549 3998 itr l2 2549 3126 itr l2 gs fi16 < 8000400020001000080004000200010000800040002000100008000400020001 > 03.84 mkfill ef gr rs sv np [] 0 sd 1598 3126 itr mt 2549 3126 itr l2 2549 3998 itr l2 1598 3998 itr l2 1598 3126 itr l2 gs fi16 < 0001000200040008001000200040008001000200040008001000200040008000 > 03.84 mkfill ef gr rs 6 sw sv np [] 0 sd 4451 3126 itr mt 5402 3126 itr l2 5402 3998 itr l2 4451 3998 itr l2 4451 3126 itr l2 gs fi16 < 0001000200040008001000200040008001000200040008001000200040008000 > 03.84 mkfill ef gr rs 12 sw sv np [] 0 sd 3500 3126 itr mt 4451 3126 itr l2 4451 3998 itr l2 3500 3998 itr l2 3500 3126 itr l2 rs 6 sw sv np [] 0 sd 1598 1383 itr mt 5402 1383 itr l2 5402 3126 itr l2 1598 3126 itr l2 1598 1383 itr l2 rs 12 sw sv np [] 0 sd 2549 3127 itr mt 2549 1383 itr l2 st rs sv np 3500 3127 itr mt 3500 1383 itr l2 st rs sv np 4451 3127 itr mt 4451 1383 itr l2 st rs sv np [] 0 sd 1598 2255 itr mt 2549 2255 itr l2 2549 3126 itr l2 1598 3126 itr l2 1598 2255 itr l2 gs 0.920 sg ef gr rs 6 sw sv np [] 0 sd 4451 2255 itr mt 5402 2255 itr l2 5402 3126 itr l2 4451 3126 itr l2 4451 2255 itr l2 gs 0.920 sg ef gr rs 12 sw sv np [] 0 sd 2549 2255 itr mt 3500 2255 itr l2 3500 3126 itr l2 2549 3126 itr l2 2549 2255 itr l2 gs fi16 < 0001800240042008101008200440028001000280044008201010200840048002 > 03.84 mkfill ef gr rs sv np [] 0 sd 3500 2255 itr mt 4451 2255 itr l2 4451 3126 itr l2 3500 3126 itr l2 3500 2255 itr l2 gs 0.850 sg ef gr rs sv np [] 0 sd 2549 1383 itr mt 3500 1383 itr l2 3500 2255 itr l2 2549 2255 itr l2 2549 1383 itr l2 gs fi16 < 8000400020001000080004000200010000800040002000100008000400020001 > 03.84 mkfill ef gr rs sv np [] 0 sd 1598 1383 itr mt 2549 1383 itr l2 2549 2255 itr l2 1598 2255 itr l2 1598 1383 itr l2 gs fi16 < 0001000200040008001000200040008001000200040008001000200040008000 > 03.84 mkfill ef gr rs sv np [] 0 sd 4451 1383 itr mt 5402 1383 itr l2 5402 2255 itr l2 4451 2255 itr l2 4451 1383 itr l2 gs fi16 < 0001000200040008001000200040008001000200040008001000200040008000 > 03.84 mkfill ef gr rs sv np [] 0 sd 3500 1383 itr mt 4451 1383 itr l2 4451 2255 itr l2 3500 2255 itr l2 3500 1383 itr l2 rs 1360 2981 itr mt -53 0 itr rt 0 71 itr rt (6) show 1360 2690 itr mt -53 0 itr rt 0 71 itr rt (7) show 1360 2400 itr mt -53 0 itr rt 0 71 itr rt (8) show 1360 2109 itr mt -53 0 itr rt 0 71 itr rt (9) show 1360 1819 itr mt -106 0 itr rt 0 71 itr rt (10) show 1360 1528 itr mt -106 0 itr rt 0 71 itr rt (11) show 1360 1238 itr mt -27 0 itr rt 0 71 itr rt (.) show 1360 947 itr mt -27 0 itr rt 0 71 itr rt (.) show 1360 1093 itr mt -27 0 itr rt 0 71 itr rt (.) show sv np [] 0 sd 2549 4869 itr mt 2549 1383 itr l2 st rs sv np 3500 4869 itr mt 3500 1383 itr l2 st rs sv np 4451 4869 itr mt 4451 1383 itr l2 st rs [153 0 0.00 153 -15 -113] /Flfon1 cfnt 1836 4724 itr mt -43 0 itr rt 0 57 itr rt (0) show 1825 4460 itr mt -43 0 itr rt 0 57 itr rt (1) show 1836 4143 itr mt -43 0 itr rt 0 57 itr rt (2) show 2311 4724 itr mt -43 0 itr rt 0 57 itr rt (3) show 2311 4433 itr mt -43 0 itr rt 0 57 itr rt (4) show 2311 4143 itr mt -43 0 itr rt 0 57 itr rt (5) show 4689 4724 itr mt -43 0 itr rt 0 57 itr rt (6) show 4689 4433 itr mt -43 0 itr rt 0 57 itr rt (7) show 4689 4143 itr mt -43 0 itr rt 0 57 itr rt (8) show 5164 4724 itr mt -43 0 itr rt 0 57 itr rt (9) show 5164 4433 itr mt -85 0 itr rt 0 57 itr rt (10) show 5164 4143 itr mt -85 0 itr rt 0 57 itr rt (11) show 1836 2981 itr mt -85 0 itr rt 0 57 itr rt (12) show 1836 2690 itr mt -85 0 itr rt 0 57 itr rt (13) show 1836 2400 itr mt -85 0 itr rt 0 57 itr rt (14) show 2311 2981 itr mt -85 0 itr rt 0 57 itr rt (15) show 2311 2690 itr mt -85 0 itr rt 0 57 itr rt (16) show 2311 2400 itr mt -85 0 itr rt 0 57 itr rt (17) show 4689 2981 itr mt -85 0 itr rt 0 57 itr rt (18) show 4689 2690 itr mt -85 0 itr rt 0 57 itr rt (19) show 4689 2400 itr mt -85 0 itr rt 0 57 itr rt (20) show 5164 2981 itr mt -85 0 itr rt 0 57 itr rt (21) show 5164 2690 itr mt -85 0 itr rt 0 57 itr rt (22) show 5164 2400 itr mt -85 0 itr rt 0 57 itr rt (23) show 2787 4724 itr mt -43 0 itr rt 0 57 itr rt (0) show 2787 4433 itr mt -43 0 itr rt 0 57 itr rt (1) show 2787 4143 itr mt -43 0 itr rt 0 57 itr rt (2) show 3262 4724 itr mt -43 0 itr rt 0 57 itr rt (3) show 3262 4433 itr mt -43 0 itr rt 0 57 itr rt (4) show 3262 4143 itr mt -43 0 itr rt 0 57 itr rt (5) show 2787 2981 itr mt -43 0 itr rt 0 57 itr rt (6) show 2787 2690 itr mt -43 0 itr rt 0 57 itr rt (7) show 2787 2400 itr mt -43 0 itr rt 0 57 itr rt (8) show 3262 2981 itr mt -43 0 itr rt 0 57 itr rt (9) show 3262 2690 itr mt -85 0 itr rt 0 57 itr rt (10) show 3262 2400 itr mt -85 0 itr rt 0 57 itr rt (11) show 3738 4724 itr mt -43 0 itr rt 0 57 itr rt (0) show 3738 4433 itr mt -43 0 itr rt 0 57 itr rt (1) show 3738 4143 itr mt -43 0 itr rt 0 57 itr rt (2) show 4213 4724 itr mt -43 0 itr rt 0 57 itr rt (3) show 4213 4433 itr mt -43 0 itr rt 0 57 itr rt (4) show 4213 4143 itr mt -43 0 itr rt 0 57 itr rt (5) show 3738 2981 itr mt -43 0 itr rt 0 57 itr rt (6) show 3738 2690 itr mt -43 0 itr rt 0 57 itr rt (7) show 3738 2400 itr mt -43 0 itr rt 0 57 itr rt (8) show 4213 2981 itr mt -43 0 itr rt 0 57 itr rt (9) show 4213 2690 itr mt -85 0 itr rt 0 57 itr rt (10) show 4213 2400 itr mt -85 0 itr rt 0 57 itr rt (11) show 1836 3852 itr mt -43 0 itr rt 0 57 itr rt (0) show 1836 3562 itr mt -43 0 itr rt 0 57 itr rt (1) show 1836 3271 itr mt -43 0 itr rt 0 57 itr rt (2) show 2311 3852 itr mt -43 0 itr rt 0 57 itr rt (3) show 2311 3562 itr mt -43 0 itr rt 0 57 itr rt (4) show 2311 3271 itr mt -43 0 itr rt 0 57 itr rt (5) show 4689 3852 itr mt -43 0 itr rt 0 57 itr rt (6) show 4689 3562 itr mt -43 0 itr rt 0 57 itr rt (7) show 4689 3271 itr mt -43 0 itr rt 0 57 itr rt (8) show 5164 3852 itr mt -43 0 itr rt 0 57 itr rt (9) show 5164 3562 itr mt -85 0 itr rt 0 57 itr rt (10) show 5164 3271 itr mt -85 0 itr rt 0 57 itr rt (11) show 1836 2109 itr mt -85 0 itr rt 0 57 itr rt (12) show 1836 1819 itr mt -85 0 itr rt 0 57 itr rt (13) show 1836 1528 itr mt -85 0 itr rt 0 57 itr rt (14) show 2311 2109 itr mt -85 0 itr rt 0 57 itr rt (15) show 2311 1819 itr mt -85 0 itr rt 0 57 itr rt (16) show 2311 1528 itr mt -85 0 itr rt 0 57 itr rt (17) show 4689 2109 itr mt -85 0 itr rt 0 57 itr rt (18) show 4689 1819 itr mt -85 0 itr rt 0 57 itr rt (19) show 4689 1528 itr mt -85 0 itr rt 0 57 itr rt (20) show 5164 2109 itr mt -85 0 itr rt 0 57 itr rt (21) show 5164 1819 itr mt -85 0 itr rt 0 57 itr rt (22) show 5164 1528 itr mt -85 0 itr rt 0 57 itr rt (23) show 2787 3852 itr mt -43 0 itr rt 0 57 itr rt (0) show 2787 3562 itr mt -43 0 itr rt 0 57 itr rt (1) show 2787 3271 itr mt -43 0 itr rt 0 57 itr rt (2) show 3262 3852 itr mt -43 0 itr rt 0 57 itr rt (3) show 3262 3562 itr mt -43 0 itr rt 0 57 itr rt (4) show 3262 3271 itr mt -43 0 itr rt 0 57 itr rt (5) show 2787 2109 itr mt -43 0 itr rt 0 57 itr rt (6) show 2787 1819 itr mt -43 0 itr rt 0 57 itr rt (7) show 2787 1528 itr mt -43 0 itr rt 0 57 itr rt (8) show 3262 2109 itr mt -43 0 itr rt 0 57 itr rt (9) show 3262 1819 itr mt -85 0 itr rt 0 57 itr rt (10) show 3262 1528 itr mt -85 0 itr rt 0 57 itr rt (11) show 3738 3852 itr mt -43 0 itr rt 0 57 itr rt (0) show 3738 3562 itr mt -43 0 itr rt 0 57 itr rt (1) show 3738 3271 itr mt -43 0 itr rt 0 57 itr rt (2) show 4213 3852 itr mt -43 0 itr rt 0 57 itr rt (3) show 4213 3562 itr mt -43 0 itr rt 0 57 itr rt (4) show 4213 3271 itr mt -43 0 itr rt 0 57 itr rt (5) show 3738 2109 itr mt -43 0 itr rt 0 57 itr rt (6) show 3738 1819 itr mt -43 0 itr rt 0 57 itr rt (7) show 3738 1528 itr mt -43 0 itr rt 0 57 itr rt (8) show 4213 2109 itr mt -43 0 itr rt 0 57 itr rt (9) show 4213 1819 itr mt -85 0 itr rt 0 57 itr rt (10) show 4213 1528 itr mt -85 0 itr rt 0 57 itr rt (11) show sv np 4927 4869 itr mt 4927 3126 itr l2 st rs sv np 4927 3127 itr mt 4927 1383 itr l2 st rs sv np 4927 4869 itr mt 4927 1383 itr l2 st rs sv np 5402 4869 itr mt 5402 3126 itr l2 st rs sv np 5402 3127 itr mt 5402 1383 itr l2 st rs sv np 5402 4869 itr mt 5402 1383 itr l2 st rs sv np 3977 4869 itr mt 3977 3126 itr l2 st rs sv np 3977 3127 itr mt 3977 1383 itr l2 st rs sv np 3977 4869 itr mt 3977 1383 itr l2 st rs sv np 3026 4869 itr mt 3026 3126 itr l2 st rs sv np 3026 3127 itr mt 3026 1383 itr l2 st rs sv np 3026 4869 itr mt 3026 1383 itr l2 st rs sv np 2075 4869 itr mt 2075 3126 itr l2 st rs sv np 2075 3127 itr mt 2075 1383 itr l2 st rs sv np 2075 4869 itr mt 2075 1383 itr l2 st rs sv np 1599 4869 itr mt 1599 3126 itr l2 st rs sv np 1599 3127 itr mt 1599 1383 itr l2 st rs sv np 1599 4869 itr mt 1599 1383 itr l2 st rs [191 0 0.00 191 -18 -142] /Flfon1 cfnt 5857 3730 itr mt 0 71 itr rt (ncells = 8) show 5857 3469 itr mt 0 71 itr rt (Hn = 3) show 5857 3209 itr mt 0 71 itr rt (Vn = 2) show 5857 2948 itr mt 0 71 itr rt (Hbs = 2) show 5857 2687 itr mt 0 71 itr rt (Vbs = 3) show sv np 1607 4870 itr mt 5415 4870 itr l2 st rs rs end %%EndDocument 75 4 a endTexFig 586 1002 a Ft(Figure)16 b(1.1:)j(P)o(artitioning)c(of)g(a)g(V)l(esta)g(File) 75 1134 y(where)21 b(0)i Fg(\024)f Fd(I)k(<)d(H)t(bs)d Ft(and)i(0)g Fg(\024)h Fd(H)j(<)c(H)t(n)p Ft(.)38 b(Consider)21 b(a)g(partition)h(of)e (the)h(matrix)g(in)o(to)g(t)o(w)o(o-)75 1190 y(dimensional)j(blo)q(c)o(ks,)f (eac)o(h)f(spanning)h Fd(V)9 b(bs)22 b Ft(consecutiv)o(e)g(ro)o(ws)f(and)h Fd(H)t(bs)f Ft(consecutiv)o(e)i(columns.)75 1247 y(Blo)q(c)o(k)c Fd(bl)q(ock)306 1254 y Fc(r)o(;s)365 1247 y Ft(spans)g(the)g Fd(V)10 b(bs)18 b Ft(ro)o(ws)g Fd(r)13 b Fg(\001)f Fd(V)d(bs;)f(:)g(:)g(:)d (;)j Ft(\()p Fd(r)k Ft(+)h(1\))f Fg(\001)g Fd(V)d(bs)k Fg(\000)f Ft(1)19 b(and)g(the)f Fd(H)t(bs)g Ft(columns)i Fd(s)12 b Fg(\001)75 1303 y Fd(H)t(bs;)c(:)g(:)g(:)t(;)g Ft(\()p Fd(s)i Ft(+)h(1\))f Fg(\001)g Fd(H)t(bs)g Fg(\000)h Ft(1.)23 b(If)16 b Fd(I)i(>)c Ft(0)i(then)g(the)g(last)g(blo)q(c)o(k)h(in)g(the)f(horizon)o(tal)h (dimension)h(\(blo)q(c)o(k)75 1360 y Fd(bl)q(ock)176 1367 y Fc(r)o(;B)q Fb(\001)p Fc(H)r(n)p Fa(+)p Fc(H)342 1360 y Ft(\))d(spans)g Fd(I)k Ft(columns)d(only)l(.)166 1416 y(Sub\014le)h Fd(par)q(t)400 1423 y Fc(v)q(;h)464 1416 y Ft(consists)f(of)e(blo)q(c)o(ks)i Fd(bl)q(ock)922 1423 y Fc(r)o(;s)963 1416 y Ft(,)e(for)701 1510 y Fd(r)f Ft(=)g Fd(v)r(;)8 b(v)j Ft(+)f Fd(V)g(n;)e(v)j Ft(+)f(2)p Fd(V)g(n;)e(:)g(:)g(:)d(;)75 1604 y Ft(and)692 1661 y Fd(s)13 b Ft(=)g Fd(h;)8 b(h)i Ft(+)g Fd(H)t(n;)e(h)h Ft(+)i(2)p Fd(H)t(n;)d(:)g(:)g(:)t(:)166 1740 y Ft(The)15 b(n)o(um)o(b)q(er)h(of)f (columns)h(in)g(sub\014le)g Fd(par)q(t)933 1747 y Fc(v)q(;h)998 1740 y Ft(is)f(equal)h(to)644 1893 y Fd(c)664 1900 y Fc(h)698 1893 y Ft(=)746 1795 y Fe(8)746 1832 y(>)746 1844 y(<)746 1919 y(>)746 1932 y(:)803 1837 y Ft(\()p Fd(B)d Ft(+)d(1\))g Fg(\001)g Fd(H)t(bs)40 b Ft(if)16 b Fd(h)d(<)g(H)803 1893 y(B)g Fg(\001)d Fd(H)t(bs)f Ft(+)i Fd(I)79 b Ft(if)16 b Fd(h)d Ft(=)g Fd(H)803 1950 y(B)g Fg(\001)d Fd(H)t(bs)154 b Ft(if)16 b Fd(h)d(>)g(H)166 2045 y Ft(When)i Fd(H)t(bs)10 b Fg(\001)g Fd(H)t(n)15 b Ft(divides)i(ev)o (enly)f Fd(ncel)q(l)q(s)e Ft(this)i(reduces)g(to)e Fd(c)1250 2052 y Fc(h)1284 2045 y Ft(=)f Fd(ncel)q(l)q(s=H)t(n)p Ft(.)166 2101 y(Consider)i(record)f(\()p Fd(i;)8 b(j)s Ft(\))k(in)k(blo)q(c)o(k)f Fd(bl)q(ock)873 2108 y Fc(r)o(;s)914 2101 y Ft(,)f(where)h Fd(r)e Ft(=)g Fd(v)d Ft(+)f Fd(m)g Fg(\001)g Fd(V)g(n)15 b Ft(and)g Fd(s)e Ft(=)g Fd(h)8 b Ft(+)i Fd(n)f Fg(\001)f Fd(H)t(n)p Ft(.)19 b(This)75 2158 y(record)e(has)f(indices)j(\()p Fd(i)11 b Ft(+)g Fd(r)h Fg(\001)f Fd(V)f(bs;)e(j)k Ft(+)g Fd(s)f Fg(\001)g Fd(H)t(bs)p Ft(\))16 b(in)h(the)g(global)h(\014le.)25 b(It)17 b(b)q(elongs)h(to)e(sub\014le)i Fd(par)q(t)1813 2165 y Fc(v)q(;h)1862 2158 y Ft(,)75 2214 y(and)d(has)g(indices)j(\()p Fd(i)9 b Ft(+)i Fd(m)f Fg(\001)f Fd(V)h(bs;)e(j)j Ft(+)g Fd(n)f Fg(\001)g Fd(H)t(bs)p Ft(\))k(in)i(the)f(sub\014le.)166 2271 y(W)l(e)20 b(further)g(need)i(to)d(sp) q(ecify)j(a)e(canonical)h(order)f(for)g(records)g(within)i(eac)o(h)e (sub\014le.)36 b(Since)75 2327 y(the)17 b(v)o(ertical)h(dimension)g(of)f(the) g(sub\014le)h(is)g(un)o(b)q(ounded,)g(one)f(cannot)g(use)g(a)g(column-ma)s (jor)g(order.)75 2384 y(Ho)o(w)o(ev)o(er,)f(a)h(\\column)g(\014rst")f (ordering)i(has)e(the)i(adv)m(an)o(tage)e(that)g(consecutiv)o(e)i(records)f (are)f(stored)75 2440 y(con)o(tiguously)l(,)e(th)o(us)f(impro)o(ving)h(lo)q (calit)o(y)g(of)f(access.)20 b(The)13 b(ordering)h(used)g(is)f(a)g (compromise)h(b)q(et)o(w)o(een)75 2497 y(these)f(t)o(w)o(o)e(con\015icting)k (requiremen)o(ts:)k(Within)14 b(eac)o(h)f(blo)q(c)o(k,)g(records)g(are)f (ordered)h(in)h(column)f(ma)s(jor)75 2553 y(order;)g(the)g(blo)q(c)o(ks)g (themselv)o(es)h(are)e(ordered)h(in)h(ro)o(w)e(ma)s(jor)f(order.)19 b(Th)o(us,)12 b(using)i(the)f(same)f(notation)75 2609 y(as)j(b)q(efore,)g (the)g(record)g(with)h(indices)h(\()p Fd(i)10 b Ft(+)g Fd(m)g Fg(\001)g Fd(V)f(bs;)f(j)k Ft(+)e Fd(n)h Fg(\001)e Fd(H)t(bs)p Ft(\))14 b(in)j(the)e(sub\014le)h Fd(par)q(t)1619 2616 y Fc(v)q(;h)1684 2609 y Ft(has)f(rank)580 2704 y Fd(m)10 b Fg(\001)f Fd(c)672 2711 y Fc(h)704 2704 y Fg(\001)g Fd(V)h(bs)g Ft(+)g Fd(n)h Fg(\001)e Fd(H)t(bs)h Fg(\001)g Fd(V)f(bs)h Ft(+)h Fd(j)h Fg(\001)e Fd(V)f(bs)h Ft(+)h Fd(i)1967 46 y Fr(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 21 22 bop 75 -100 a Fp(1.8.)34 b(VEST)l(A)16 b(FUNCTIONS)1206 b Ft(21)75 4 y 19608222 14208860 0 0 42626580 30785863 startTexFig 75 4 a %%BeginDocument: vst-par2.eps /Freelance_Plus dup 100 dict def load begin [ ] {bind} stopped { (patching the bind operator...) = flush /*bind /bind load def /bind { dup xcheck { *bind } if } *bind def } if pop /bdf {bind def} bind def /ldf {load def} bdf /mt /moveto ldf /rt /rmoveto ldf /l2 /lineto ldf /c2 /curveto ldf /sg /setgray ldf /gs /gsave ldf /ef /eofill ldf /rl2 /rlineto ldf /st /stroke ldf /gr /grestore ldf /np /newpath ldf /sv /save ldf /su /statusdict ldf /rs /restore ldf /sw /setlinewidth ldf /sd /setdash ldf /cp /closepath ldf /ed {exch def } bdf /cfnt {findfont exch makefont setfont} bdf /itr {transform round exch round exch itransform} bdf /fres 72 0 matrix currentmatrix dtransform exch dup mul exch dup mul add sqrt def /res fres def /mcm matrix currentmatrix bdf end Freelance_Plus begin save newpath .1 .1 scale /ecm matrix currentmatrix bdf /sem {ecm setmatrix} bdf -720 -720 translate 0 setlinecap 0 setlinejoin 1.42 setmiterlimit 0.000 sg 6 sw sv np [] 0 sd 1598 3126 itr mt 5402 3126 itr l2 5402 4869 itr l2 1598 4869 itr l2 1598 3126 itr l2 cp st rs /Ich 256 array def StandardEncoding Ich copy pop /bullet/paragraph/section/dieresis/tilde/ring /circumflex/grave/acute/quotedblleft/quotesingle /ellipsis/endash/emdash/guilsinglleft/guilsinglright /quotedblbase/quotesinglbase/quotedblright/OE/oe /Ydieresis/fi/fl/dagger/daggerdbl/Ccedilla/udieresis /eacute/acircumflex/adieresis/agrave/aring/ccedilla /ecircumflex/edieresis/egrave/idieresis/icircumflex /igrave/Adieresis/Aring/Eacute/ae/AE/ocircumflex /odieresis/ograve/ucircumflex/ugrave/ydieresis /Odieresis/Udieresis/oslash/sterling/Oslash/florin /aacute/iacute/oacute/uacute/ntilde/Ntilde/ordfeminine /ordmasculine/questiondown/exclamdown/guillemotleft /guillemotright/Aacute/Acircumflex/Agrave/cent/yen /atilde/Atilde/currency/Ecircumflex/Edieresis/Egrave /dotlessi/Iacute/Icircumflex/Idieresis/Igrave/Oacute /germandbls/Ocircumflex/Ograve/otilde/Otilde/Uacute /Ucircumflex/Ugrave/macron/cedilla/periodcentered Ich 127 97 getinterval astore pop /Ienc { /ncs Ich def /nfn ed /bfn ed /bfd bfn findfont def /nf bfd maxlength dict def bfd{exch dup dup /FID ne exch /Encoding ne and {exch nf 3 1 roll put}{pop pop} ifelse }forall nf/FontName nfn put nf/Encoding ncs put nfn nf definefont pop}bdf /IencO { /ncs Ich def /nfn ed /bfn ed /lnw ed /bfd bfn findfont def /nf bfd maxlength 4 add dict def bfd{exch dup dup /FID ne exch /Encoding ne and {exch nf 3 1 roll put}{pop pop} ifelse }forall nf/FontName nfn put nf/Encoding ncs put nf/PaintType 2 put nf/StrokeWidth lnw put nfn nf definefont pop}bdf /IencSO { /nfn ed /bfn ed /lnw ed /bfd bfn findfont def /nf bfd maxlength 4 add dict def bfd{exch dup /FID ne { exch nf 3 1 roll put}{pop pop} ifelse }forall nf/FontName nfn put nf/PaintType 2 put nf/StrokeWidth lnw put nfn nf definefont pop}bdf /Helvetica /Flfon1 Ienc [191 0 0.00 191 -18 -142] /Flfon1 cfnt 1843 5009 itr mt -53 0 itr rt 0 71 itr rt (0) show 2311 5014 itr mt -53 0 itr rt 0 71 itr rt (1) show 2787 5014 itr mt -53 0 itr rt 0 71 itr rt (2) show 3262 5014 itr mt -53 0 itr rt 0 71 itr rt (3) show 3738 5014 itr mt -53 0 itr rt 0 71 itr rt (4) show 4213 5014 itr mt -53 0 itr rt 0 71 itr rt (5) show 4689 5014 itr mt -53 0 itr rt 0 71 itr rt (6) show 5164 5014 itr mt -53 0 itr rt 0 71 itr rt (7) show 1360 4724 itr mt -53 0 itr rt 0 71 itr rt (0) show 1360 4433 itr mt -53 0 itr rt 0 71 itr rt (1) show 1360 4143 itr mt -53 0 itr rt 0 71 itr rt (2) show 1360 3852 itr mt -53 0 itr rt 0 71 itr rt (3) show 1360 3562 itr mt -53 0 itr rt 0 71 itr rt (4) show 1360 3271 itr mt -53 0 itr rt 0 71 itr rt (5) show 12 sw sv np 2549 4869 itr mt 2549 3126 itr l2 st rs sv np 3500 4869 itr mt 3500 3126 itr l2 st rs sv np 4451 4869 itr mt 4451 3126 itr l2 st rs sv np [] 0 sd 1598 3998 itr mt 2549 3998 itr l2 2549 4869 itr l2 1598 4869 itr l2 1598 3998 itr l2 gs 0.920 sg ef gr rs sv np [] 0 sd 4451 3998 itr mt 5402 3998 itr l2 5402 4869 itr l2 4451 4869 itr l2 4451 3998 itr l2 gs 0.920 sg ef gr rs /fi16 {0 16 2} def /bitval {/yb ed /xb ed bs yb bw mul xb 8 idiv add get 1 7 xb 8 mod sub bitshift and 0 ne} bdf /spotf {/y ed /x ed x 1 add 2 div bps mul cvi y 1 add 2 div bps mul cvi bitval {/onb onb 1 add def 1} {/ofb ofb 1 add def 0} ifelse} def /uset {/uf ed /ua ed /uc ed uc dup matrix scale ua matrix rotate mcm mcm concatmatrix mcm concatmatrix pop 1 0 mcm dtransform /y1 ed /x1 ed /nsc y1 x1 atan def res x1 dup mul y1 dup mul add sqrt div nsc /uf load setscreen} bdf /mkfill {/cz ed /bs ed /bw ed /bps ed /an ed /onb 0 def /ofb 0 def cz an /spotf load uset {} settransfer ofb ofb onb add div sg} bdf sv np [] 0 sd 2549 3998 itr mt 3500 3998 itr l2 3500 4869 itr l2 2549 4869 itr l2 2549 3998 itr l2 gs fi16 < 0001800240042008101008200440028001000280044008201010200840048002 > 03.84 mkfill ef gr rs sv np [] 0 sd 3500 3998 itr mt 4451 3998 itr l2 4451 4869 itr l2 3500 4869 itr l2 3500 3998 itr l2 gs 0.850 sg ef gr rs sv np [] 0 sd 2549 3126 itr mt 3500 3126 itr l2 3500 3998 itr l2 2549 3998 itr l2 2549 3126 itr l2 gs fi16 < 8000400020001000080004000200010000800040002000100008000400020001 > 03.84 mkfill ef gr rs sv np [] 0 sd 1598 3126 itr mt 2549 3126 itr l2 2549 3998 itr l2 1598 3998 itr l2 1598 3126 itr l2 gs fi16 < 0001000200040008001000200040008001000200040008001000200040008000 > 03.84 mkfill ef gr rs 6 sw sv np [] 0 sd 4451 3126 itr mt 5402 3126 itr l2 5402 3998 itr l2 4451 3998 itr l2 4451 3126 itr l2 gs fi16 < 0001000200040008001000200040008001000200040008001000200040008000 > 03.84 mkfill ef gr rs 12 sw sv np [] 0 sd 3500 3126 itr mt 4451 3126 itr l2 4451 3998 itr l2 3500 3998 itr l2 3500 3126 itr l2 rs 6 sw sv np [] 0 sd 1598 1383 itr mt 5402 1383 itr l2 5402 3126 itr l2 1598 3126 itr l2 1598 1383 itr l2 rs 12 sw sv np [] 0 sd 2549 3127 itr mt 2549 1383 itr l2 st rs sv np 3500 3127 itr mt 3500 1383 itr l2 st rs sv np 4451 3127 itr mt 4451 1383 itr l2 st rs sv np [] 0 sd 1598 2255 itr mt 2549 2255 itr l2 2549 3126 itr l2 1598 3126 itr l2 1598 2255 itr l2 gs 0.920 sg ef gr rs 6 sw sv np [] 0 sd 4451 2255 itr mt 5402 2255 itr l2 5402 3126 itr l2 4451 3126 itr l2 4451 2255 itr l2 gs 0.920 sg ef gr rs 12 sw sv np [] 0 sd 2549 2255 itr mt 3500 2255 itr l2 3500 3126 itr l2 2549 3126 itr l2 2549 2255 itr l2 gs fi16 < 0001800240042008101008200440028001000280044008201010200840048002 > 03.84 mkfill ef gr rs sv np [] 0 sd 3500 2255 itr mt 4451 2255 itr l2 4451 3126 itr l2 3500 3126 itr l2 3500 2255 itr l2 gs 0.850 sg ef gr rs sv np [] 0 sd 2549 1383 itr mt 3500 1383 itr l2 3500 2255 itr l2 2549 2255 itr l2 2549 1383 itr l2 gs fi16 < 8000400020001000080004000200010000800040002000100008000400020001 > 03.84 mkfill ef gr rs sv np [] 0 sd 1598 1383 itr mt 2549 1383 itr l2 2549 2255 itr l2 1598 2255 itr l2 1598 1383 itr l2 gs fi16 < 0001000200040008001000200040008001000200040008001000200040008000 > 03.84 mkfill ef gr rs sv np [] 0 sd 4451 1383 itr mt 5402 1383 itr l2 5402 2255 itr l2 4451 2255 itr l2 4451 1383 itr l2 gs fi16 < 0001000200040008001000200040008001000200040008001000200040008000 > 03.84 mkfill ef gr rs sv np [] 0 sd 3500 1383 itr mt 4451 1383 itr l2 4451 2255 itr l2 3500 2255 itr l2 3500 1383 itr l2 rs 1360 2981 itr mt -53 0 itr rt 0 71 itr rt (6) show 1360 2690 itr mt -53 0 itr rt 0 71 itr rt (7) show 1360 2400 itr mt -53 0 itr rt 0 71 itr rt (8) show 1360 2109 itr mt -53 0 itr rt 0 71 itr rt (9) show 1360 1819 itr mt -106 0 itr rt 0 71 itr rt (10) show 1360 1528 itr mt -106 0 itr rt 0 71 itr rt (11) show 1360 1238 itr mt -27 0 itr rt 0 71 itr rt (.) show 1360 947 itr mt -27 0 itr rt 0 71 itr rt (.) show 1360 1093 itr mt -27 0 itr rt 0 71 itr rt (.) show sv np [] 0 sd 2549 4869 itr mt 2549 1383 itr l2 st rs sv np 3500 4869 itr mt 3500 1383 itr l2 st rs sv np 4451 4869 itr mt 4451 1383 itr l2 st rs [153 0 0.00 153 -15 -113] /Flfon1 cfnt 1836 4724 itr mt -43 0 itr rt 0 57 itr rt (0) show 1825 4460 itr mt -43 0 itr rt 0 57 itr rt (1) show 1836 4143 itr mt -43 0 itr rt 0 57 itr rt (2) show 2311 4724 itr mt -43 0 itr rt 0 57 itr rt (3) show 2311 4433 itr mt -43 0 itr rt 0 57 itr rt (4) show 2311 4143 itr mt -43 0 itr rt 0 57 itr rt (5) show 4689 4724 itr mt -43 0 itr rt 0 57 itr rt (6) show 4689 4433 itr mt -43 0 itr rt 0 57 itr rt (7) show 4689 4143 itr mt -43 0 itr rt 0 57 itr rt (8) show 5164 4724 itr mt -43 0 itr rt 0 57 itr rt (9) show 5164 4433 itr mt -85 0 itr rt 0 57 itr rt (10) show 5164 4143 itr mt -85 0 itr rt 0 57 itr rt (11) show 1836 2981 itr mt -85 0 itr rt 0 57 itr rt (12) show 1836 2690 itr mt -85 0 itr rt 0 57 itr rt (13) show 1836 2400 itr mt -85 0 itr rt 0 57 itr rt (14) show 2311 2981 itr mt -85 0 itr rt 0 57 itr rt (15) show 2311 2690 itr mt -85 0 itr rt 0 57 itr rt (16) show 2311 2400 itr mt -85 0 itr rt 0 57 itr rt (17) show 4689 2981 itr mt -85 0 itr rt 0 57 itr rt (18) show 4689 2690 itr mt -85 0 itr rt 0 57 itr rt (19) show 4689 2400 itr mt -85 0 itr rt 0 57 itr rt (20) show 5164 2981 itr mt -85 0 itr rt 0 57 itr rt (21) show 5164 2690 itr mt -85 0 itr rt 0 57 itr rt (22) show 5164 2400 itr mt -85 0 itr rt 0 57 itr rt (23) show 2787 4724 itr mt -43 0 itr rt 0 57 itr rt (0) show 2787 4433 itr mt -43 0 itr rt 0 57 itr rt (1) show 2787 4143 itr mt -43 0 itr rt 0 57 itr rt (2) show 3262 4724 itr mt -43 0 itr rt 0 57 itr rt (3) show 3262 4433 itr mt -43 0 itr rt 0 57 itr rt (4) show 3262 4143 itr mt -43 0 itr rt 0 57 itr rt (5) show 2787 2981 itr mt -85 0 itr rt 0 57 itr rt (12) show 2787 2690 itr mt -85 0 itr rt 0 57 itr rt (13) show 2787 2400 itr mt -85 0 itr rt 0 57 itr rt (14) show 3262 2981 itr mt -85 0 itr rt 0 57 itr rt (15) show 3262 2690 itr mt -85 0 itr rt 0 57 itr rt (16) show 3262 2400 itr mt -85 0 itr rt 0 57 itr rt (17) show 3738 4724 itr mt -43 0 itr rt 0 57 itr rt (0) show 3738 4433 itr mt -43 0 itr rt 0 57 itr rt (1) show 3738 4143 itr mt -43 0 itr rt 0 57 itr rt (2) show 4213 4724 itr mt -43 0 itr rt 0 57 itr rt (3) show 4213 4433 itr mt -43 0 itr rt 0 57 itr rt (4) show 4213 4143 itr mt -43 0 itr rt 0 57 itr rt (5) show 3738 2981 itr mt -85 0 itr rt 0 57 itr rt (12) show 3738 2690 itr mt -85 0 itr rt 0 57 itr rt (13) show 3738 2400 itr mt -85 0 itr rt 0 57 itr rt (14) show 4213 2981 itr mt -85 0 itr rt 0 57 itr rt (15) show 4213 2690 itr mt -85 0 itr rt 0 57 itr rt (16) show 4213 2400 itr mt -85 0 itr rt 0 57 itr rt (17) show 1836 3852 itr mt -43 0 itr rt 0 57 itr rt (0) show 1836 3562 itr mt -43 0 itr rt 0 57 itr rt (1) show 1836 3271 itr mt -43 0 itr rt 0 57 itr rt (2) show 2311 3852 itr mt -43 0 itr rt 0 57 itr rt (3) show 2311 3562 itr mt -43 0 itr rt 0 57 itr rt (4) show 2311 3271 itr mt -43 0 itr rt 0 57 itr rt (5) show 4689 3852 itr mt -43 0 itr rt 0 57 itr rt (6) show 4689 3562 itr mt -43 0 itr rt 0 57 itr rt (7) show 4689 3271 itr mt -43 0 itr rt 0 57 itr rt (8) show 5164 3852 itr mt -43 0 itr rt 0 57 itr rt (9) show 5164 3562 itr mt -85 0 itr rt 0 57 itr rt (10) show 5164 3271 itr mt -85 0 itr rt 0 57 itr rt (11) show 1836 2109 itr mt -85 0 itr rt 0 57 itr rt (12) show 1836 1819 itr mt -85 0 itr rt 0 57 itr rt (13) show 1836 1528 itr mt -85 0 itr rt 0 57 itr rt (14) show 2311 2109 itr mt -85 0 itr rt 0 57 itr rt (15) show 2311 1819 itr mt -85 0 itr rt 0 57 itr rt (16) show 2311 1528 itr mt -85 0 itr rt 0 57 itr rt (17) show 4689 2109 itr mt -85 0 itr rt 0 57 itr rt (18) show 4689 1819 itr mt -85 0 itr rt 0 57 itr rt (19) show 4689 1528 itr mt -85 0 itr rt 0 57 itr rt (20) show 5164 2109 itr mt -85 0 itr rt 0 57 itr rt (21) show 5164 1819 itr mt -85 0 itr rt 0 57 itr rt (22) show 5164 1528 itr mt -85 0 itr rt 0 57 itr rt (23) show 2787 3852 itr mt -43 0 itr rt 0 57 itr rt (0) show 2787 3562 itr mt -43 0 itr rt 0 57 itr rt (1) show 2787 3271 itr mt -43 0 itr rt 0 57 itr rt (2) show 3262 3852 itr mt -43 0 itr rt 0 57 itr rt (3) show 3262 3562 itr mt -43 0 itr rt 0 57 itr rt (4) show 3262 3271 itr mt -43 0 itr rt 0 57 itr rt (5) show 2787 2109 itr mt -85 0 itr rt 0 57 itr rt (12) show 2787 1819 itr mt -85 0 itr rt 0 57 itr rt (13) show 2787 1528 itr mt -85 0 itr rt 0 57 itr rt (14) show 3262 2109 itr mt -85 0 itr rt 0 57 itr rt (15) show 3262 1819 itr mt -85 0 itr rt 0 57 itr rt (16) show 3262 1528 itr mt -85 0 itr rt 0 57 itr rt (17) show 3738 3852 itr mt -43 0 itr rt 0 57 itr rt (0) show 3738 3562 itr mt -43 0 itr rt 0 57 itr rt (1) show 3738 3271 itr mt -43 0 itr rt 0 57 itr rt (2) show 4213 3852 itr mt -43 0 itr rt 0 57 itr rt (3) show 4213 3562 itr mt -43 0 itr rt 0 57 itr rt (4) show 4213 3271 itr mt -43 0 itr rt 0 57 itr rt (5) show 3738 2109 itr mt -85 0 itr rt 0 57 itr rt (12) show 3738 1819 itr mt -85 0 itr rt 0 57 itr rt (13) show 3738 1528 itr mt -85 0 itr rt 0 57 itr rt (14) show 4213 2109 itr mt -85 0 itr rt 0 57 itr rt (15) show 4213 1819 itr mt -85 0 itr rt 0 57 itr rt (16) show 4213 1528 itr mt -85 0 itr rt 0 57 itr rt (17) show sv np 4927 4869 itr mt 4927 3126 itr l2 st rs sv np 4927 3127 itr mt 4927 1383 itr l2 st rs sv np 4927 4869 itr mt 4927 1383 itr l2 st rs sv np 5402 4869 itr mt 5402 3126 itr l2 st rs sv np 5402 3127 itr mt 5402 1383 itr l2 st rs sv np 5402 4869 itr mt 5402 1383 itr l2 st rs sv np 3977 4869 itr mt 3977 3126 itr l2 st rs sv np 3977 3127 itr mt 3977 1383 itr l2 st rs sv np 3977 4869 itr mt 3977 1383 itr l2 st rs sv np 3026 4869 itr mt 3026 3126 itr l2 st rs sv np 3026 3127 itr mt 3026 1383 itr l2 st rs sv np 3026 4869 itr mt 3026 1383 itr l2 st rs sv np 2075 4869 itr mt 2075 3126 itr l2 st rs sv np 2075 3127 itr mt 2075 1383 itr l2 st rs sv np 2075 4869 itr mt 2075 1383 itr l2 st rs sv np 1599 4869 itr mt 1599 3126 itr l2 st rs sv np 1599 3127 itr mt 1599 1383 itr l2 st rs sv np 1599 4869 itr mt 1599 1383 itr l2 st rs [191 0 0.00 191 -18 -142] /Flfon1 cfnt 5857 3730 itr mt 0 71 itr rt (ncells = 8) show 5857 3469 itr mt 0 71 itr rt (Hn = 3) show 5857 3209 itr mt 0 71 itr rt (Vn = 2) show 5857 2948 itr mt 0 71 itr rt (Hbs = 2) show 5857 2687 itr mt 0 71 itr rt (Vbs = 3) show sv np 1607 4870 itr mt 5415 4870 itr l2 st rs rs end %%EndDocument 75 4 a endTexFig 467 1002 a Ft(Figure)15 b(1.2:)k(Alternativ)o(e)d(P)o(artitioning)f(of)g(a)g (V)l(esta)g(File)75 1140 y(in)h(the)f(sub\014le.)166 1196 y(This)h(n)o(um)o (b)q(ering)g(sc)o(heme)f(is)h(illustrated)h(in)f(Figure)f(1.1.)166 1253 y(Example)h(1:)j Fd(H)t(bs)12 b Ft(=)h(1,)i Fd(H)t(n)d Ft(=)h Fd(ncel)q(l)q(s)p Ft(,)h Fd(V)c(n)j Ft(=)g(1.)75 1309 y(Then)k(eac)o(h)g(sub\014le)h(spans)f(one)f(cell.)26 b(The)17 b(records)f(in)i(eac)o(h)f(sub\014le)h(are)e(ordered)h(in)g(column)h(index)75 1366 y(order.)166 1422 y(Example)e(2:)j Fd(H)t(bs)12 b Ft(=)h Fd(ncel)q(l)q(s)p Ft(,)h Fd(H)t(n)f Ft(=)g Fd(V)c(n)k Ft(=)g(1.)75 1479 y(Then)18 b(the)h(unique)g(sub\014le)g(spans)f(the)h(en)o(tire)f(V)l (esta)g(\014le;)i(the)e(records)g(are)f(ordered)h(in)h(ro)o(w-ma)s(jor)75 1535 y(order,)d(if)h Fd(V)9 b(bs)15 b Ft(=)g(1;)h(di\013eren)o(t)g(\\snak)o (e-lik)o(e")h(orderings)g(can)f(b)q(e)h(obtained)g(b)o(y)g(v)m(arying)f(the)h (v)m(alue)g(of)75 1592 y Fd(V)10 b(bs)p Ft(.)75 1713 y Fm(Alternative)16 b(De\014nition)75 1799 y Ft(The)d(curren)o(t)g(V)l(esta)g(do)q(cumen)o(t)g (sp)q(eci\014es)i(another)d(partitioning)i(sc)o(heme:)19 b(The)13 b(n)o(um)o(b)q(er)h(of)e(columns)75 1856 y(in)18 b(the)f(V)l(esta)g(\014le)h (is)f(extended)h(to)f(b)q(e)g(an)g(ev)o(en)g(m)o(ultiple)i(of)e Fd(H)t(n)11 b Fg(\001)g Fd(H)t(bs)p Ft(,)16 b(and)h(the)g(expanded)h(\014le)g (is)75 1912 y(then)g(distributed,)h(and)f(the)f(records)g(n)o(um)o(b)q(ered,) i(as)e(describ)q(ed)i(ab)q(o)o(v)o(e.)27 b(The)17 b(additional)i(columns)75 1969 y(are)d(actually)h(empt)o(y:)22 b(all)17 b(the)g(records)f(that)f(b)q (elong)j(to)d(these)i(columns)g(are)f(empt)o(y)g(records.)23 b(Suc)o(h)75 2025 y(record)15 b(cannot)g(b)q(e)h(written;)f(a)g(read)g(of)g (suc)o(h)g(record)g(returns)g(zero)g(b)o(ytes.)166 2082 y(This)20 b(de\014nition)i(is)e(illustrated)h(in)g(Figure)f(1.2.)32 b(The)20 b(sub\014les)h Fd(par)q(t)1410 2089 y Fa(0)p Fc(;)p Fa(0)1475 2082 y Ft(and)f Fd(par)q(t)1653 2089 y Fa(1)p Fc(;)p Fa(0)1718 2082 y Ft(ha)o(v)o(e)f(no)75 2138 y(\\holes")e(in)g(them.)25 b(The)17 b(sub\014les)h Fd(par)q(t)770 2145 y Fc(i;j)809 2138 y Ft(,)f(for)f Fd(i)e Ft(=)i(0)p Fd(;)8 b Ft(1)15 b(and)i Fd(j)g Ft(=)f(1)p Fd(;)8 b Ft(2)15 b(all)j(miss)f(the)g(records)f(6)h({)f(11,)75 2195 y(18)f({)f(23,)h(etc.)166 2252 y(Note)e(that,)f(with)h(this)h (de\014nition,)h(it)e(is)g(not)g(required)h(an)o(ymore)e(that)h(\()p Fd(H)t(n)6 b Fg(\000)g Ft(1\))g Fg(\001)g Fd(H)t(b)o(s)k(<)j(ncel)q(l)q(s)p Ft(.)75 2308 y(If)i(\()p Fd(H)t(n)10 b Fg(\000)g Ft(1\))g Fg(\001)g Fd(H)t(bs)i Fg(\025)h Fd(ncel)q(l)q(s)p Ft(,)h(then)h(the)h(partition)f(ma)o (y)g(con)o(tain)g(empt)o(y)g(sub\014les.)166 2365 y(Both)j(de\014nitions)j (asso)q(ciate)d(the)h(same)f(records)h(with)g(the)g(same)f(sub\014les.)32 b(Both)18 b(de\014nitions)75 2421 y(pro)o(vide)g(the)f(same)g(record)g(n)o (um)o(b)q(ering)h(when)f Fd(H)t(n)12 b Fg(\001)f Fd(H)t(bs)16 b Ft(ev)o(enly)i(divides)h Fd(ncel)q(l)q(s)p Ft(.)25 b(Otherwise,)18 b(the)75 2478 y(second)12 b(de\014nition)i(results)f(in)f(sub\014les)i(with)e (\\holes")g(in)h(them,)f(non-existing)h(records.)19 b(On)12 b(the)g(other)75 2534 y(hand,)i(the)h(corresp)q(ondence)g(b)q(et)o(w)o(een)g (the)f(ro)o(w)f(and)i(column)g(indices)h(of)e(a)g(record)g(and)g(its)h (sequence)75 2590 y(n)o(um)o(b)q(er)h(is)f(the)h(same)e(for)h(eac)o(h)g (sub\014le)i(when)f(the)f(second)h(de\014nition)h(is)e(used.)166 2647 y(W)l(e)f(assume)f(in)h(the)g(sequel)h(the)e(\014rst)g(de\014nition)j (of)d(a)g(V)l(esta)g(partition,)h(to)f(a)o(v)o(oid)g(the)h(messiness)75 2704 y(of)e(sequen)o(tial)i(accesses)f(to)f(\014les)i(with)f(\\holes".)19 b(But)12 b(use)h(of)g(suc)o(h)g(de\014nition)h(somewhat)e(complicates)-32 46 y Fr(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 22 23 bop 75 -100 a Ft(22)1387 b Fp(CHAPTER)16 b(1.)34 b(IO)75 49 y Ft(V)l(esta)15 b(implemen)o(tation.)166 232 y Fl(Discussion:)34 b(Cho)q(ose)15 b(an)h(alternativ)o(e.)75 687 y Fo(1.8.2)49 b(V)o(esta)16 b(\014le)g(op)q(erations)75 905 y Fm(MPI)p 160 905 14 2 v 16 w(GET)p 264 905 V 17 w(PFS)p 361 905 V 16 w(STORA)o(GE\(numb)q (er\))117 1026 y Fk(OUT)108 b Fm(numb)q(er)442 b Fk(n)o(um)o(b)q(er)13 b(of)h(storage)g(no)q(des)h(\(in)o(teger\))75 1194 y Fh(int)47 b(MPI)p 269 1194 15 2 v 17 w(Get)p 358 1194 V 17 w(pfs)p 447 1194 V 17 w(storage\()22 b(int)i(*number\))75 1325 y(MPI)p 150 1325 V 17 w(GET)p 239 1325 V 17 w(PFS)p 328 1325 V 16 w(STORAGE\()f (NUMBER,)g(IERROR\))170 1381 y(INTEGER)g(NUMBER,)g(IERROR)166 1511 y Ft(Returns)16 b(the)f(n)o(um)o(b)q(er)g(of)g(storage)f(no)q(des.)75 1659 y Fm(MPI)p 160 1659 14 2 v 16 w(GET)p 264 1659 V 17 w(PFS)p 361 1659 V 16 w(STRUCTURE\(\014lename,)g(ncells,)i(rec)p 1031 1659 V 16 w(size\))117 1780 y Fk(IN)155 b Fm(\014lename)428 b Fk(name)13 b(of)g(queried)i(\014le)f(\(string\))117 1943 y(OUT)108 b Fm(ncells)485 b Fk(n)o(um)o(b)q(er)13 b(of)h(cells)g(\(in)o (teger\))117 2105 y(OUT)108 b Fm(rec)p 377 2105 V 17 w(size)446 b Fk(record)15 b(size)g(\(in)o(teger\))75 2274 y Fh(int)47 b(MPI)p 269 2274 15 2 v 17 w(Get)p 358 2274 V 17 w(pfs)p 447 2274 V 17 w(structure\(const)21 b(char)j(*filename,)e(int)i(*ncells,)e(int)i (*rec)p 1729 2274 V 17 w(size\))75 2404 y(MPI)p 150 2404 V 17 w(GET)p 239 2404 V 17 w(PFS)p 328 2404 V 16 w(STRUCTURE\()f(FILENAME,)f (NCELLS,)h(REC)p 1108 2404 V 17 w(SIZE,)g(IERROR\))170 2460 y(CHARACTER\(*\))g(FILENAME)170 2517 y(INTEGER)g(NCELLS,)g(REC)p 627 2517 V 17 w(SIZE,)g(IERROR)166 2647 y Ft(Returns)11 b(the)g(\014le)h (structure)f(parameters)f(\(n)o(um)o(b)q(er)g(of)h(columns)h(and)f(record)f (size\))i(of)e(an)h(existing)75 2704 y(V)l(esta)k(\014le.)1967 46 y Fr(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 23 24 bop 75 -100 a Fp(1.8.)34 b(VEST)l(A)16 b(FUNCTIONS)1206 b Ft(23)75 49 y Fm(MPI)p 160 49 14 2 v 16 w(PFS)p 256 49 V 16 w(OPEN\()21 b(\014lename,)e(status,)24 b(action,)d(t)o(yp)q(e,)h(mo)q(de,)d(concurrency)l (,)k(\014lepa)o(rams,)18 b(pa)o(rtpa)o(rams,)75 106 y(comm)m(,)12 b(new)o(comm)n(\))117 183 y Fk(IN)155 b Fm(\014lename)428 b Fk(name)13 b(of)g(op)q(ened)i(\014le)f(\(string\))117 259 y(IN)155 b Fm(status)476 b Fk(one)10 b(of)e Fj(MPI)p 1096 259 13 2 v 15 w(OLD,)g(MPI)p 1289 259 V 15 w(NEW,)h(MPI)p 1493 259 V 14 w(REPLA)o(CE,)e(MPI)p 1781 259 V 15 w(SCRA)m(TCH)905 315 y Fk(or)14 b Fj(MPI)p 1033 315 V 14 w(UNKNO)o(WN)g Fk(\(state\))117 390 y(IN)155 b Fm(action)472 b Fk(one)14 b(of)f Fj(MPI)p 1105 390 V 14 w(READ,)g(MPI)p 1329 390 V 14 w(WRITE)p Fk(,)f(or)i Fj(MPI)p 1624 390 V 14 w(READ)o(WRITE)905 447 y Fk(\(state\))117 522 y(IN)155 b Fm(t)o(yp)q(e)507 b Fk(one)10 b(of)f Fj(MPI)p 1097 522 V 14 w(STREAM)p Fk(,)g(or)g Fj(MPI)p 1420 522 V 14 w(FIXED)f Fk(\(\014xed)i(size)h(record\))905 579 y(\(state\))117 654 y(IN)155 b Fm(mo)q(de)481 b Fk(one)10 b(of)e Fj(MPI)p 1096 654 V 15 w(PRIV)m(A)m(TE,)e(MPI)p 1373 654 V 14 w(SHARED)j Fk(or)g Fj(MPI)p 1679 654 V 15 w(COLLECTIVE)905 711 y Fk(\(state\))117 786 y(IN)155 b Fm(metho)q(d)442 b Fk(one)14 b(of)g Fj(MPI)p 1106 786 V 14 w(UNI)f Fk(or)h Fj(MPI)p 1329 786 V 14 w(MUL)m(TI)g Fk(\(state\))117 862 y(IN)155 b Fm(concurrency)362 b Fk(one)10 b(of)e Fj(MPI)p 1096 862 V 15 w(EX)o(CLUSIVE)f Fk(or)i Fj(MPI)p 1457 862 V 14 w(CONCURRENT)f Fk(\(state\))117 937 y(IN)155 b Fm(\014lepa)o(rams)394 b Fk(arra)o(y)14 b(describing)g(\014le)g(structure)i (\(arra)o(y)e(of)f(in)o(teger\))117 1013 y(IN)155 b Fm(pa)o(rtpa)o(rams)373 b Fk(arra)o(y)14 b(describing)h(partitioning)d(parameters)i(\(arra)o(y)g(of)g (in-)905 1069 y(teger\))117 1144 y(IN)155 b Fm(comm)466 b Fk(comm)o(unicator) 11 b(of)j(group)f(that)h(op)q(ens)h(the)f(\014le)g(\(handle\))117 1220 y(OUT)108 b Fm(new)o(comm)393 b Fk(comm)o(unicator)13 b(asso)q(ciated)j(with)f(\014le)h(b)o(y)f(op)q(en)h(op)q(eration)905 1276 y(\(handle\))75 1401 y Fh(int)47 b(MPI)p 269 1401 15 2 v 17 w(Pfs)p 358 1401 V 17 w(open\(const)22 b(char)i(*filename,)e(MPI)p 1091 1401 V 17 w(IO)i(status,)e(MPI)p 1442 1401 V 17 w(IO)i(action,)393 1457 y(MPI)p 468 1457 V 17 w(IO)g(type,)f(MPI)p 772 1457 V 16 w(IO)h(mode,)f(MPI)p 1075 1457 V 17 w(IO)h(method,)f(MPI)p 1427 1457 V 16 w(IO)h(concurrency,)393 1514 y(int)g(*fileparams,)e(int)h (*partparams,)g(MPI)p 1280 1514 V 16 w(Comm)h(comm,)393 1570 y(MPI)p 468 1570 V 17 w(Comm)f(*newcomm\))75 1657 y(MPI)p 150 1657 V 17 w(PFS)p 239 1657 V 17 w(OPEN\(FILENAME,)e(STATUS,)i(ACTION,)g (TYPE,)g(MODE,)h(METHOD,)f(CONCURRENCY,)393 1713 y(FILEPARAMS,)g(PARTPARAMS,) f(COMM,)h(NEWCOMM,)g(IERROR\))170 1770 y(CHAR\(*\))g(FILENAME)170 1826 y(INTEGER)g(STATUS,)g(ACTION,)g(TYPE,)g(MODE,)h(METHOD,)e(CONCURRENCY,)h (FILEPARAMS\(*\),)170 1883 y(PARTPARAMS\(*\),)f(COMM,)i(NEWCOMM,)e(IERROR)166 1969 y Ft(Op)q(en)h(a)e(V)l(esta)g(sub\014le)i(for)e(I/O.)h(This)g(is)g(a)f (collectiv)o(e)i(call)g(executed)f(b)o(y)g(all)g(pro)q(cesses)g(in)75 2026 y(the)d(comm)o(unication)g(group.)29 b(It)19 b(returns)f(a)g(comm)o (unicator)g(that)g(can)h(b)q(e)g(subsequen)o(tly)h(used)f(for)75 2082 y(accessing)d(a)f(sub\014le)h(of)f(the)h(op)q(ened)g(V)l(esta)f(\014le.) 166 2139 y(The)i(parameters)f(are)h(the)g(same)f(as)h(for)f Fm(MPI)p 985 2139 14 2 v 16 w(OPEN)p Ft(,)h(except)g(that)f(the)h Fm(p)q(osition)h Ft(argumen)o(t)e(is)75 2195 y(not)g(a)o(v)m(ailable.)26 b(Tw)o(o)16 b(additional)i(argumen)o(ts,)e Fm(\014lepa)o(rams)e Ft(and)j Fm(pa)o(rtpa)o(ram)o(s)p Ft(,)d(are)j(arra)o(ys)e(describing)75 2252 y(the)g(structure)g(of)g(the)g(V)l(esta)g(\014le)i(and)e(of)g(a)g (sub\014le)h(within)h(it.)166 2308 y(The)f Fm(\014lepa)o(rams)d Ft(argumen)o(t)j(is)g(an)g(arra)o(y)f(with)h(t)o(w)o(o)e(en)o(tries,)j Fm(ncells)f Ft(and)h Fm(rec)p 1518 2308 V 16 w(size)f Ft(that)f(describ)q(e) 75 2365 y(the)j(n)o(um)o(b)q(er)f(of)g(columns)i(\(cells\))f(and)g(the)f (record)h(size)g(of)f(the)h(V)l(esta)f(\014le.)28 b(All)19 b(pro)q(cesses)e(pro)o(vide)75 2421 y(the)d(same)f(v)m(alues)i(for)f(this)g (argumen)o(t.)19 b(A)14 b(v)m(alue)h(of)e Fj(MPI)p 1066 2421 13 2 v 14 w(UNKNO)o(WN)h Ft(can)g(b)q(e)h(used,)f(in)h(whic)o(h)g(case,)e(a) 75 2478 y(system)h(pro)o(vided)g(default)h(is)g(used.)20 b(F)l(or)13 b(a)h(new)g(\014le,)h(the)f(default)h(v)m(alue)g(for)e Fm(ncells)i Ft(is)g(the)f(n)o(um)o(b)q(er)g(of)75 2534 y(storage)i(no)q(des;)h(the)g (default)h(v)m(alue)g(for)e Fm(rec)p 860 2534 14 2 v 16 w(size)h Ft(is)h(implemen)o(tation)g(dep)q(enden)o(t.)26 b(F)l(or)16 b(an)h(existing)75 2591 y(\014le,)g(the)g(default)g(v)m(alues)g(are)f(the)h (curren)o(t)f(\014le)h(structure)g(parameters.)22 b(One)17 b(cannot)f(c)o(hange)h(these)75 2647 y(t)o(w)o(o)12 b(v)m(alues)j(for)e(an)h (existing)g(\014le;)h(the)f(\014le)g(structure)g(parameters)e(should)j (either)f(matc)o(h)f(the)h(curren)o(t)75 2704 y(\014le)i(setting,)f(or)g(b)q (e)h(equal)g(to)e Fj(MPI)p 680 2704 13 2 v 14 w(UNKNO)o(WN)p Ft(.)-32 46 y Fr(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 24 25 bop 75 -100 a Ft(24)1387 b Fp(CHAPTER)16 b(1.)34 b(IO)166 49 y Ft(The)15 b Fm(pa)o(rtpa)o(rams)d Ft(argumen)o(t)i(is)h(an)g(arra)o(y)f (with)h(\014v)o(e)g(en)o(tries:)20 b Fm(Hn,)15 b(Vn,)h(Hbs,)f(Vbs)h Ft(and)f Fm(numpa)o(rt)p Ft(.)75 106 y(The)d(\014rst)f(four)h(de\014ne)h(a)e (partitioning)i(sc)o(heme;)g(the)e(\014fth)h(de\014nes)h(a)f(sub\014le)h(n)o (um)o(b)q(er;)g Fm(0)f Fg(\024)h Fm(numpa)o(rt)e Fd(<)75 162 y Fm(Hn)g Fg(\001)f Fm(Vn)15 b Ft(should)i(hold.)166 466 y(If)d(the)h(\014le) g(is)f(op)q(ened)i(in)f(mo)q(de)f Fj(MPI)p 809 466 13 2 v 14 w(PRIV)m(A)m(TE)e Ft(then)i(the)g(\014fth)h(en)o(try)e(of)h Fm(pa)o(rtpa)o(rams)d Ft(is)k(ignored:)75 523 y(eac)o(h)g(pro)q(cess)g(gets)f (to)g(access)h(a)g(disjoin)o(t)g(sub\014le,)h(where)f(the)g(pro)q(cess)g (with)g(rank)g Fm(i)g Ft(attac)o(hes)e(sub\014le)75 579 y Fm(i)p Ft(.)20 b(In)15 b(particular,)g(if)h(the)f(n)o(um)o(b)q(er)g(of)g(cells)h (equals)g(the)f(n)o(um)o(b)q(er)g(of)g(pro)q(cesses,)g(and)g(the)g (partitioning)75 636 y(parameters)d(sp)q(ecify)i(a)f(sub\014le)h(for)e(eac)o (h)h(cell)h(\()p Fm(Hn)f(=)g(ncells,)h(Vn)g(=)f(1)p Ft(\),)f(then)h(eac)o(h)g (pro)q(cess)g(has)g(access)75 692 y(to)j(a)g(priv)m(ate)h(\014le,)h(stored)e (on)g(a)g(distinct)i(cell.)25 b(In)17 b(general,)g(eac)o(h)g(pro)q(cess)f (has)h(access)f(to)g(a)g(disjoin)o(t)75 749 y(priv)m(ate)21 b(\014le,)i(carv)o(ed)d(out)g(of)g(the)h(shared)g(V)l(esta)f(\014le.)37 b(The)21 b(call)g(is)g(erroneous)g(if)g(the)g(n)o(um)o(b)q(er)f(of)75 805 y(sub\014les)d(is)e(less)h(than)f(the)g(group)g(size.)166 1109 y(If)i(the)g(\014le)h(is)f(op)q(ened)h(in)g(an)o(y)f(other)f(mo)q(de)h (then)h(all)f(pro)q(cesses)h(pro)o(vide)f(the)g(same)g(argumen)o(t)75 1166 y(v)m(alues,)f(and)g(all)g(pro)q(cesses)g(access)f(the)h(same)f (sub\014le.)22 b(The)15 b(\014le)h(p)q(oin)o(ter)g(is)g(p)q(ositioned)h(at)e (the)g(start)75 1222 y(of)g(the)g(sub\014le.)166 1526 y(A)k(V)l(esta)g (\014le)h(ma)o(y)f(b)q(e)g(op)q(ened)i(at)d(the)h(same)g(time)h(with)f(sev)o (eral)g(di\013eren)o(t)h(comm)o(unicators;)75 1583 y(sub\014les)k(accessed)f (with)g(di\013eren)o(t)g(comm)o(unicators)f(ma)o(y)f(o)o(v)o(erlap.)42 b(If)23 b(the)f(\014le)i(is)f(op)q(ened)h(with)75 1639 y Fm(concurrency)16 b Ft(=)g Fj(MPI)p 444 1639 V 14 w(CONCURRENT)e Ft(then)i(the)g(V)l(esta)f (\014le)h(system)f(will)i(use)f(a)f(concurrency)h(con)o(trol)75 1696 y(proto)q(col)d(that)e(ensures)j(that)d(concurren)o(t)i(accesses)g(to)f (the)h(\014le)g(are)f(atomic,)h(serializable)i(and)e(causal:)75 1752 y(the)22 b(outcome)f(is)h(as)g(if)g(MPI-IO)h(calls)f(w)o(ere)g(executed) g(in)h(some)e(serial)i(order,)g(compatible)g(with)75 1809 y(the)17 b(program)g(order.)26 b(If)18 b(the)f(\014le)i(is)f(op)q(ened)g(with)g Fm(concurrency)g Ft(=)g Fj(MPI)p 1371 1809 V 14 w(EX)o(CLUSIVE)e Ft(then)h(no)h(suc)o(h)75 1865 y(concurrency)j(proto)q(col)g(is)g(used.)37 b(This)21 b(mo)q(de)g(should)h(b)q(e)f(used)g(only)g(if)g(no)g(record)f(of)h (the)f(same)75 1922 y(\014le)c(is)g(concurren)o(tly)f(accessed)h(and)f(up)q (dated)h(using)g(t)o(w)o(o)e(di\013eren)o(t)h(comm)o(unicators)g(for)f (\014le)i(access.)75 1978 y(In)i(particular,)g(this)g(mo)q(de)g(can)f(b)q(e)h (used)g(if)g(sub\014les)h(concurren)o(tly)f(op)q(ened)h(with)f(distinct)g (calls)h(to)75 2034 y Fm(MPI)p 160 2034 14 2 v 16 w(PFS)p 256 2034 V 16 w(OPEN)d Ft(do)f(not)g(o)o(v)o(erlap.)166 2421 y Fl(Discussion:)39 b Fk(If)15 b(distinct)g(pro)q(cesses)j(w)o(an)o(t)d(to)g (ha)o(v)o(e)g(priv)n(ate,)g(unco)q(ordinated)g(accesses)j(to)d(unrelated)75 2478 y(sub\014les)h(out)g(of)f(the)h(same)f(V)m(esta)h(\014le,)f(then)h(eac)o (h)g(of)f(them)g(can)h(op)q(en)g(separately)g(the)g(\014le.)23 b(The)16 b(comm)o(uni-)75 2534 y(cator)g Fj(MPI)p 260 2534 13 2 v 14 w(COMM)p 404 2534 V 15 w(I)p Fk(,)f(whic)o(h)g(has)h(a)g(singleton) f(group)h(including)f(only)f(the)j(lo)q(cal)e(pro)q(cess,)i(can)f(b)q(e)h (used)f(for)75 2591 y(that)g(purp)q(ose.)27 b(The)17 b(collectiv)o(e)g(op)q (ening)f(of)g(the)h(same)e(\014le)i(b)o(y)f(m)o(ultiple)e(pro)q(cesses)19 b(either)e(allo)o(ws)e(them)h(to)75 2647 y(co)q(ordinate)e(allo)q(cation)e (of)g(disjoin)o(t)h(sub\014les)h(\()p Fj(MPI)p 901 2647 V 14 w(PRIV)m(A)m(TE)d Fk(mo)q(de\),)h(or)i(allo)o(ws)e(them)g(to)i(share)g(use)g (of)f(a)g(\014le)75 2704 y(p)q(oin)o(ter)h(\()p Fj(MPI)p 311 2704 V 15 w(SHARED)p Fk(\),)f(or)h(allo)o(ws)e(them)h(to)h(access)i(a)d (sub\014le)i(collectiv)o(ely)m(.)1967 46 y Fr(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 25 26 bop 75 -100 a Fp(1.8.)34 b(VEST)l(A)16 b(FUNCTIONS)1206 b Ft(25)75 49 y Fm(MPI)p 160 49 14 2 v 16 w(PFS)p 256 49 V 16 w(REOPEN\()17 b(action,)e(p)q(osition,)h(mo)q(de,)c(concurrency)l(,)k(pa)o(rtpa)o(rams,)c (comm)m(\))117 126 y Fk(IN)155 b Fm(action)472 b Fk(one)14 b(of)f Fj(MPI)p 1105 126 13 2 v 14 w(READ,)g(MPI)p 1329 126 V 14 w(WRITE)p Fk(,)f(or)i Fj(MPI)p 1624 126 V 14 w(READ)o(WRITE)905 183 y Fk(\(state\))117 257 y(IN)155 b Fm(p)q(osition)439 b Fk(one)18 b(of)f Fj(MPI)p 1113 257 V 14 w(ASIS)g Fk(\(curren)o(t)i(p)q (osition\))e(or)g Fj(MPI)p 1701 257 V 14 w(REWIND)905 313 y Fk(\(start)e(of)e(\014le\))h(\(state\))117 387 y(IN)155 b Fm(mo)q(de)481 b Fk(one)10 b(of)e Fj(MPI)p 1096 387 V 15 w(PRIV)m(A)m(TE,)e(MPI)p 1373 387 V 14 w(SHARED)j Fk(or)g Fj(MPI)p 1679 387 V 15 w(COLLECTIVE)905 443 y Fk(\(state\))117 517 y(IN)155 b Fm(metho)q(d)442 b Fk(one)14 b(of)g Fj(MPI)p 1106 517 V 14 w(UNI)f Fk(or)h Fj(MPI)p 1329 517 V 14 w(MUL)m(TI)g Fk(\(state\))117 591 y(IN)155 b Fm(concurrency)362 b Fk(one)10 b(of)e Fj(MPI)p 1096 591 V 15 w(EX)o(CLUSIVE)f Fk(or)i Fj(MPI)p 1457 591 V 14 w(CONCURRENT)f Fk(\(state\))117 664 y(IN)155 b Fm(pa)o(rtpa)o(rams)373 b Fk(arra)o(y)18 b(describing)g (partition)f(parameters)g(\(arra)o(y)h(of)f(in)o(te-)905 721 y(ger\))117 794 y(INOUT)62 b Fm(comm)466 b Fk(comm)o(unicator)11 b(of)j(group)f(that)h(op)q(ens)h(the)f(\014le)g(\(handle\))75 919 y Fh(int)47 b(MPI)p 269 919 15 2 v 17 w(Pfs)p 358 919 V 17 w(reopen\()23 b(MPI)p 638 919 V 17 w(IO)g(action,)g(MPI)p 989 919 V 17 w(IO)g(position,)g(MPI)p 1388 919 V 17 w(IO)g(mode,)393 975 y(MPI)p 468 975 V 17 w(IO)h(method,)e(MPI)p 819 975 V 17 w(IO)i(concurrency,)e(int)h(*partparams,)393 1032 y(MPI)p 468 1032 V 17 w(Comm)g(comm\))75 1118 y(MPI)p 150 1118 V 17 w(PFS)p 239 1118 V 17 w(REOPEN\()f(ACTION,)h(POSITION,)g(MODE,)g(METHOD,)g (CONCURRENCY,)f(PARTPARAMS,)393 1175 y(COMM,)h(IERROR\))170 1231 y(INTEGER)g(ACTION,)g(POSITION,)g(MODE,)g(METHOD,)g(CONCURRENCY,)f (PARTPARAMS\(*\),)170 1288 y(COMM,)i(IERROR)166 1374 y Ft(Reop)q(ens)15 b(an)f(already)g(op)q(ened)h(\014le,)g(p)q(ossibly)h(c)o(hanging)e(the)g (partition)h(parameters)e(and)h(access)75 1430 y(mo)q(de.)20 b(This)c(op)q(eration)f(is)h(equiv)m(alen)o(t)h(to)d(closing)j(and)e(op)q (ening)h(anew)g(a)e(\014le,)i(but)g(it)f(allo)o(ws)g(one)h(to)75 1487 y(preserv)o(e)d(the)h(use)f(of)g(the)g(same)g(comm)o(unicator)g(and)h(a) o(v)o(oids)f(sp)q(ecifying)i(anew)e(the)g(parameters)g(that)75 1543 y(do)f(not)h(c)o(hange)f(\(\014le)i(name,)e(status,)g(\014le)i (parameters\).)k(The)12 b Fm(p)q(osition)i Ft(parameter)e(is)h(signi\014can)o (t)g(only)75 1600 y(if)j(the)g(\014le)h(is)f(reop)q(ened)h(with)f(the)g(same) g(partition)g(parameters,)e(in)j(whic)o(h)g(case)e(the)h(\014le)h(p)q(oin)o (ter\(s\))75 1656 y(is)e(\(are\))e(rep)q(ositioned)j(as)e(indicated)i(b)o(y)e (this)h(parameter;)e(if)i(the)f(\014le)i(is)f(repartitioned)g(or)e(the)i (access)75 1713 y(mo)q(de)e(is)h(c)o(hanged)f(from)f Fj(MPI)p 594 1713 13 2 v 14 w(PRIV)m(A)m(TE)e Ft(to)j Fj(MPI)p 916 1713 V 14 w(SHARED)f Ft(or)h Fj(MPI)p 1233 1713 V 14 w(COLLECTIVE)p Ft(,)e(or)h(vice)i(v)o(ersa,)f(then)75 1769 y(the)20 b(\014le)h(p)q(oin)o (ter\(s\))f(is)g(\(are\))f(p)q(ositioned)i(at)f(the)g(b)q(eginning)i(of)d (the)h(sub\014le\(s\),)i(and)e(the)g Fm(p)q(osition)75 1826 y Ft(parameter)14 b(is)i(ignored.)166 1882 y(The)c Fm(MPI)p 341 1882 14 2 v 16 w(GET)p 445 1882 V 16 w(FILE)f Ft(function)i(can)e(b)q(e)i (used)f(to)f(\014nd)h(whether)g(a)f(comm)o(unicator)g(is)i(attac)o(hed)e(to) 75 1939 y(a)h(V)l(esta)h(\014le)g(and,)g(if)g(so,)g(to)f(\014nd)h(the)g (\014lename)h(and)e(the)h(v)m(alues)h(of)e(the)h Fm(action,)g(t)o(yp)q(e,)h (rec)p 1627 1939 V 16 w(size,)f(concur-)75 1995 y(rency)l(,)f(p)q(osition)g Ft(and)f Fm(mo)q(de)e Ft(argumen)o(ts.)17 b(In)12 b(addition,)g(the)f (function)h Fm(MPI)p 1366 1995 V 16 w(GET)p 1470 1995 V 17 w(PFS)p 1567 1995 V 16 w(P)l(ARAMETERS)75 2051 y Ft(can)j(b)q(e)h(used)g(to)f (query)g(on)g(the)g(curren)o(t)g(\014le)i(structure)e(and)g(partitioning.)75 2155 y Fm(MPI)p 160 2155 V 16 w(GET)p 264 2155 V 17 w(PFS)p 361 2155 V 16 w(P)l(ARAMETERS\(comm)o(,)d(\014lepa)o(rams,)g(pa)o(rtpa)o(ram) o(s\))117 2232 y Fk(IN)155 b Fm(comm)466 b Fk(comm)o(unicator)11 b(attac)o(hed)k(to)f(\014le)f(\(handle\))117 2306 y(OUT)108 b Fm(\014lepa)o(rams)394 b Fk(arra)o(y)14 b(describing)g(\014le)g(structure)i (\(arra)o(y)e(of)f(in)o(tegers\))117 2380 y(OUT)108 b Fm(pa)o(rtpa)o(rams)373 b Fk(arra)o(y)14 b(describing)h(partitioning)d(parameters)i(\(arra)o(y)g(of)g (in-)905 2436 y(tegers\))75 2561 y Fh(int)47 b(MPI)p 269 2561 15 2 v 17 w(Get)p 358 2561 V 17 w(pfs)p 447 2561 V 17 w(parameters\()22 b(MPI)p 822 2561 V 17 w(Comm)h(comm,)g(int)h(*fileparams,)393 2617 y(int)g(*partparams\))75 2704 y(MPI)p 150 2704 V 17 w(GET)p 239 2704 V 17 w(PFS)p 328 2704 V 16 w(PARAMETERS\()f(COMM,)g(FILEPARAMS,)f (PARTPARAMS,)h(IERROR\))-32 46 y Fr(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 26 27 bop 75 -100 a Ft(26)1387 b Fp(CHAPTER)16 b(1.)34 b(IO)170 49 y Fh(INTEGER)23 b(COMM,)h(FILEPARAMS\(*\),)d(PARTPARAMS\(*\),)h(IERROR)166 136 y Ft(Returns)15 b(the)f(\014le)i(structure)e(parameters)f(and)i(the)f (partitioning)i(parameters)d(of)h(the)h(V)l(esta)f(\014le)75 192 y(and)20 b(the)g(sub\014le)h(attac)o(hed)e(to)g(the)h(comm)o(unicator.)33 b(The)19 b(call)i(is)f(erroneous)g(if)g(there)g(is)g(no)g(suc)o(h)75 249 y(attac)o(hed)15 b(V)l(esta)g(\014le.)75 370 y Fo(1.8.3)49 b(Access)15 b(to)h(V)o(esta)g(\014les)75 456 y Ft(Once)g(a)f(V)l(esta)f (\014le)i(is)g(op)q(ened)g(\(or)e(reop)q(ened\),)h(then)h(pro)q(cesses)f(ma)o (y)f(access)h(the)g(asso)q(ciated)g(sub\014le)75 513 y(using)h(an)o(y)g(of)f (the)h(MPI-IO)h(commands)e(for)g(\014le)i(access)f(and)g(p)q(ositioning,)h (as)e(a)h(regular)f(\014le.)23 b(If)16 b(the)75 569 y(\014le)f(w)o(as)e(op)q (ened)h(as)g(a)f(record)h(\014le)h(\()p Fm(t)o(yp)q(e)f Ft(=)g Fj(MPI)p 925 569 13 2 v 14 w(FIXED)p Ft(\))e(then)i(the)g(sub\014le)h(is)g (treated)e(as)g(a)h(\014xed)g(size)75 626 y(record)i(\014le:)23 b(eac)o(h)16 b(V)l(esta)g(record)g(is)h(a)f(\014le)h(record,)f(and)g(the)g (records)g(are)g(ordered)g(in)h(the)g(canonical)75 682 y(order)g(describ)q (ed)i(ab)q(o)o(v)o(e.)26 b(If)18 b(the)f(\014le)h(w)o(as)f(op)q(ened)h(as)f (a)g(stream)g(\014le)h(\()p Fm(t)o(yp)q(e)g Ft(=)f Fj(MPI)p 1572 682 V 15 w(STREAM)p Ft(\))g(then)75 738 y(the)f(sub\014le)i(is)f (treated)f(as)g(a)g(sequence)h(of)f(b)o(ytes)g({)g(the)h(sequence)g(obtained) g(b)o(y)f(concatenating)h(the)75 795 y(comp)q(onen)o(t)e(records)g(in)h (canonical)g(order.)k(With)15 b(this)h(in)o(terpretation,)f(all)h(MPI-IO)f (\014le)i(access)e(and)75 851 y(p)q(ositioning)i(functions)f(w)o(ork)e(as)h (on)g(a)g(regular)g(\014le,)h(with)g(few)f(c)o(hanges.)166 908 y(The)g(main)h(discrepancy)g(is)g(that)e(a)h(V)l(esta)f(sub\014le)j(do)q (es)e(not)g(ha)o(v)o(e)g(an)g(end-of-\014le)h(mark.)j(Th)o(us,)75 964 y(one)f(cannot)f(execute)h(a)f(call)h(to)f Fm(MPI)p 745 964 14 2 v 16 w(SEEK)p Ft(,)g(with)h Fm(whence)h(=)e(MPI)p 1298 964 V 16 w(END)p Ft(,)g(nor)g(can)h(one)f(execute)h(a)75 1021 y(call)e(to)f Fm(MPI)p 299 1021 V 16 w(ENDFILE)p Ft(.)166 1160 y Fl(Discussion:)41 b Fk(Do)15 b(w)o(e)h(really)f(need)h(this)g (discrepancy?)25 b(W)m(ould)14 b(it)h(mak)o(e)f(sense)k(for)d(MPI-IO)h(to)f (k)o(eep)75 1216 y(its)f(o)o(wn)f(end-of-\014le)h(mark)o(er?)166 1355 y Ft(It)20 b(is)h(erroneous)f(to)f(read-access)i(a)e(V)l(esta)h(record)g (that)g(w)o(as)f(nev)o(er)h(previously)i(written.)34 b(In)75 1412 y(the)16 b(curren)o(t)f(implemen)o(tation,)i(suc)o(h)f(access)g(will)h (either)f(return)g(zero)g(b)o(ytes)f(or)g(an)h(all-zero)g(record.)75 1468 y(Ho)o(w)o(ev)o(er,)g(the)h(conditions)h(under)g(whic)o(h)g(either)f(o)q (ccur)h(are)e(somewhat)g(complex;)j(the)e(outcome)f(of)75 1525 y(suc)o(h)g(an)f(access)g(ma)o(y)f(b)q(e)i(a\013ected)f(b)o(y)g(accesses)h (to)e(other)h(disjoin)o(t)h(sub\014les.)1967 46 y Fr(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 27 28 bop 75 381 a Fv(Bibliograph)m(y)75 604 y Ft([1])22 b(W.)11 b(S.)h(Brainerd,)h(C.)f(H.)g(Goldb)q(ergs,)g(and)h(Jeanne)g(C.)e(Adams.)k Fs(Pr)n(o)n(gr)n(ammers)e(Guide)h(to)g(F)m(ortr)n(an)146 660 y(90)p Ft(.)20 b(McGra)o(w-Hill,)15 b(New)h(York,)e(1990.)75 754 y([2])22 b(P)o(eter)12 b(F.)g(Corb)q(ett,)h(Sandra)g(Johnson)g(Ba)o (ylor,)g(and)g(Dror)f(G.)g(F)l(eitelson.)17 b(Ov)o(erview)d(of)e(the)h(V)l (esta)146 811 y(parallel)h(\014le)h(system.)h(In)e Fs(Pr)n(o)n(c.)g(IPPS)f ('93)i(Workshop)g(on)f(I/O)h(in)f(Par)n(al)r(lel)f(Computer)i(Systems)p Ft(,)146 867 y(pages)g(1{16,)e(Apr)j(1993.)i(\(Reprin)o(ted)f(in)f Fs(Comput.)h(A)o(r)n(ch.)e(News)h Fn(21\(5\))p Ft(,)g(pp.)f(7{14,)f(Dec)h (1993\).)75 961 y([3])22 b(P)o(eter)d(F.)g(Corb)q(ett,)h(Dror)f(G.)g(F)l (eitelson,)i(Jean-Pierre)g(Prost,)f(and)g(Sandra)f(Johnson)i(Ba)o(ylor.)146 1017 y(P)o(arallel)c(access)g(to)e(\014les)j(in)f(the)f(V)l(esta)g(\014le)i (system.)k(In)17 b Fs(Sup)n(er)n(c)n(omputing)h('93)p Ft(,)e(pages)g (472{481,)146 1074 y(No)o(v)e(1993.)75 1168 y([4])22 b(Message)15 b(P)o(assing)g(In)o(terface)g(F)l(orum.)21 b(Do)q(cumen)o(t)15 b(for)g(a)g(standard)g(message-passing)h(in)o(terface.)146 1224 y(T)l(ec)o(hnical)h(Rep)q(ort)e(CS-93-214,)f(Univ)o(ersit)o(y)i(of)f(T)l (ennessee,)h(No)o(v)o(em)o(b)q(er)e(1993.)75 1318 y([5])22 b(C.)13 b(H.)g(Ko)q(elb)q(el,)j(D.)d(B.)g(Lo)o(v)o(eman,)g(R.)h(S.)f(Sc)o (hrieb)q(er,)i(G.)e(L.)h(Steele,)h(and)e(M.)g(E.)g(Zosel.)18 b Fs(The)c(High)146 1374 y(Performanc)n(e)i(F)m(ortr)n(an)f(Handb)n(o)n(ok)p Ft(.)k(The)d(MIT)f(Press,)g(1994.)75 1468 y([6])22 b(P)o(arasoft)9 b(Corp)q(oration.)14 b Fs(Expr)n(ess)e(V)m(ersion)f(1.0:)20 b(A)12 b(Communic)n(ation)h(Envir)n(onment)f(for)h(Par)n(al)r(lel)146 1525 y(Computers)p Ft(,)i(1988.)75 1618 y([7])22 b(Jean-Pierre)e(Prost.)29 b Fs(Par)n(al)r(lel)18 b(File)h(System)g(External)f(User)h(Interfac)n(e)g(Pr) n(op)n(osal)g(V1.2)p Ft(.)30 b(IBM)146 1675 y(TJWR)o(C,)14 b(Decem)o(b)q(er)i(1993.)-32 46 y Fr(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-comm@CS.UTK.EDU Wed Jan 26 14:04:14 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id OAA27461; Wed, 26 Jan 1994 14:04:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id OAA05147; Wed, 26 Jan 1994 14:00:35 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 26 Jan 1994 14:00:34 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with ESMTP (8.6.4/2.8s-UTK) id OAA05128; Wed, 26 Jan 1994 14:00:30 -0500 Received: from mcs.anl.gov (donner.mcs.anl.gov [140.221.5.134]) by antares.mcs.anl.gov (8.6.4/8.6.4) with ESMTP id MAA22970; Wed, 26 Jan 1994 12:59:57 -0600 Message-Id: <199401261859.MAA22970@antares.mcs.anl.gov> To: mpi-comm@CS.UTK.EDU Subject: MPI_PACK, MPI_UNPACK, MPI_PACK_SIZE, MPI_UNPACK_SIZE Cc: snir@watson.ibm.com, walker@sun4.EPM.ORNL.GOV, gropp@antares.mcs.anl.gov, lederman@super.org, tony@Aurora.CS.MsState.Edu, doss@ERC.MsState.Edu, jim@meiko.co.uk, carl@vlsi.cs.caltech.edu Date: Wed, 26 Jan 1994 12:59:55 -0600 From: Rusty Lusk Here is a concrete proposal for explicit and incremental packing and unpacking of buffers, that some of us talked about at the European meeting. It was generated in response to a strong request from the users. If it looks reasonable to you, I will try to flesh it out more for the draft to be presented at the Knoxville meeting. It probably would be added to the pt2pt chapter. Comments explicitly solicited. Rusty & Bill Proposal: Explicit and incremental packing and unpacking of MPI buffers At the recent European Workshop, several attendees expressed the need for incremental packing and unpacking of MPI buffers, a feature currently available in PVM. The idea is to provide for transferring data to/from memory described by an MPI datatype from/to a contiguous area that is to be sent/has been received with type MPI_BYTE. In our original discussions, we recommended essentially that users always allow MPI to do packing and unpacking automatically in in the course of moving data to and from user space. This allows for optimization by aggressive implementations and special hardware. Nonetheless, there are reasons for adding explicit packing and unpacking routines to MPI: 1. MPI currently does not allow *incremental* packing and unpacking. The incremental pack is for simplicity (to avoid creating several complex MPI datatypes, when efficiency is not important) or for situations where the contiguous buffer will be reused. The incremental unpack definitely adds functionality. For example when receiving an integer n together with n floats, one might want to know n before allocating precisely the right amount of space for the floats. 2. The users want it, and it will make translation of existing PVM programs particularly straightforward. 3. As pointed out by Nathan Doss et al., there are times when one wants to be able to reuse the packed version of the buffer. This will make implementation of the collective operations, for example, easier to do on top of pt2pt. 4. The essential functionality is likely to be present in an implementation anyway, and might as well be made visible to the user. The parameters of MPI_PACK and MPI_UNPACK arise out of the need to identify the memory area described by the datatype (addr,datatype,count), the contiguous memory to packed into or unpacked from (buffer, maxsize), the other process involved (communicator, rank) in order to be able to do translations if necessary, and an inout variable (size_so_far) needed to hold state across calls, to support incrementality. On the unpack we also need to be told how many items were there (actualcount). ________________________________________________________________________ MPI_PACK(addr,datatype,count,dest,comm,buffer,maxsize,size_so_far) The inout variable size_so_far represents a displacement into the buffer that has been used so far. MPI_PACK takes care of supplying pad bytes where needed. Other parameters are as described above. ________________________________________________________________________ Example of use (in C): Puts in { int size_so_far; int numfloats_a; int numfloats_b; float a[100]; float b[100]; char buffer[100000]; /* better solution below, using MPI_PACK_SIZE */ dest = 2; numfloats_a = 25; numfloats_b = 17; size_so_far = 0; MPI_PACK(numfloats_a, MPI_INT, 1, dest, MPI_COMM_WORLD, buffer, 100000, &size_so_far); MPI_PACK(a, MPI_FLOAT, numfloats_a, dest, MPI_COMM_WORLD, buffer, 100000, &size_so_far); MPI_PACK(numfloats_b, MPI_INT, 1, dest, MPI_COMM_WORLD, buffer, 100000, &size_so_far); MPI_PACK(b, MPI_FLOAT, numfloats_b, dest, MPI_COMM_WORLD, buffer, 100000, &size_so_far); MPI_SEND(buffer, size_so_far, MPI_BYTE, dest, tag, MPI_COMM_WORLD); } ________________________________________________________________________ MPI_UNPACK(addr,datatype,count,source,comm,buffer,maxsize,size_so_far,actcount) moves data from buffer into the area designated by (addr,datatype,count). Actcount has the number of items actually moved. ________________________________________________________________________ Example: (unpack above message) Instead of using a "real big" scratch area as above, one should be able to find out how much space a message described as MPI_BYTE would take up if it were to unpacked, or how much space a message described by an MPI buffer would take up if it were to be packed. These routines are very similar to the above, but don't move data. Therefore they don't need the two memory location parameters. The size_so_far parameter is inout as above, so they can be called incrementally. ________________________________________________________________________ MPI_PACK_SIZE(datatype,count,dest,comm,size_so_far) MPI_UNPACK_SIZE(datatype,count,source,comm,size_so_far) ________________________________________________________________________ Example: pack and unpack into dynamically allocated memory. This still needs a little more fleshing out, but I hope the idea is clear. From owner-mpi-comm@CS.UTK.EDU Wed Jan 26 14:23:02 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id OAA27655; Wed, 26 Jan 1994 14:23:01 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id OAA07368; Wed, 26 Jan 1994 14:22:21 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 26 Jan 1994 14:22:19 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with ESMTP (8.6.4/2.8s-UTK) id OAA07355; Wed, 26 Jan 1994 14:22:16 -0500 Received: from mcs.anl.gov (donner.mcs.anl.gov [140.221.5.134]) by antares.mcs.anl.gov (8.6.4/8.6.4) with ESMTP id NAA23614; Wed, 26 Jan 1994 13:22:17 -0600 Message-Id: <199401261922.NAA23614@antares.mcs.anl.gov> To: mpi-comm@CS.UTK.EDU Subject: MPI_PACK, MPI_UNPACK, MPI_PACK_SIZE, MPI_UNPACK_SIZE Cc: snir@watson.ibm.com, walker@sun4.EPM.ORNL.GOV, gropp@antares.mcs.anl.gov, lederman@super.org, tony@Aurora.CS.MsState.Edu, doss@ERC.MsState.Edu, jim@meiko.co.uk, carl@vlsi.cs.caltech.edu Date: Wed, 26 Jan 1994 13:22:16 -0600 From: Rusty Lusk (Apologies if this is your second copy; a glitch may have attacked the previous posting) Here is a concrete proposal for explicit and incremental packing and unpacking of buffers, that some of us talked about at the European meeting. It was generated in response to a strong request from the users. If it looks reasonable to you, I will try to flesh it out more for the draft to be presented at the Knoxville meeting. It probably would be added to the pt2pt chapter. Comments explicitly solicited. Rusty & Bill Proposal: Explicit and incremental packing and unpacking of MPI buffers At the recent European Workshop, several attendees expressed the need for incremental packing and unpacking of MPI buffers, a feature currently available in PVM. The idea is to provide for transferring data to/from memory described by an MPI datatype from/to a contiguous area that is to be sent/has been received with type MPI_BYTE. In our original discussions, we recommended essentially that users always allow MPI to do packing and unpacking automatically in in the course of moving data to and from user space. This allows for optimization by aggressive implementations and special hardware. Nonetheless, there are reasons for adding explicit packing and unpacking routines to MPI: 1. MPI currently does not allow *incremental* packing and unpacking. The incremental pack is for simplicity (to avoid creating several complex MPI datatypes, when efficiency is not important) or for situations where the contiguous buffer will be reused. The incremental unpack definitely adds functionality. For example when receiving an integer n together with n floats, one might want to know n before allocating precisely the right amount of space for the floats. 2. The users want it, and it will make translation of existing PVM programs particularly straightforward. 3. As pointed out by Nathan Doss et al., there are times when one wants to be able to reuse the packed version of the buffer. This will make implementation of the collective operations, for example, easier to do on top of pt2pt. 4. The essential functionality is likely to be present in an implementation anyway, and might as well be made visible to the user. The parameters of MPI_PACK and MPI_UNPACK arise out of the need to identify the memory area described by the datatype (addr,datatype,count), the contiguous memory to packed into or unpacked from (buffer, maxsize), the other process involved (communicator, rank) in order to be able to do translations if necessary, and an inout variable (size_so_far) needed to hold state across calls, to support incrementality. On the unpack we also need to be told how many items were there (actualcount). ________________________________________________________________________ MPI_PACK(addr,datatype,count,dest,comm,buffer,maxsize,size_so_far) The inout variable size_so_far represents a displacement into the buffer that has been used so far. MPI_PACK takes care of supplying pad bytes where needed. Other parameters are as described above. ________________________________________________________________________ Example of use (in C): Puts in { int size_so_far; int numfloats_a; int numfloats_b; float a[100]; float b[100]; char buffer[100000]; /* better solution below, using MPI_PACK_SIZE */ dest = 2; numfloats_a = 25; numfloats_b = 17; size_so_far = 0; MPI_PACK(numfloats_a, MPI_INT, 1, dest, MPI_COMM_WORLD, buffer, 100000, &size_so_far); MPI_PACK(a, MPI_FLOAT, numfloats_a, dest, MPI_COMM_WORLD, buffer, 100000, &size_so_far); MPI_PACK(numfloats_b, MPI_INT, 1, dest, MPI_COMM_WORLD, buffer, 100000, &size_so_far); MPI_PACK(b, MPI_FLOAT, numfloats_b, dest, MPI_COMM_WORLD, buffer, 100000, &size_so_far); MPI_SEND(buffer, size_so_far, MPI_BYTE, dest, tag, MPI_COMM_WORLD); } ________________________________________________________________________ MPI_UNPACK(addr,datatype,count,source,comm,buffer,maxsize,size_so_far,actcount) moves data from buffer into the area designated by (addr,datatype,count). Actcount has the number of items actually moved. ________________________________________________________________________ Example: (unpack above message) Instead of using a "real big" scratch area as above, one should be able to find out how much space a message described as MPI_BYTE would take up if it were to unpacked, or how much space a message described by an MPI buffer would take up if it were to be packed. These routines are very similar to the above, but don't move data. Therefore they don't need the two memory location parameters. The size_so_far parameter is inout as above, so they can be called incrementally. ________________________________________________________________________ MPI_PACK_SIZE(datatype,count,dest,comm,size_so_far) MPI_UNPACK_SIZE(datatype,count,source,comm,size_so_far) ________________________________________________________________________ Example: pack and unpack into dynamically allocated memory. This still needs a little more fleshing out, but I hope the idea is clear. From owner-mpi-comm@CS.UTK.EDU Wed Jan 26 14:49:19 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id OAA27797; Wed, 26 Jan 1994 14:49:18 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id OAA09642; Wed, 26 Jan 1994 14:48:40 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 26 Jan 1994 14:48:38 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with ESMTP (8.6.4/2.8s-UTK) id OAA09626; Wed, 26 Jan 1994 14:48:36 -0500 Received: from mcs.anl.gov (donner.mcs.anl.gov [140.221.5.134]) by antares.mcs.anl.gov (8.6.4/8.6.4) with ESMTP id NAA24405 for ; Wed, 26 Jan 1994 13:48:32 -0600 Message-Id: <199401261948.NAA24405@antares.mcs.anl.gov> To: mpi-comm@CS.UTK.EDU Subject: Some notes on the January MPI workshop in Europe Date: Wed, 26 Jan 1994 13:48:31 -0600 From: Rusty Lusk This is not a complete set of minutes of the January meeting, but it describes immediate responses to the presentations of those in the European parallel computing community who came to the workshop and told us of their impressions of MPI and how it would or would not meet their needs. (Rolf, can you get this into the hands of as many attendees as is convenient?) Rusty Lusk ___________________________________________________________________________ Notes on how we decided to respond to the user "requests" at the January meeting. This note represents the thoughts of the 12 committee memeber present at the European meeting. They will be presented to the whole committee for a formal vote at the meeting in Knoxville. Buffering --------- The users were clearly terrified by the discussion of "safety" into thinking that we were mandating that there be no buffering. We decided to change the wording in the draft to make it clear that MPI implementations were intended to support buffering. We decided to consider whether user-specifies-buffer should be mandatory. Error Codes ----- ----- They noticed, correctly, that without at least some list of error codes, it would be difficult to write portable programs that checked for errors. Rusty Lusk volunteered to collect some error codes and propose them at the Knoxville meeting. Paul Pierce suggested that there be a way of combining detailed error codes into "error classes". Perhaps MPI could at least specify the error classes. We will discuss this at the next meeting. Timers ------ The users asked for some kind of portable timer routine. We decided to find out what the POSIX clock routines were and then mandate that some of them be part of any MPI implementation. Bob Knighten will send us the specs for the POSIX clock routines. Tags ---- The users asked for some guaranteed tag range, for portability. We decided to specify that the tag lower bound is at most 0 and there is a contiguous range available, up to X, where X is greater than or equal to 256. (We may have changed this to 128 later.) An implementation note will be added to the draft to state that 256 (128) was chosen since it could allow for a lightweight message header but implementors on most MPP systems should consider this a minimum that should be exceeded. There will be an environmental enquiry function to give the actual value of X and the lower bound.. Validation Suite ---------- ----- ANL and ORNL will collaborate on a publicly available set of MPI programs, but we as yet have no plans for a rigorous validation suite. MPI Forum Followup --- ----- -------- David Walker will organize mailing lists managed by current chapter authors to allow comments to continue to be received forever. A committee will continue to exist for the forseeable future to provide interpretation of the standard when questions arise. Partial Receive ------- ------- The users asked for a way to unpack a message incrementally, as in PVM. We decided to keep thinking about this. (It was later in the meeting when some of us came up with a plan for a MPI_PACK and MPI_UNPACK that would meet this requirement; it will be proposed on the net soon and discussed at the Knoxville meeting.) Parameterized Datatypes ------------- --------- We decided to defer this topic to the Fortran-90 binding discussion, whenever that occurs. Remote Copy ------ ---- We decided to stick to our original position that remote copy was outside the scope of the MPI goal. It is anticipated that this would be a topic to be included in MPI-2 discussions. From owner-mpi-comm@CS.UTK.EDU Thu Jan 27 03:34:31 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id DAA01634; Thu, 27 Jan 1994 03:34:31 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id DAA03314; Thu, 27 Jan 1994 03:31:56 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 27 Jan 1994 03:31:54 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from gmdzi.gmd.de by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id DAA03307; Thu, 27 Jan 1994 03:31:48 -0500 Received: from f1neuman.gmd.de (f1neuman) by gmdzi.gmd.de with SMTP id AA07098 (5.65c8/IDA-1.4.4 for ); Thu, 27 Jan 1994 09:29:51 +0100 Received: by f1neuman.gmd.de (AIX 3.2/UCB 5.64/4.03) id AA16328; Thu, 27 Jan 1994 09:31:47 GMT Date: Thu, 27 Jan 1994 09:31:47 GMT From: Rolf.Hempel@gmd.de (Rolf Hempel) Message-Id: <9401270931.AA16328@f1neuman.gmd.de> To: mpi-comm@CS.UTK.EDU Subject: Packing/unpacking routines Cc: gmap10@f1neuman.gmd.de I like the proposal by Rusty and Bill for packing and unpackine routines in MPI, and I strongly support the inclusion of those or similar functions in the standard. Some proponents of this functionality wanted to have those functions in situations where they send a single buffer to many different locations. Thus, the buffer has to be packed one time only. However, with the proposed functions the buffer has to be packed for each destination process separately, because the conversion happens in the packing routine (arguments dest,comm). The application programmer cannot reuse the once packed buffer, because he/she does not know whether the target processors have the same data representation or not. Personally I have no problem with this, but we should at least mention that the functions cannot be used for reusing the packed buffer for many sends. The main reason for including the functions in the standard will be to ease code migration from systems like PARMACS or PVM to MPI, and I think that's a good enough reason. - Rolf From owner-mpi-comm@CS.UTK.EDU Thu Jan 27 11:09:32 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id LAA06978; Thu, 27 Jan 1994 11:09:32 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id LAA08634; Thu, 27 Jan 1994 11:08:40 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 27 Jan 1994 11:08:38 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with ESMTP (8.6.4/2.8s-UTK) id LAA08627; Thu, 27 Jan 1994 11:08:36 -0500 Received: from mcs.anl.gov (donner.mcs.anl.gov [140.221.5.134]) by antares.mcs.anl.gov (8.6.4/8.6.4) with ESMTP id KAA19553; Thu, 27 Jan 1994 10:07:21 -0600 Message-Id: <199401271607.KAA19553@antares.mcs.anl.gov> To: Rolf.Hempel@gmd.de (Rolf Hempel) Cc: mpi-comm@CS.UTK.EDU Subject: Re: Packing/unpacking routines In-reply-to: Your message of "Thu, 27 Jan 1994 09:31:47 GMT." <9401270931.AA16328@f1neuman.gmd.de> Date: Thu, 27 Jan 1994 10:07:12 -0600 From: Rusty Lusk | Some proponents of this functionality wanted to have those functions in | situations where they send a single buffer to many different locations. | Thus, the buffer has to be packed one time only. However, with the | proposed functions the buffer has to be packed for each destination | process separately, because the conversion happens in the packing | routine (arguments dest,comm). The application programmer cannot reuse | the once packed buffer, because he/she does not know whether the | target processors have the same data representation or not. | | Personally I have no problem with this, but we should at least mention | that the functions cannot be used for reusing the packed buffer for | many sends. | The idea of making packing process-specific is to be able to avoid translations in the cases were you know that none will be needed because MPI knows something about the other process, i.e., it knows precisely what has to be done (byte-swapping instead of full xdr conversion or whatever). Maybe we can solve this problem. Suppose that if you used MPI_ANY_PROC as the destination or source, MPI_PACK and MPI_UNPACK would then convert to a canonical form (like XDR) that could be used at any other process. That is, if you wanted to re-use the packed buffer, you would say so and potentially pay the price of a more general packing and unpacking procedure, but everything would be safe, and maximally efficient when you specified the process. Rusty & Bill From owner-mpi-comm@CS.UTK.EDU Thu Jan 27 11:35:05 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id LAA07204; Thu, 27 Jan 1994 11:35:04 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id LAA11798; Thu, 27 Jan 1994 11:33:45 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 27 Jan 1994 11:33:43 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id LAA11788; Thu, 27 Jan 1994 11:33:39 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA27713; Thu, 27 Jan 94 10:33:39 CST Date: Thu, 27 Jan 94 10:33:39 CST From: Tony Skjellum Message-Id: <9401271633.AA27713@Aurora.CS.MsState.Edu> To: lusk@mcs.anl.gov Subject: Re: Packing/unpacking routines Cc: mpi-comm@CS.UTK.EDU Hi, As long as the scope for resending is within one communicator, I do not see the problem, because unpack would choose the canonical form for that communicator (implementation defined, opaque). The canonical form of MPI_COMM_WORLD should be equivalent to what Bill/Rusty suggest for MPI_PROC_ANY, for consistency. For intercommunication, it would be nice to have a mechanism to extract the canonical form of other communicators, rather than just MPI_PROC_ANY, but this is perhaps too much to ask. Note that we have never promised to do optimizations of data conversion within a group boundary... but that we expect that multicanonical implementations are going to exist. In short: Rusty/Bill's suggestion seems fine in this regard. -Tony ----- Begin Included Message ----- From owner-mpi-comm@CS.UTK.EDU Thu Jan 27 10:18:37 1994 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 27 Jan 1994 11:08:38 EST To: Rolf.Hempel@gmd.de (Rolf Hempel) Cc: mpi-comm@CS.UTK.EDU Subject: Re: Packing/unpacking routines Date: Thu, 27 Jan 1994 10:07:12 -0600 From: Rusty Lusk Content-Length: 1481 | Some proponents of this functionality wanted to have those functions in | situations where they send a single buffer to many different locations. | Thus, the buffer has to be packed one time only. However, with the | proposed functions the buffer has to be packed for each destination | process separately, because the conversion happens in the packing | routine (arguments dest,comm). The application programmer cannot reuse | the once packed buffer, because he/she does not know whether the | target processors have the same data representation or not. | | Personally I have no problem with this, but we should at least mention | that the functions cannot be used for reusing the packed buffer for | many sends. | The idea of making packing process-specific is to be able to avoid translations in the cases were you know that none will be needed because MPI knows something about the other process, i.e., it knows precisely what has to be done (byte-swapping instead of full xdr conversion or whatever). Maybe we can solve this problem. Suppose that if you used MPI_ANY_PROC as the destination or source, MPI_PACK and MPI_UNPACK would then convert to a canonical form (like XDR) that could be used at any other process. That is, if you wanted to re-use the packed buffer, you would say so and potentially pay the price of a more general packing and unpacking procedure, but everything would be safe, and maximally efficient when you specified the process. Rusty & Bill ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Thu Jan 27 13:04:59 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id NAA07978; Thu, 27 Jan 1994 13:04:58 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id NAA19191; Thu, 27 Jan 1994 13:03:58 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 27 Jan 1994 13:03:57 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from SSD.intel.com by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id NAA19182; Thu, 27 Jan 1994 13:03:54 -0500 From: Received: from nautilus.ssd.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA00547; Thu, 27 Jan 94 10:02:28 PST Message-Id: <9401271802.AA00547@SSD.intel.com> To: Rusty Lusk Cc: Rolf.Hempel@gmd.de (Rolf Hempel), Tony Skjellum , mpi-comm@CS.UTK.EDU, prp@ssd.intel.com Subject: Re: Packing/unpacking routines In-Reply-To: Your message of "Thu, 27 Jan 94 10:07:12 CST." <199401271607.KAA19553@antares.mcs.anl.gov> Date: Thu, 27 Jan 94 10:02:22 -0800 Both Rolf and Tony make good points. I think Tony's comment that the communicator can make it all work is perhaps a little too restrictive of the implementation space. I propose we consider the following two restrictions: 1. The first call to pack or unpack for a new buffer area must have size_so_far equal to zero. That way, the implementation can be guaranteed a place to put a single generic representation descriptor if it wants to. 2. The calls to unpack for a particular buffer must be made with the same datatypes and counts as the calls to pack that built the buffer. For instance, if you pack an int and a bunch of floats this would be correct: pack the int pack the floats unpack the int unpack the floats while this would be erroneous: pack the int and the floats unpack the int unpack the floats. This way the implementation could put a representation descriptor in for each pack call but wouldn't have to put in a marker for every element. I like the first restriction because it doesn't constrict the application programmer in any meaningful way. I don't know that the second restriction is so good, but it does place a pretty powerful constraint on the use of pack and unpack that might keep some implementations out of trouble. As a separate issue, I would avoid the use of the size_so_far by splitting it into an start and finish. Doesn't change the code much, just adds one extra parameter to the list. A reasonable restriction on the implementation would be that the programmer be allowed to put the same variable in both positions. Not an issue in the C binding, but must be mentioned for the FORTRAN binding. MPI_PACK( ... , 0, &size_so_far); MPI_PACK( ... , size_so_far, &size_so_far); MPI_PACK( ... , size_so_far, &size_so_far); Paul From owner-mpi-comm@CS.UTK.EDU Thu Jan 27 18:01:51 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id SAA09787; Thu, 27 Jan 1994 18:01:50 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id SAA11840; Thu, 27 Jan 1994 18:00:37 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 27 Jan 1994 18:00:35 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with ESMTP (8.6.4/2.8s-UTK) id SAA11794; Thu, 27 Jan 1994 18:00:32 -0500 Received: from mcs.anl.gov (donner.mcs.anl.gov [140.221.5.134]) by antares.mcs.anl.gov (8.6.4/8.6.4) with ESMTP id RAA28862; Thu, 27 Jan 1994 17:00:32 -0600 Message-Id: <199401272300.RAA28862@antares.mcs.anl.gov> To: prp@ssd.intel.com Cc: mpi-comm@CS.UTK.EDU Subject: Re: Packing/unpacking routines In-reply-to: Your message of "Thu, 27 Jan 1994 10:02:22 PST." <9401271802.AA00547@SSD.intel.com> Date: Thu, 27 Jan 1994 17:00:29 -0600 From: Rusty Lusk | 1. The first call to pack or unpack for a new buffer area must have | size_so_far equal to zero. That way, the implementation can be guaranteed a | place to put a single generic representation descriptor if it wants to. | Agreed. | 2. The calls to unpack for a particular buffer must be made with the same | datatypes and counts as the calls to pack that built the buffer. I am not too keen on this one. It seems like a restriction on the programmer, who might find that he wants normal packing but incremental unpacking, and the benefits are not clear to me. | As a separate issue, I would avoid the use of the size_so_far by | splitting it into an start and finish. Doesn't change the code | much, just adds one extra parameter to the list. A reasonable restriction on | the implementation would be that the programmer be allowed to put the same | variable in both positions. Not an issue in the C binding, but must be | mentioned for the FORTRAN binding. | This seems to me to be going too far in order to avoid inouts. Here, the inout variable doesn't come from trying to overload some parameter but because the caller is holding state for the callee, which makes an inout parameter the natural thing. I don't care very much, however. - Rusty From owner-mpi-comm@CS.UTK.EDU Fri Jan 28 02:22:49 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id CAA11479; Fri, 28 Jan 1994 02:22:48 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id CAA11884; Fri, 28 Jan 1994 02:22:01 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 28 Jan 1994 02:21:59 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from gmdzi.gmd.de by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id CAA11877; Fri, 28 Jan 1994 02:21:55 -0500 Received: from f1neuman.gmd.de (f1neuman) by gmdzi.gmd.de with SMTP id AA04465 (5.65c8/IDA-1.4.4 for ); Fri, 28 Jan 1994 08:20:05 +0100 Received: by f1neuman.gmd.de (AIX 3.2/UCB 5.64/4.03) id AA17057; Fri, 28 Jan 1994 08:22:01 GMT Date: Fri, 28 Jan 1994 08:22:01 GMT From: Rolf.Hempel@gmd.de (Rolf Hempel) Message-Id: <9401280822.AA17057@f1neuman.gmd.de> To: mpi-comm@CS.UTK.EDU Subject: Re: Packing/unpacking routines Cc: gmap10@f1neuman.gmd.de I think the compromise of using the target address MPI_ANY_PROC in cases where you want to reuse the send buffer is okay. Since the communicator is still specified, the implementation can choose the canonical representation within that communicator, which means no data conversion if all processors have the same data representation. This way the XDR translation only occurs in heterogeneous systems where the overhead somehow has to be paid anyway. It would certainly not be acceptable if XDR had to be used in every call to a pack/unpack routine. Paul made a valid point with his suggestion to use two arguments for size_so_far instead of an inout argument, since we tried to avoid inouts for scalar arguments throughout MPI. Nevertheless, I sympathize with the inout in this case, because this is just a counter which the user will not be likely to use for other purposes in between calls. Making it an inout shortens the argument lists. On the other hand, making it two arguments in almost all cases will lead to the ugly Fortran style suggested by Paul (the same actual argument bound to two function arguments). I have no strong opinion about Paul's proposed restriction that the buffer must be unpacked exactly the same way as it was packed. If the ease of porting codes from existing interfaces to MPI is our main reason for including the buffering routines, then this restriction would not be too bad, because that's exactly how you would use them, anyway. However, together with Rusty I don't see the benefits resulting from that restriction. Could you please explain it to us, Paul? - Rolf From owner-mpi-comm@CS.UTK.EDU Fri Jan 28 13:02:45 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id NAA14740; Fri, 28 Jan 1994 13:02:44 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id NAA29927; Fri, 28 Jan 1994 13:01:08 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 28 Jan 1994 13:01:02 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from SSD.intel.com by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id NAA29916; Fri, 28 Jan 1994 13:00:59 -0500 From: Received: from nautilus.ssd.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA06928; Fri, 28 Jan 94 09:59:58 PST Message-Id: <9401281759.AA06928@SSD.intel.com> To: Rolf.Hempel@gmd.de (Rolf Hempel) Cc: mpi-comm@CS.UTK.EDU, gmap10@f1neuman.gmd.de, prp@ssd.intel.com Subject: Re: Packing/unpacking routines In-Reply-To: Your message of "Fri, 28 Jan 94 08:22:01 GMT." <9401280822.AA17057@f1neuman.gmd.de> Date: Fri, 28 Jan 94 09:59:21 -0800 > Date: Fri, 28 Jan 1994 08:22:01 GMT > From: Rolf.Hempel@gmd.de (Rolf Hempel) > To: mpi-comm@CS.UTK.EDU > Subject: Re: Packing/unpacking routines > I have no strong opinion about Paul's proposed restriction that the > buffer must be unpacked exactly the same way as it was packed. > If the ease of porting codes from existing interfaces to MPI is our > main reason for including the buffering routines, then this restriction > would not be too bad, because that's exactly how you would use them, > anyway. However, together with Rusty I don't see the benefits resulting > from that restriction. Could you please explain it to us, Paul? No, I can't :-). I don't see any clear benefits either, unless the first restriction (that packing start at zero) is rejected. If we are going to restrict use of pack and unpack for the benefit of the implementation, it seems like an obvious restriction to propose. It might make implementation of things like strict type checking easier. But I can't think of anything I would want to do in an implementation that couldn't be done without the restriction. So to further explore the specification space let me propose an alternative anti-restriction, which requires the implementation to be flexible: -2. An application is allowed to pack and unpack a buffer in any manner as long as you start at zero and the basic datatypes match. For example, if you pack a buffer like this: pack 2 int pack 7 real pack 3 real you can unpack it like this if you want: count type size_so_far in size_so_far out unpack 1 int 0 a unpack 1 int a b unpack 5 real b d unpack 5 real d e unpack 1 (1 int, 1 real) a c unpack 9 real c e unpack 1 (1 int, 5 real) a d Note in this example that parts of the buffer are unpacked more than once without starting back at zero, and that different ways to unpack ending at the same place return the same output size_so_far. Any of the unpack calls could be replaced by unpack_size calls to just get the size_so_far output values. I can't think of any problems implementing to this spec. > > - Rolf Paul From owner-mpi-comm@CS.UTK.EDU Mon Jan 31 02:15:12 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id CAA02251; Mon, 31 Jan 1994 02:15:11 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id CAA09809; Mon, 31 Jan 1994 02:14:14 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 31 Jan 1994 02:14:13 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from gmdzi.gmd.de by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id CAA09802; Mon, 31 Jan 1994 02:14:11 -0500 Received: from f1neuman.gmd.de (f1neuman) by gmdzi.gmd.de with SMTP id AA05121 (5.65c8/IDA-1.4.4 for ); Mon, 31 Jan 1994 08:12:21 +0100 Received: by f1neuman.gmd.de (AIX 3.2/UCB 5.64/4.03) id AA16792; Mon, 31 Jan 1994 08:14:16 GMT Date: Mon, 31 Jan 1994 08:14:16 GMT From: Rolf.Hempel@gmd.de (Rolf Hempel) Message-Id: <9401310814.AA16792@f1neuman.gmd.de> To: mpi-comm@CS.UTK.EDU Subject: buffering routines Cc: gmap10@f1neuman.gmd.de The flexibility in unpacking buffers which Paul outlined in his latest mail is certainly more than most users would wish for. By incremental unpacking, the user can decide at every step how to proceed with unpacking the rest of the buffer. In particular, he or she can encode a description of the buffer contents at the beginning of the buffer. So, an unpacking by trial and error does not seem necessary to me. If this flexibility comes at no price, however, it's fine with me. On a different vein, in my latest message I argued that the user would not manipulate the size_so_far argument in between calls to the unpacking routine, so we could make it an inout. Now it seems that Paul is our first user who does play with the size_so_far argument. The users are always more inventive than you immagine! - Rolf From owner-mpi-comm@CS.UTK.EDU Tue Feb 8 05:29:23 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id FAA23931; Tue, 8 Feb 1994 05:29:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id FAA13844; Tue, 8 Feb 1994 05:27:35 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 8 Feb 1994 05:27:33 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from gmdzi.gmd.de by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id FAA13801; Tue, 8 Feb 1994 05:27:28 -0500 Received: from f1neuman.gmd.de (f1neuman) by gmdzi.gmd.de with SMTP id AA07836 (5.65c8/IDA-1.4.4 for ); Tue, 8 Feb 1994 09:27:17 +0100 Received: by f1neuman.gmd.de (AIX 3.2/UCB 5.64/4.03) id AA17063; Tue, 8 Feb 1994 09:27:23 GMT Date: Tue, 8 Feb 1994 09:27:23 GMT From: Rolf.Hempel@gmd.de (Rolf Hempel) Message-Id: <9402080927.AA17063@f1neuman.gmd.de> To: mpi-comm@CS.UTK.EDU Subject: pack/unpack routines Cc: gmap10@f1neuman.gmd.de After reading Marc's text on pack/unpack routines, I have a few comments/questions: 1. From the definitions I conclude that it's allowed to use a regular send (without packing), and to receive the message into a character string with subsequent unpacking. In many cases one will need the unpacking on the receiver side only (because the structure of the message is encoded in the message), whereas on the sending side the standard send will be okay (and saves additional copying). The freedom to match a regular send with an unpack-receive will be very appreciated by users in those cases. On the other hand, this puts some constraint on how the pack/unpack is implemented, because the packed buffer must look the same as the byte stream which is produced by the regular send. In particular, it seems to me that the pack/unpack routines must be an integral part of the MPI implementation, and cannot be written as an add-on library. Is this our intention, or do I misunderstand anyhting here? 2. On page 69, second paragraph, (at least in my printout) it says that "the comm and source arguments in a call to MPI_UNPACK should match the arguments previously used to receive the packed buffer." Shouldn't we request that the unpack must use the actual source rank of the received message? Assume a heterogeneous process group and a wildcard receive. Then the unpack operation must know the exact source process for deciding on changes in data representations. Thus, a wildcard in the unpack source field will not help. 3. We already had the discussion on multiple destinations for a message. Marc suggests that in case of dest = MPI_ANY_DEST a common interchange format be used. This is only necessary in case of a heterogeneous communicator. Since we don't have a proposal for a MPI_ANY_COMM, the communicator must be specified explicitely. Thus, the implementation can decide to use XDR or not, depending on the uniformity of data representations. This will lead to less overhead in perhaps 99 percent of applications. - Rolf From owner-mpi-comm@CS.UTK.EDU Wed Feb 9 12:00:16 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id MAA10891; Wed, 9 Feb 1994 12:00:15 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id LAA25253; Wed, 9 Feb 1994 11:58:14 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 9 Feb 1994 11:58:12 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id LAA25246; Wed, 9 Feb 1994 11:58:10 -0500 Received: from WATSON by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 7949; Wed, 09 Feb 94 11:58:11 EST Received: from YKTVMV by watson.vnet.ibm.com with "VAGENT.V1.0" id 4965; Wed, 9 Feb 1994 11:58:09 EST Received: from snir.watson.ibm.com by yktvmv.watson.ibm.com (IBM VM SMTP V2R3) with TCP; Wed, 09 Feb 94 11:58:08 EST Received: by snir.watson.ibm.com (AIX 3.2/UCB 5.64/930311) id AA24620; Wed, 9 Feb 1994 11:58:08 -0500 From: snir@watson.ibm.com (Marc Snir) Message-Id: <9402091658.AA24620@snir.watson.ibm.com> To: Rolf.Hempel@gmd.de (Rolf Hempel) Subject: Re: pack/unpack routines Cc: mpi-comm@CS.UTK.EDU In-Reply-To: (Your message of Tue, 08 Feb 94 09:27:23 GMT.) Date: Wed, 09 Feb 94 11:58:08 EST Date: Wed, 09 Feb 94 11:58:08 -0500 Date: Tue, 8 Feb 1994 09:27:23 GMT From: Rolf.Hempel@gmd.de (Rolf Hempel) Message-Id: <9402080927.AA17063@f1neuman.gmd.de> To: mpi-comm@CS.UTK.EDU Subject: pack/unpack routines Cc: gmap10@f1neuman.gmd.de After reading Marc's text on pack/unpack routines, I have a few comments/questions: 1. From the definitions I conclude that it's allowed to use a regular send (without packing), and to receive the message into a character string with subsequent unpacking. In many cases one will need the unpacking on the receiver side only (because the structure of the message is encoded in the message), whereas on the sending side the standard send will be okay (and saves additional copying). The freedom to match a regular send with an unpack-receive will be very appreciated by users in those cases. On the other hand, this puts some constraint on how the pack/unpack is implemented, because the packed buffer must look the same as the byte stream which is produced by the regular send. In particular, it seems to me that the pack/unpack routines must be an integral part of the MPI implementation, and cannot be written as an add-on library. Is this our intention, or do I misunderstand anyhting here? --- It would be quite wasteful to write pack/unpack routines as an --- add-on library since they need all the derived datatype --- interpretation machinery that is required for regular --- communication. My intention is to have them, indeed, integrated. --- this requires little additional work, since the default --- communication mechanism uses pack/unpack in the current --- Argonne/UMiss/IBM/... implementations. 2. On page 69, second paragraph, (at least in my printout) it says that "the comm and source arguments in a call to MPI_UNPACK should match the arguments previously used to receive the packed buffer." Shouldn't we request that the unpack must use the actual source rank of the received message? Assume a heterogeneous process group and a wildcard receive. Then the unpack operation must know the exact source process for deciding on changes in data representations. Thus, a wildcard in the unpack source field will not help. --- I stand corrected 3. We already had the discussion on multiple destinations for a message. Marc suggests that in case of dest = MPI_ANY_DEST a common interchange format be used. This is only necessary in case of a heterogeneous communicator. Since we don't have a proposal for a MPI_ANY_COMM, the communicator must be specified explicitely. Thus, the implementation can decide to use XDR or not, depending on the uniformity of data representations. This will lead to less overhead in perhaps 99 percent of applications. --- We need a mechanism that does not force the use of XDR, even when --- communication is between different types of nodes. Only when --- the destination type is unknown at pack time, or the same packed --- message is to be sent to several different destinations, do we --- want to force the use of a machine indepedent representation, --- possibly different both form the source and the destination --- representation. Therefore, I suggested that the caller --- explicitly request the use of XDR when it packs, and that the --- unpack routine similarly ask for use of XDR. How this request --- is coded (ANY_SOURCE/ANY_DEST, or ANY_COMM, or ...) is something --- of secondary importance. - Rolf --- Marc From owner-mpi-comm@CS.UTK.EDU Sat Feb 12 00:09:47 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id AAA00632; Sat, 12 Feb 1994 00:09:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id AAA11936; Sat, 12 Feb 1994 00:08:00 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sat, 12 Feb 1994 00:07:59 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with ESMTP (8.6.4/2.8s-UTK) id AAA11929; Sat, 12 Feb 1994 00:07:57 -0500 Received: from mcs.anl.gov (donner.mcs.anl.gov [140.221.5.134]) by antares.mcs.anl.gov (8.6.4/8.6.4) with ESMTP id XAA05386 for ; Fri, 11 Feb 1994 23:07:58 -0600 Message-Id: <199402120507.XAA05386@antares.mcs.anl.gov> To: mpi-comm@CS.UTK.EDU Subject: Timers for MPI Date: Fri, 11 Feb 1994 23:07:57 -0600 From: Rusty Lusk Here are some thoughts on timer functions. Comments are welcome. Rusty Lusk Timers for MPI ============== A number of users have suggested that MPI provide a mechanism for timing sections of programs. While this is not strictly speaking part of message passing, parallel programmers, perhaps more than others, do a lot of timing, and have expressed the hope that a portable library like MPI could provide another portable function that they need. At the recent European meeting, users expressed particularly strongly the need for a timing mechanism that was portable, easy to use, and capable of expressing high resolution. Both wall-clock time and process time are needed. There has been a reluctance on the part of the MPI Forum to define a timing mechanism, on the grounds that other standards should do so. A tentative decision made at the European meeting was that we should mandate that every MPI implementation provide an appropriate subset of POSIX timer functions. I agreed to look into what that might mean precisely. In this note I survey the current Unix and POSIX mechanisms from the point of view of portability, ease of use, and high resolution, and I conclude that none of the existing POSIX mechanisms really meet these requirements. I conclude, somewhat reluctantly, that MPI should define its own, and make a concrete proposal. POSIX 1003.1-1988 ----------------- The "time" function for wall-clock time and the "times" function for process time are what are currently commonly available. time_t time ( time_t *tloc ) fills in the time_t structure with seconds "since the Epoch". Its resolution is too low. clock_t times ( tms *buffer ) fills in at least clock_t tms_utime (user time) clock_t tms_stime (system time) for the process, in the tms structure pointed to by buffer. It returns the elapsed time since an arbitrary time in the past. The type clock_t is defined to be a long integer, at least on some systems. The integer represents time in CLK_TCK-ths of a second, where this number is implementation-dependent, but is typically a 60th of a second. Therefore this function is not truly portable, since the numbers it returns may differ widely on different systems. It is also not typically high-resolution. Berkeley -------- Another common timer is gettimeofday, which is referred to as a "Berkeleyism" by the POSIX documents. It is not a standard, but is widely used. int gettimeofday ( tp, tzp ) where tp points to a structure containing at least long seconds; long microseconds; since midnight January 1, 1970, GMT. The variable tzp points to a time-zone structure, or is NULL. The resolution is allowed to vary per implementation, but the values do represent fixed time intervals. This is not a POSIX standard, allows for only medium-high resolution, and adds the unneeded cumbersomeness of the time-zone structure. (It also becomes obsolete in the year 2038.) POSIX P1003.4D 14.1 ------------------- This is the current POSIX draft proposal. Several calls use the timespec structure: time_t seconds long nanoseconds where time_t is the current structure used by time(), containing years, months, days, etc., down to seconds. Paragraph 14.2.1 - Clocks -------------------------- int clock_settime(clockid_t, clock_id, struct timespec *tp) int clock_gettime(clockid_t, clock_id, struct timespec *tp) int clock_getres(clockid_t, clock_id, struct timespec *tp) These functions are for setting and getting the value of assorted clocks. It is required that all POSIX implementations support at least one clock_id, called CLOCK_REALTIME. For this clock, time is measured "since the Epoch". So this is the POSIX version of gettimeofday() for getting wall-clock time, and its resolution can be retrieved explicitly. Paragraph 14.2.2 - Timers -------------------------- These functions manage timers for measuring process time, but primarily for managing alarm-type timers and signals. int timer_create ( clockid_t clock_id, struct sigevent *evp, timer_t *timerid ) int timer_delete ( timer_t timerid ) int timer_settime(timer_t timer_id, int flags, struct itimerspec *value, struct itimerspec *ovalue ) int timer_gettime(timer_t timerid, struct itimerspec *value) int timer_getoverrun(timer_t timerid ) For example, timer_gettime returns, in the itimerspec structure, the time left until the specified timer expires. This is not the kind of timer that the MPI users are looking for. For all of these, only the C binding is given in the POSIX documents. I will try to find out what Fortran-90 proposes, for example. Criticism --------- All of these are cumbersome to use, even in C. They do provide what is needed to implement the simple timers that people want, but they don't provide these functions themselves. Proposal -------- double MPI_Wtime() returns a floating-point number of seconds, representing elapsed wall-clock time since some time in the past. double MPI_Ptime() returns a floating-point number of seconds, representing elapsed process time since some time in the past. The "time in the past" is guaranteed not to change during the life of the process. The user is responsible for converting large numbers of seconds to other units if they are preferred. (Some things are *really* outside the scope of MPI.) The Fortran bindings are the same; the functions return double-precision floating point numbers. These functions are portable (return seconds, not "ticks"), allow high-resolution, and carry no unnecessary baggage. One would use them like this: { double starttime, endtime; starttime = double MPI_Wtime(); .... stuff to be timed ... endtime = double MPI_Wtime(); printf("That took %f seconds\n",endtime-starttime); } which is what I think the users had in mind. I suggest these be added to the environmental-inquiry chapter. Comments and improvements are welcome. From owner-mpi-comm@CS.UTK.EDU Mon Feb 14 08:25:13 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id IAA14741; Mon, 14 Feb 1994 08:25:12 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id IAA15820; Mon, 14 Feb 1994 08:23:45 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 14 Feb 1994 08:23:44 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from hub by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id IAA15799; Mon, 14 Feb 1994 08:23:36 -0500 Received: from tycho.co.uk (tycho.meiko.co.uk) by hub with SMTP id AA02751 (5.65c/IDA-1.4.4 for mpi-comm@CS.UTK.EDU); Mon, 14 Feb 1994 13:23:23 GMT Received: by tycho.co.uk (5.0/SMI-SVR4) id AA08667; Mon, 14 Feb 1994 13:22:50 +0000 Date: Mon, 14 Feb 1994 13:22:50 +0000 From: jim@meiko.co.uk (James Cownie) Message-Id: <9402141322.AA08667@tycho.co.uk> To: mpi-comm@CS.UTK.EDU In-Reply-To: <199402120507.XAA05386@antares.mcs.anl.gov> (message from Rusty Lusk on Fri, 11 Feb 1994 23:07:57 -0600) Subject: Re: Timers for MPI We should make clear that the time values returned are process local, and that even the epoch for the timers need not be the same on each node, also that the resolution may differ on different nodes ! This is implicit in the proposal, but should be mentioned explicitly as a warning point to users. In other words that it makes no sense to do something like read timer on node 0 broadcast value use value as base on all nodes instead you must do gsync remember start time on each node use local base value (This is NOT what users want, but it'll be hard to implement better on a network of workstations ...) -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Mon Feb 14 08:59:41 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id IAA14904; Mon, 14 Feb 1994 08:59:41 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id IAA19208; Mon, 14 Feb 1994 08:58:51 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 14 Feb 1994 08:58:43 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from sun2.nsfnet-relay.ac.uk by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id IAA19119; Mon, 14 Feb 1994 08:58:38 -0500 Via: uk.ac.daresbury.cxa; Mon, 14 Feb 1994 13:36:31 +0000 From: "R.J. Allan" Date: Mon, 14 Feb 94 13:36:57 GMT Message-Id: <6455.9402141336@uk.ac.dl.cxa> To: lusk@mcs.anl.gov Cc: mpi-comm@CS.UTK.EDU In-Reply-To: Rusty Lusk's message of Fri, 11 Feb 1994 23:07:57 -0600 <199402120507.XAA05386@antares.mcs.anl.gov> Subject: Timers for MPI I like Rust Lusk's suggestions for MPI_Wtime and MPI_Ptime. I beleive it is very important to have a fully portable implementation of such timers for both benchmarking and optimisation purposes and to inferface to high level tools for analysis and load balancing... Robert J. Allan Advanced Research Computing Group, Science and Engineering Research Council, Daresbury Laboratory, Daresbury, Warrington, WA4 4AD, U.K. 'phone +44 925 603207 FAX +44 925 603634 e-mail r.j.allan@cxa.ac.uk From owner-mpi-comm@CS.UTK.EDU Mon Feb 14 08:59:45 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id IAA14915; Mon, 14 Feb 1994 08:59:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id IAA19266; Mon, 14 Feb 1994 08:58:54 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 14 Feb 1994 08:58:49 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from sun2.nsfnet-relay.ac.uk by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id IAA19162; Mon, 14 Feb 1994 08:58:42 -0500 Via: uk.ac.daresbury.cxa; Mon, 14 Feb 1994 13:37:43 +0000 From: "R.J. Allan" Date: Mon, 14 Feb 94 13:38:06 GMT Message-Id: <6510.9402141338@uk.ac.dl.cxa> To: jim@meiko.co.uk Cc: mpi-comm@CS.UTK.EDU In-Reply-To: James Cownie's message of Mon, 14 Feb 1994 13:22:50 +0000 <9402141322.AA08667@tycho.co.uk> Subject: Timers for MPI I agree with James Cownies comments on the timers - all our software is programmed to work in this way using local clocks... Robert J. Allan Advanced Research Computing Group, Science and Engineering Research Council, Daresbury Laboratory, Daresbury, Warrington, WA4 4AD, U.K. 'phone +44 925 603207 FAX +44 925 603634 e-mail r.j.allan@cxa.ac.uk From owner-mpi-comm@CS.UTK.EDU Thu Feb 17 09:40:51 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id JAA18833; Thu, 17 Feb 1994 09:40:50 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id JAA18929; Thu, 17 Feb 1994 09:39:36 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 17 Feb 1994 09:39:34 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from dasher.cs.utk.edu by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id JAA18920; Thu, 17 Feb 1994 09:39:33 -0500 From: Jack Dongarra Received: by dasher.cs.utk.edu (5.61+IDA+UTK-930922/2.7c-UTK) id AA06325; Thu, 17 Feb 94 09:39:25 -0500 Date: Thu, 17 Feb 94 09:39:25 -0500 Message-Id: <9402171439.AA06325@dasher.cs.utk.edu> To: mpi-comm@CS.UTK.EDU Subject: draft agenda for mpi meeting Below is a draft agenda for the MPI meeting next week. If you would like to see other items added to the list please send them to me. Jack MPI Draft Agenda 2/23-25/1993 Wednesday 1:30 Report on Comments to the Draft (D. Walker) 2:30 Subcommittee reports on proposed changes 6:00 Dinner Thursday 9:00 Subcommittee reports on proposed changes (continued) 12:00 Lunch (provided) 1:00 Approval of changes 6:00 Dinner (Somewhere in downtown Knoxville) Friday 9:00 Approval of changes (continued) 10:00 Reference Implementation (R. Lusk) 10:30 MPI Book(s) (S. Otto) 11:00 Discussion of MPI2 We will be collecting the $75 registration fee as usual. From owner-mpi-comm@CS.UTK.EDU Thu Feb 17 10:11:05 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id KAA19133; Thu, 17 Feb 1994 10:11:04 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id KAA20841; Thu, 17 Feb 1994 10:10:34 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 17 Feb 1994 10:10:32 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with ESMTP (8.6.4/2.8s-UTK) id KAA20826; Thu, 17 Feb 1994 10:10:28 -0500 Received: from godzilla (godzilla.mcs.anl.gov [140.221.5.136]) by antares.mcs.anl.gov (8.6.4/8.6.4) with ESMTP id JAA25943 for ; Thu, 17 Feb 1994 09:10:30 -0600 From: William Gropp Date: Thu, 17 Feb 1994 09:09:14 -0600 Message-Id: <199402171509.JAA03394@godzilla> To: mpi-comm@CS.UTK.EDU Subject: Environmental Inquiry Draft Here is the updated Environment chapter. It contains a number of points for discussion at next week's meeting. Bill % % Updated to include comments from MPI goes to Europe... % % Here it is. I think I've handled the typos and problems. Let me know % if anything else needs to be done. % Bill % % \raggedbottom \newenvironment{discussion}{\ifvmode\else\par\fi\vspace{\discussSpace}% \begin{small}\narrower{\bf Discussion:}}{\ifvmode\else\par\fi\end{small}% \vspace{\discussSpace}} %\def\comm{comm} %\def\Comm{Communicator} %\begin{document} \chapter{MPI Environmental Management} \label{sec:environment} \label{chap:environment} %\footnotetext[1]{Version as of October 14, 1993} % changed significantly buffer setting and suggesting % added HOST inquiry % deleted type decode (because it is still incomplete) This chapter discusses routines for getting and, where appropriate, setting various parameters that relate to the MPI implementation and the execution environment. (such as error handling and communication buffering). The procedures for entering and leaving the MPI execution environment are also dexcribed here. \section{Implementation information} \label{sec:inquiry-impl} \subsection{Environmental Inquiries} \label{subsec:inquiry-inquiry} The following routines allow to inquire on the execution environment of an MPI program. \begin{funcdef}{MPI\_GET\_VALID\_TAG\_RANGE ( low, high )} \funcarg{\OUT}{low}{Smallest valid tag value (integer)} \funcarg{\OUT}{high}{Largest valid tag value (integer)} \end{funcdef} \mpibind{MPI\_Get\_valid\_tag\_range(int~*low, int~*high)} \mpifbind{MPI\_GET\_VALID\_TAG\_RANGE(LOW, HIGH, IERROR)\fargs INTEGER LOW, HIGH, IERROR} \noindent This routine gets the range of valid tags; that is, the user may use any tag value from \code{low} to \code{high} inclusive. These values are guaranteed to be unchanging during the execution of an MPI program. \change \begin{discussion} After MPI in Europe, there was considerable demand for the two features: \begin{enumerate} \item zero is guaranteed to be valid (low \(\le 0 \le\) high). \item high is guaranteed to be at least 127 (gives a minimum range that users can count on; virtually everyone expects a larger range). \end{enumerate} An alternate proposal is to eliminate \code{low}, requiring that \code{0} be the minimum value. \end{discussion} \begin{funcdef}{MPI\_GET\_PROCESSOR\_NAME( len, name )} \funcarg{\IN}{len}{Length (in characters) of the space in \code{name}} \funcarg{\OUT}{name}{A unique specifier for the actual (as opposed to virtual) node.} \end{funcdef} \mpibind{MPI\_Get\_processor\_name(int~len, char~*name)} \mpifbind{MPI\_GET\_PROCESSOR\_NAME(NAME, IERROR)\fargs CHARACTER*(*) NAME\\INTEGER IERROR} The name is a character string for maximum flexibility. From this value it must be possible to identify a specific piece of hardware; possible values include ``processor 9 in rack 4 of mpp.cs.org'' and ``231'' (where 231 is the actual processor number in the running homogeneous system). A error code of \const{MPI\_UNDEFINED} is allowed but strongly discouraged. The value of \const{MPI\_MAX\_PROCESSOR\_NAME} is the maximum length required by \mpifunc{MPI\_GET\_PROCESSOR\_NAME} (including, in the C binding, the terminating null). \change This routine returns the name of the processor on which it was called at the moment of the call. This is to allow MPI implementation that do process migration to return the current processor. Note that nothing in MPI {\em requires} or defines process migration; this definition of \mpifunc{MPI\_GET\_PROCESSOR\_NAME} simply allows such an implementation. \begin{funcdef}{MPI\_GET\_HOST( comm, rank)} \funcarg{\IN}{comm}{communicator (handle)} \funcarg{\OUT}{rank}{rank of {\tt HOST} in group of {\tt comm}; \const{MPI\_PROCNULL} if there is no {\tt HOST} (integer)} \end{funcdef} \mpibind{MPI\_Get\_host(MPI\_Comm~comm, int~*rank)} \mpifbind{MPI\_GET\_HOST(COMM, RANK, IERROR)\fargs INTEGER COMM, RANK, IERROR} This routine gets the rank of the {\tt HOST} process in the group associated with communicator {\tt comm}, if there is such. MPI does not specify what it means for a process to be a {\tt HOST}, nor does it requires that a {\tt HOST} exists. \subsubsection{Alternative} \change A set of predefined integer-valued attributes is associated with the communicator \const{MPI\_COMM\_WORLD} at initialization time. These define the execution environment of MPI programs. The values of these attributes can be retrieved by calling \mpifunc{MPI\_GET\_ATTR}. These attributes cannot be deleted. The list of predefined attributes keys include \begin{description} \item[\const{MPI\_TAG\_UB}] Upper bound for tag value \item[\const{MPI\_TAG\_LB}] Lower bound for tag value (if it is not set to be zero) \item[\const{MPI\_HOST}] Host process rank, if such exists, \const{MPI\_UNDEFINED}, otherwise. \item[\const{MPI\_SERVER}] Server rank, or 0/1 flag, indicating whether a server exists (depending on final decision on intercommunicator name servers) \item[\const{MPI\_IO}] rank of a node that has regular I/O facilities (possibly myrank). Nodes in the same communicator may return different values for this attribute. \item[\const{MPI\_ERROR}] error handler type: fatal, noerror, or user defined. \end{description} Vendors may add implementation specific attributes (such as node number, real memory size, virtual memory size, etc.) \begin{rationale} (1) Why have a slew of inquiry functions if we already have an attribute mechanism? (2) The open ended form of this mechanism allows to easily add attributes. \end{rationale} We may still want the \mpifunc{MPI\_GET\_PROCESSOR\_NAME} call, which returns a string. \subsection{Buffering} \label{subsec:inquiry-buffer} While it is possible to design algorithms that do not require that any buffering be provided for sending and receiving ``blocking'' messages, many practitioners expect that the message-passing system will provide some buffering. That is, there is often an expectation that the program fragment, executed by all tasks, \begin{verbatim} MPI_send( buf, size, datatype, destination, tag, comm ) MPI_recv( buf, size, datatype, destination, tag, comm ) \end{verbatim} \noindent will execute for messages whose \const{size} is not too large. In fact, this code may deadlock at the first \code{MPI\_send}. The purpose of this section is to give the user a way to determine what ``too large'' is and to help the implementation provide enough buffering for the user's need. Note that nothing in this section {\em requires\/} an implementation to provide any buffering; that is, an implementation may define the above program fragment to be erroneous for any positive value of \const{size}. \subsubsection{General buffering parameters} \begin{funcdef}{MPI\_GET\_BUFFER\_PARAMS( numsend, sizesend, numrecv, sizerecv, totalnumsend, totalsizesend, totalnumrecv, totalsizerecv)} \funcarg{\OUT}{numsend}{Bound on the number of pending messages with source at calling process and same destination} \funcarg{\OUT}{sizesend}{Bound on the total size in bytes of pending messages with source at calling process and same destination} \funcarg{\OUT}{numrecv}{Bound on the number of pending messages with destination at calling process and same source} \funcarg{\OUT}{sizerecv}{Bound on the total size in bytes of pending messages with destination at calling process and same source} \funcarg{\OUT}{totalnumsend}{Bound on the number of pending messages with source at the calling process and any destination} \funcarg{\OUT}{totalsizesend}{Bound on the total size in bytes of pending messages with source at the calling process and any destination} \funcarg{\OUT}{totalnumrecv}{Bound on the number of pending messages from any source with the calling process as destination} \funcarg{\OUT}{totalsizerecv}{Bound on the total size in bytes of pending messages from any source with destination at the calling process} \end{funcdef} \mpibind{MPI\_Get\_buffer\_params(int~*numsend, int~*sizesend, int~*numrecv, int~*sizerecv, int~*totalnumsend, int~*totalsizesend, int~*totalnumrecv, int~*totalsizerecv)} \mpifbind{MPI\_GET\_BUFFER\_PARAMS(NUMSEND, SIZESEND, NUMRECV, SIZERECV, TOTALNUMSEDND, TOTALSIZESEN, TOTALNUMRECV, TOTALSIZERECV, IERROR)\fargs INTEGER NUMSEND, SIZESEND, NUMRECV, SIZERECV, TOTALNUMSEDND, TOTALSIZESEN, TOTALNUMRECV, TOTALSIZERECV, IERROR} A program where at any point in time no process has more then \mpiarg{numsend} pending messages sent by it to the same destination, no more than \mpiarg{sizesend} bytes of data in pending messages sent by it to the same destination, no more than \mpiarg{totalnumsend} pending messages sent by it to any destination, no more than \mpiarg{totalsizesend} bytes of data in pending messages sent by it to any destination, no more than \mpiarg{numrecv} pending messages sent to it from the same source, no more than \mpiarg{sizerecv} bytes in pending messages sent to it from the same source, no more than \mpiarg{totalnumrecv} pending messages sent to it from any source, and no more than \mpiarg{totalsizerecv} bytes in pending messages sent to it from any source is guaranteed not to deadlock because of the lack of buffer space for pending messages. Examples: 1. Suppose that pending messages are always buffered at the sender, all buffers are shared, total buffer size at the process is {\tt B}, each pending message takes {\tt h} bytes of storage, in addition to the storage for its data, and no storage is lost due to fragmentation. Then possible return values for \mpifunc{MPI\_GET\_BUFFER\_PARAMS} are {\tt numsend = totalnumsend = k, sizesend = totalsizesend = B-kh}, and {\tt numrecv = totalnumrecv = sizerecv = totalsizerecv = INT\_MAX}, for some $\tt 0 < k < B/h$. 2. Suppose that messages are always buffered at the receiver, and that a separate buffer with {\tt C} bytes is allocated for each pair of communicating processes; entries for pending messages are kept in a seeparate table, with room for total of {\tt m} pending messages at each receiving process; the total number of processes is {\tt p}, and no storage is lost due to fragmentation. Then possible return values for \mpifunc{MPI\_GET\_BUFFER\_PARAMS} are {\tt sendcount = totalsendcount = sendsize = totalsendsize = INT\_MAX, recvcount = totalrecvcount = m, recsize = C} and {\tt totalrecvsize = (p-1)$\times$C}. The call may return the value \const{MPI\_UNDEFINED} in any argument. If this is the case, then the MPI implementation makes no guarantee about buffering. Some implementations may be able to adjust their buffering. In addition, since the parameters returned by \mpifunc{MPI\_GET\_BUFFER\_PARAMS} represent a guarantee about a particular choice of number of buffers, it may not guarantee buffering that it can in fact perform. Thus, in example 1 above, the implementation picks an arbitrary value for {\tt k}, trading off number of messages for total message size, rather than letting the user make this trade-off. In order to allow the MPI user to inquire about a specific buffering need, the routine \mpifunc{MPI\_SUGGEST\_BUFFER\_PARAMS} is provided. \begin{funcdef}{MPI\_SUGGEST\_BUFFER\_PARAMS( numsend, sizesend, numrecv, sizerecv, totalnumsend, totalsizesend, totalnumrecv, totalsizerecv, flag)} \funcarg{\IN}{numsend}{Bound on number of pending messages with source at calling process and same destination} \funcarg{\IN}{sizesend}{Bound on total size in bytes of pending messages with source at calling process and same destination} \funcarg{\IN}{numrecv}{Bound on the number of pending messages with destination at calling process and same source} \funcarg{\IN}{sizerecv}{Bound on total size in bytes of pending messages with destination at calling process and same source} \funcarg{\IN}{totalnumsend}{Bound on the number of pending messages with source at the calling process and any destination} \funcarg{\IN}{totalsizesend}{Bound on total size in bytes of pending messages with source at the calling process and any destination} \funcarg{\IN}{totalnumrecv}{Bound on the number of pending messages from any source with the calling process as destination} \funcarg{\IN}{totalsizerecv}{Bound on the total size in bytes of pending messages from any source with destination at the calling process} \funcarg{\OUT}{flag}{\const{TRUE} if the suggestion was accepted, \const{FALSE}, otherwise} \end{funcdef} \mpibind{MPI\_Suggest\_buffer\_params(int~numsend, int~sizesend, int~numrecv, int~sizerecv, int~totalnumsend, int~totalsizesen, int~totalnumrecv, int~totalsizerecv, int~*flag)} \mpifbind{MPI\_SUGGEST\_BUFFER\_PARAMS(NUMSEND, SIZESEND, NUMRECV, SIZERECV, TOTALNUMSEND, TOTALSIZESEN, TOTALNUMRECV, TOTALSIZERECV, FLAG, IERROR)\fargs INTEGER NUMSEND, SIZESEND, NUMRECV, SIZERECV, TOTALNUMSEND, TOTALSIZESEN, TOTALNUMRECV, TOTALSIZERECV, IERROR \\ LOGICAL FLAG} If a value of true is returned in \mpiarg{flag} then it is as if the values provided by the user in \mpifunc{MPI\_SUGGEST\_BUFFER\_PARAMS} were returned by a call to \mpifunc{MPI\_GET\_BUFFER\_PARAMS}: the user is guaranteed that no deadlock will occur due to lack of buffers, provided it never exceeds the limitations on number and size of pending messages implied by these parameters. This routines allows the user to tell the MPI implementation that a certain amount of buffering would be desireable, The MPI implementation is not required to provide such buffering. It is expected that a common use of these routines will be \begin{verbatim} MPI_Get_buffer_params( &sendnum, &sendsize, ... ); if (sendsize < buffer_needed) { fprintf( stderr, "Program requires more buffering than is available\n" ); MPI_Abort( 1 ); } else exploit buffering \end{verbatim} This allows a program to fail rather than deadlock. It is also possible for a program to switch to a different algorithm that requires less or no buffering. Another use is to inform the MPI implementation of the buffer space desires of the application: \begin{verbatim} MPI_Suggest_buffer_params( 1, buffer_needed, 1, buffer_needed, 0, 0, 0, 0, &flag ); if (flag) { exploit buffering } else { fprintf( stderr, "Program requires more buffering than is available\n" ); MPI_abort( 1 ); } \end{verbatim} \discuss{An implementation is free to accept suggestions only after \mpifunc{MPI\_INIT} and before any MPI call that performs any communication. It is always correct to call \mpifunc{MPI\_SUGGEST\_BUFFER\_PARAMS} after \mpifunc{MPI\_INIT}.} \discuss{ These functions are complex (eight parameters), and still do not provide an accurate information on an implementation (do nonblocking messages share buffer space with blocking ones? How many messages are generated by collective communications? Or communicator generating calls?) Are these functions really useful, as a ``noop'' implementation is legitimate? especially if we have packing/unpacking? The generic attribute mechanism can be used for these communicator parameters, if we wish to do so. I.e., add eight buffering attributes to each communicator, that can be queried and sometimes modified. Problems with this approach are that the eight-parameter model allows the values to be coupled (messages traded against buffer space). User comments at MPI in Europe were extremely negative about MPI not mandating large amounts of buffering. Many users also read this as mandating no buffering; this caused a great deal of confusion. } \subsubsection{Application provided buffering} An application program can optionally provide space for MPI's use in buffering message data via the following call: % %\begin{funcdef}{MPI\_USER\_SPECIFIES\_BUFFER( comm, b, t, flag )} %\funcarg{\IN}{comm}{ is a communicator with which a %buffer is to be associated} %\funcarg{\IN}{b}{the length of the buffer, in bytes} %\funcarg{\IN}{t}{the number of blocking sends that can be outstanding %while sharing this buffer} %\funcarg{\OUT}{flag}{indicates whether the request was successful. If %the call is not successful, flag indicates why by using %\code{MPI\_TOO\_LARGE} for \code{b} to large and \code{MPI\_TOO\_MANY} %for \code{t} too large. If both \code{b} and \code{t} are too large, an %implemtation is free to return either \code{MPI\_TOO\_LARGE} or %\code{MPI\_TOO\_MANY}.} %\end{funcdef} % %\mpibind{MPI\_User\_specifies\_buffer(MPI\_Comm~comm, int b, int t, int *flag )} % %\mpifbind{MPI\_USER\_SPECIFIES\_BUFFER(COMM, B, T, FLAG)\fargs INTEGER COMM, B, T, FLAG} \begin{funcdef}{MPI\_USER\_SPECIFIES\_BUFFER( comm, buffer, size)} \funcarg{\IN}{comm}{ is a communicator with which the buffer is to be associated (handle)} \funcarg{\IN}{buffer}{initial buffer address (choice)} \funcarg{\IN}{size}{buffer size, in bytes (integer)} \end{funcdef} \mpibind{MPI\_User\_specifies\_buffer(MPI\_Comm~comm, void*~buffer, int~size)} \mpifbind{MPI\_USER\_SPECIFIES\_BUFFER(COMM, BUFFER, SIZE, IERROR)\fargs BUFFER(*) \\ INTEGER COMM, SIZE, IERROR} If \mpifunc{MPI\_USER\_SPECIFIES\_BUFFER} is called and the value returned in \code{flag} is \code{MPI\_OK}, then for subsequent regular sends using the specified communicator, MPI must provide as much safety {\em as if\/} outgoing message data were buffered by the sending process, in the specified buffer space, using a circular contiguous-space allocation policy. %Following the call to \mpifunc{MPI\_USER\_SPECIFIES\_BUFFER}, the %application is not permitted to access the buffer until the %corresponding communicator has been freed. The buffer is managed by MPI. Access to the provided buffer space is not inherited by communicators derived from the initially specified one. \implement{ One approach is for MPI to simply transform all blocking sends on the specified communicator into \begin{itemize} \item copy data to buffer space \item issue non-blocking send from the buffer copy \item return to application \item check completion on subsequent MPI call(s) \end{itemize} No doubt many optimizations within MPI are possible---the proposal just requires that MPI provide at least this much safety. %The reason that MPI provides the buffer rather than the user is that once the %user frees a communicator, only MPI knows when all data-structures associated %with that communicator can be freed. } Note that collective communications and other nonlocal operations will share suggested or user-supplied communication buffers with point to point communication. \discuss{ When a communicator associated with a user buffer is freed, then \mpifunc{MPI\_COMM\_FREE} should not return until all pending communications have terminated. Thus, after it returns, the buffer can be freed. } \section{Error handling} \label{sec:inquiry-error} This section contains the suggested POSIX binding for error handling in MPI. MPI implementations that do not run in a POSIX-compliant environment may need to use alternative mechanisms. An MPI implementation cannot or may choose not to handle some errors that occur during MPI calls. These can include errors that generate exceptions or traps, such as floating point errors or access violations. The set of errors that are handled by MPI is implementation-dependent. Each such error generates an {\bf MPI exception}. A user can associate an error handler with a communicator. The specified error handling routine will be used for any MPI exception that occurs during a call to MPI for a communication with this communicator. MPI calls that are not related to any communicator are considered to be attached to the communicator \const{MPI\_COMM\_WORLD}. The attachment of error handlers to communicators is purely local: different processes may attach different error handlers to the same communicator. A newly created communicator inherits the error handler that is associated with the ``parent'' communicator. In particular, the user can specify a ``global'' error handler for all communicators by associating this handler with the communicator \const{MPI\_COMM\_WORLD} immediately after initialization. Several predefined error handlers are available in MPI: \begin{description} \item[\const{MPI\_ERRORSFATAL}] The handler, when called, causes the program to abort on all executing processes. This has the same effect as if \mpifunc{MPI\_ABORT} was called by the process that invoked the handler. \item[\const{MPI\_ERRORSRETURN}] The handler has no effect. \end{description} Implementations may provide additional predefined error handlers and programmers can code their own error handlers. The error handler \const{MPI\_ERRORSFATAL} is associated by default with \const{MPI\_COMM\_WORLD} after initialization. Thus, if the user chooses not to control error handling, every error that MPI handles is treated as fatal. Since (almost) all MPI calls return an error code, a user may choose to handle errors in its main code, by testing the return code of MPI calls and executing a suitable recovery code when the call was not successful. In such case, the error handler \const{MPI\_ERRORSRETURN} will be used. Usually it is more convenient and more efficient not to test for errors after each MPI call, and have such error handled by a non trivial MPI error handler. \discuss{ Do we specify that \const{MPI\_ERRORSFATAL} should generate a \const{SIGABRT} signal at each executing process? Is that the desired effect? (e.g., so as to allow a debugger to take over.) Same remark applies to \mpifunc{MPI\_ABORT}. } An MPI error handler is an opaque object, which is accessed by a handle. MPI calls are provided to create new error handlers, to associate error handlers with communicators, and to test which error handler is associated with a communicator. \begin{funcdef}{MPI\_CREATE\_ERRHANDLER( function, errhandler )} \funcarg{\IN}{function}{user defined error handling procedure} \funcarg{\OUT}{errhandler}{MPI error handler (handle)} \end{funcdef} \mpibind{MPI\_Create\_errhandler(MPI\_Handlerfunction~*function, MPI\_Errhandler~*errhandler)} \mpifbind{MPI\_CREATE\_ERRHANDLER(FUNCTION, HANDLER, IERROR)\fargs EXTERNAL FUNCTION \\ INTEGER ERRHANDLER, IERROR} Register the user routine \mpiarg{handler\_function} for use as an MPI exception handler. Returns in \mpiarg{errhandler} a handle to the registered exception handler. \implement{ The handle returned may contain the address of the error handling routine. Such call is superfluous in C, which has a referencing operator, but is necessary in Fortran.} The user routine should be a C function of type \const{MPI\_Handler\_function}, which is defined as \begin{verbatim} typedef void (MPI_Handler_function)(MPI_Comm, int , ...); \end{verbatim} The first argument is the communicator in use. The second is the error code to be returned by the MPI routine. The remaining arguments are ``\code{varargs}'' arguments whose number and meaning is implementation-dependent. An implementation should clearly document these arguments. \begin{rationale} The variable argument list is provided because it provides an ANSI-standard hook for providing additional information to the error handler; without this hook, ANSI C prohibits additional arguments. \end{rationale} \begin{funcdef}{MPI\_SET\_ERRHANDLER( comm, errhandler )} \funcarg{\IN}{comm}{communicator to set the error handler for (handle)} \funcarg{\IN}{errhandler}{new MPI error handler for communicator (handle)} \end{funcdef} \mpibind{MPI\_Set\_errhandler(MPI\_Comm~comm, MPI\_Errhandler~errhandler)} \mpifbind{MPI\_SET\_ERRHANDLER(COMM, ERRHANDLER, IERROR)\fargs INTEGER COMM, ERRHANDLER, IERROR} Associates the new error handler \mpiarg{errorhandler} with communicator \mpiarg{comm} at the calling process. \begin{funcdef}{MPI\_GET\_ERRHANDLER( comm, errhandler )} \funcarg{\IN}{comm}{communicator to get the error handler from (handle)} \funcarg{\OUT}{errhandler}{MPI error handler currently associated with communicator (handle)} \end{funcdef} \mpibind{MPI\_Get\_errhandler(MPI\_Comm~comm, MPI\_Errhandler~*errhandler)} \mpifbind{MPI\_GET\_ERRHANDLER(COMM, ERRHANDLER, IERROR)\fargs INTEGER COMM, ERRHANDLER, IERROR} Returns in \mpiarg{errhandler} (a handle to) the error handler that is currently associated with communicator \mpiarg{comm}. Example: A library function may register at its entry point the current error handler for a communicator, set its own private error handler for this communicator, and restore before exiting the previous error handler. \discuss{ One may have the function \mpifunc{MPI\_SET\_ERRHANDLER} both set a new error handler and return the previous error handler (the function would have two errhandler arguments), thus saving one call when both \_GET and \_SET are needed. But, in any case, one needs both the old and new error-handlers. } \begin{funcdef}{MPI\_ERROR\_STRING( errorcode, len, string )} \funcarg{\IN}{errorcode}{Error code returned by an MPI routine} \funcarg{\IN}{len}{Length of \const{string}} \funcarg{\OUT}{string}{Text that corresponds to the \const{errorcode}} \end{funcdef} \mpibind{MPI\_Error\_string(int~errorcode, int~len, char~*string)} \mpifbind{MPI\_ERROR\_STRING(ERRORCODE, STRING, IERROR)\fargs INTEGER ERRORCODE, IERROR \\ CHARACTER(*) STRING} Returns the error string associated with an error code. \begin{rationale} The form of this proposal was chosen to make the Fortran and C bindings similar. A version that returns a pointer to a string has two difficulties. First, the return string must be statically allocated and different for each error message (allowing the pointers returned by successive calls to \const{MPI\_ERROR\_STRING} to point to the correct message. Second, in Fortran, a function declared as returning \const{CHARACTER*(*)} can not be referenced in, for example, a \const{PRINT} statement. \end{rationale} \section{Error codes} MPI specifies some of the error return values that an MPI implementation may return. This is not an exhaustive list, but MPI implementations must use these where appropriate. \change \begin{center} \begin{tabular}{ll} MPI\_ERR\_BUFFER &Invalid buffer pointer \\ MPI\_ERR\_COUNT &Invalid count argument \\ MPI\_ERR\_TYPE &Invalid datatype argument \\ MPI\_ERR\_TAG &Invalid tag argument \\ MPI\_ERR\_COMM &Invalid communicator\\ MPI\_ERR\_RANK &Invalid rank \\ MPI\_ERR\_REQUEST &Invalid request (handle)\\ MPI\_ERR\_ROOT &Invalid root \\ MPI\_ERR\_GROUP &Invalid group\\ MPI\_ERR\_OP &Invalid operation \\ MPI\_ERR\_TOPOLOGY &Invalid topology \\ MPI\_ERR\_DIMS &Invalid dimension argument \\ MPI\_ERR\_ARG &Invalid argument of some other kind \\ MPI\_ERR\_UNKNOWN &Unknown error \\ MPI\_ERR\_TRUNCATE &Message truncated on receive\\ MPI\_ERR\_INTERN &Internal MPI error\\ MPI\_ERR\_LASTCODE &Last standard error code\\ \end{tabular} \end{center} \section{Timers} MPI defines two timers. These timers are specified even though they are not ``message-passing'' because timing parallel programs is important in ``performance debugging'' and because existing timers (both in POSIX 1003.1-1988 and 1003.4D 14.1 and in Fortran 90) are either inconvenient or do not provide adequate access to high-resolution timers. \begin{funcdefna}{MPI\_WTIME()} \end{funcdefna} \mpibind{double MPI\_Wtime()} \mpifbind{DOUBLE PRECISION MPI\_WTIME()} returns a floating-point number of seconds, representing elapsed wall-clock time since some time in the past. \begin{funcdefna}{MPI\_PTIME()} \end{funcdefna} \mpibind{double MPI\_Ptime()} \mpifbind{DOUBLE PRECISION MPI\_PTIME()} returns a floating-point number of seconds, representing elapsed process time since some time in the past. The ``time in the past'' is guaranteed not to change during the life of the process. The user is responsible for converting large numbers of seconds to other units if they are preferred. These functions are portable (return seconds, not "ticks"), allow high-resolution, and carry no unnecessary baggage. One would use them like this: \begin{verbatim} { double starttime, endtime; starttime = double MPI_Wtime(); .... stuff to be timed ... endtime = double MPI_Wtime(); printf("That took %f seconds\n",endtime-starttime); } \end{verbatim} The times returned by these routines are local to the node that called them. There is no requirement that different nodes return ``the same time.'' \begin{discussion} User's at MPI in Europe were strongly in favor of having timing functions. \end{discussion} \section{Startup} \label{sec:inquiry-startup} One goal of MPI is to achieve {\em source code portability}. By this we mean that a program written using MPI and complying with the relevant language standards is portable as written, and must not require any source code changes when moved from one system to another. This explicitly does {\em not\/} say anything about how an MPI program is started or launched from the command line, nor what the user must do to set up the environment in which an MPI program will run. However, an implementation may require some setup to be performed before other MPI routines may be called. To provide for this, MPI includes an initialization routine \mpifunc{MPI\_INIT}. \begin{funcdefna}{MPI\_INIT()} \end{funcdefna} \mpibind{MPI\_Init(int~*argc, char~**argv)} \mpifbind{MPI\_INIT(IERROR)\fargs INTEGER IERROR} This routine must be called before any other MPI routine. It must be called at most once; subsequent calls are erroneous (see \mpifunc{MPI\_INITIALIZED}). All MPI programs must contain a call to \mpifunc{MPI\_init}; this routine must be called before any other MPI routine (apart from \code{MPI\_INITIALIZED}) is called. The version for ANSI C accepts the \const{argc} and \const{argv} that are provided by the arguments to \code{main}: \begin{verbatim} MPI_init( argc, argv ); \end{verbatim} The Fortran version takes no arguments. \discuss{Note that POSIX includes a {\em requirement\/} that access to the command line be provided to Fortran programs (see 8.9 in Std 1003.9--1992). MPI implementations that read command-line arguments should insure the the MPI command-line arguments are clearly identifed, for example, by prefixing them with \code{-mpi}. Note the use of the {\em address} of \code{argc} in the C binding of \mpifunc{MPI\_INIT}. This is to allow \mpifunc{MPI\_INIT} to remove those arguments that it uses from the argument list. } \begin{funcdefna}{MPI\_FINALIZE()} \end{funcdefna} \mpibind{MPI\_Finalize()} \mpifbind{MPI\_FINALIZE(IERROR)\fargs INTEGER IERROR} This routines cleans all MPI state. Once this routine is called, no MPI routine (even \mpifunc{MPI\_INIT}) may be called. The user must ensure that all pending communications involving a process complete before the process calls \mpifunc{MPI\_FINALIZE}. \begin{funcdef}{MPI\_INITIALIZED( flag )} \funcarg{\OUT}{flag}Flag is true if \mpifunc{MPI\_INIT} has been called and false otherwise. \end{funcdef} \mpibind{MPI\_Initialized(int~*flag)} \mpifbind{MPI\_INITIALIZED(FLAG, IERROR)\fargs LOGICAL FLAG \\ INTEGER IERROR} This routine may be used to determine whether \mpifunc{MPI\_INIT} has been called. It is the {\em only} routine that may be called before \mpifunc{MPI\_INIT} is called. \begin{funcdef}{MPI\_ABORT( comm, errorcode )} \funcarg{\IN}{comm}{communicator of tasks to abort} \funcarg{\IN}{errorcode}{error code to return to invoking environment} \end{funcdef} \mpibind{MPI\_Abort(MPI\_Comm~comm, int~errorcode)} \mpifbind{MPI\_ABORT(COMM, ERRORCODE, IERROR)\fargs INTEGER COMM, ERRORCODE, IERROR} This routine makes a ``best attempt'' to abort all tasks in the group of \mpiarg{comm}. This function does not require that the invoking environment take any action with the error code. However, a Unix or POSIX environment should handle this as a \code{return errorcode} from the main program or an \code{abort(errorcode)}. MPI implementations are required to define the behavior of \mpifunc{MPI\_ABORT} at least for a \code{comm} of \const{MPI\_COMM\_WORLD}. From owner-mpi-comm@CS.UTK.EDU Thu Feb 17 14:03:08 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id OAA21239; Thu, 17 Feb 1994 14:03:08 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id OAA08123; Thu, 17 Feb 1994 14:01:56 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 17 Feb 1994 14:01:54 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with ESMTP (8.6.4/2.8s-UTK) id OAA08116; Thu, 17 Feb 1994 14:01:52 -0500 Received: from godzilla (godzilla.mcs.anl.gov [140.221.5.136]) by antares.mcs.anl.gov (8.6.4/8.6.4) with ESMTP id NAA03987; Thu, 17 Feb 1994 13:01:50 -0600 From: William Gropp Date: Thu, 17 Feb 1994 13:01:48 -0600 Message-Id: <199402171901.NAA23335@godzilla> To: D.Lamptey@sheffield.ac.uk Cc: mpi-comm@CS.UTK.EDU In-reply-to: <9402171637.AA00531@ntsc.shef.ac.uk> (D.Lamptey@sheffield.ac.uk) Subject: Re: Environmental Inquiry Draft I've placed a compressed postscript version of the inquiry chapter in pub/mpi/inquiry.ps.Z on info.mcs.anl.gov . Bill From owner-mpi-comm@CS.UTK.EDU Thu Mar 24 22:48:51 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id WAA14917; Thu, 24 Mar 1994 22:48:51 -0500 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.8s-UTK) id WAA21780; Thu, 24 Mar 1994 22:46:44 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 24 Mar 1994 22:46:43 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (cf v2.8s-UTK) id WAA21773; Thu, 24 Mar 1994 22:46:42 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA12662; Thu, 24 Mar 94 21:43:50 CST Date: Thu, 24 Mar 94 21:43:50 CST From: Tony Skjellum Message-Id: <9403250343.AA12662@Aurora.CS.MsState.Edu> To: mpi-comm@CS.UTK.EDU Subject: fyi From ned1@Ra.MsState.Edu Thu Mar 24 20:33:50 1994 Received: from Walt.CS.MsState.Edu by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA12616; Thu, 24 Mar 94 20:33:50 CST Received: from Tut.MsState.Edu by Walt.CS.MsState.Edu (4.1/6.0s-FWP); id AA26642; Thu, 24 Mar 94 20:36:33 CST Received: from Isis.MsState.Edu (ned1@Isis.MsState.Edu [130.18.80.11]); by Tut.MsState.Edu using ESMTP (8.6.7/6.5m-FWP); id UAA21050; Thu, 24 Mar 1994 20:36:28 -0600 From: Nathan Doss Received: by Isis.MsState.Edu (8.6.7/6.0c-FWP); id UAA11791; Thu, 24 Mar 1994 20:36:23 -0600 Date: Thu, 24 Mar 1994 20:36:23 -0600 Message-Id: <199403250236.UAA11791@Isis.MsState.Edu> To: tony@cs.MsState.Edu, gropp@mcs.anl.gov, lusk@mcs.anl.gov, doss@erc.msstate.edu Subject: LAM Status: R >Path: nntp.msstate.edu!darwin.sura.net!howland.reston.ans.net!math.ohio-state.edu!mane.cgrg.ohio-state.edu!tbag.osc.edu!not-for-mail >From: gdburns@osc.edu (Greg Burns) >Newsgroups: comp.parallel.pvm >Subject: Get started with MPI now (s/w available). >Date: 24 Mar 1994 16:52:43 -0500 >Organization: The Ohio Supercomputer Center >Lines: 111 >Distribution: world >Message-ID: <2mt23b$3bk@tbag.osc.edu> >NNTP-Posting-Host: tbag.osc.edu Ohio Supercomputer Center (OSC) announces the general release of LAM, a UNIX cluster implementation of the MPI interprocess communication standard. The software is available in source code form under the terms of the GNU general license (V2). Distribution is via anonymous ftp from tbag.osc.edu under pub/lam. The factory default software supports the following platforms: Sun 4 (sparc), SunOS 4.1.3 Sun 4 (sparc), Solaris 2.3 SGI IRIX 4.0.5 IBM RS/6000, AIX v3r2 DEC AXP, OSF/1 V1.3 Technical Summary Key Features - standard MPI communication - heterogeneous networks - micro-kernel, multi-process, modular structure - hands-on monitoring and control - easy transition to dedicated h/w - PVM compatibility - parallel I/O Standard MPI Interprocess Communication The MPI Forum, which included representatives from every end of the parallel computing community, has specified a complete interface for message based interprocess communication, named MPI (Message Passing Interface). Widespread use of MPI will benefit the general advancement of parallel processing technology. The specific semantics of the communication interface, its perceived popular and future prospects, need no longer be a factor in choosing an environment or a vendor for developing message passing applications. Users and buyers can focus on other factors such as efficiency of implementation and related development tools. Given the breadth of participation in MPI Forum and the depth of their deliberations (which have been available on netlib), widespread adoption by vendors and acceptance by users seems likely. LAM is a non-proprietary platform for users to evaluate MPI hands-on and to begin developing MPI applications. LAM is a full cluster computing environment. MPI does not specify file I/O, machine configuration and process management, so LAM fills in these gaps to form a complete system. LAM has debugging commands to examine the MPI synchronization status of processes and messages. At the time of this initial release, the first MPI specification has not been finalized. The implementation for the current release is confined to the sanctioned MPI subset and is based on the draft document of Nov. 2, 1993. LAM's MPI implementation of MPI will be updated as soon as the final document is released. We are confident that the current implementation is a very usable starting point. PVM Compatibility LAM also has a library to run PVM 3 applications. As with MPI, there are debugging commands to examine the PVM synchronization status of processes and messages. Configuration and Process Management LAM unpacks and installs anywhere. The user shell's search path is used to locate the LAM server, utilities and application programs on every machine. Configuring a cluster is listing machine names in a file and invoking a utility. Process management is completely dynamic. The user or any process situated at any machine can run a program residing in any second machine on any third machine. NFS is not required. The user or any process can kill any other process. An executable may reside in the target node's file space or LAM can transfer the file from a third node. I/O When NFS or friends are not readily available, LAM has a set of functions that mirror the POSIX functions and provide remote access to the filesystem of other machines. User Interface LAM's UI is just your usual shell. There is no special environment, only a set of utility programs. Documentation There are user guides for C and Fortran programmers as well as a complete set of manual pages covering every function and command program. -=- LAM is a subset of Trollius, an environment for UNIX clusters + MPP. Trollius was originally developed at the Cornell Theory Center. Trollius is a trademark of The Ohio State University and the Cornell Research Foundation. The MPI implementation is based in part on the implementation provided by Argonne National Laboratory and Mississippi State University. LAM development was sponsored by OSC, OSU, Cray Research Inc., and Alex Informatique. From owner-mpi-comm@CS.UTK.EDU Mon Apr 4 12:32:50 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id MAA15724; Mon, 4 Apr 1994 12:32:50 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.8s-UTK) id MAA16491; Mon, 4 Apr 1994 12:31:02 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 4 Apr 1994 12:30:58 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (cf v2.8s-UTK) id MAA16482; Mon, 4 Apr 1994 12:30:57 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA19546; Mon, 4 Apr 94 11:26:22 CDT Date: Mon, 4 Apr 94 11:26:22 CDT From: Tony Skjellum Message-Id: <9404041626.AA19546@Aurora.CS.MsState.Edu> To: mpi-comm@CS.UTK.EDU Subject: Posting of MPI newsgroup RFD I have posted the following request for discussion for comp.parallel.mpi, as we agreed at the last meeting. - Tony Skjellum Date: 4 Apr 94 16:18:02 GMT Message-ID: Newsgroups: news.announce.newgroups,comp.parallel,comp.parallel.pvm Subject: RFD: comp.parallel.mpi unmoderated Summary: An unmoderated newsgroup, analogous to comp.parallel.pvm, is proposed Keywords: MPI, Message Passing, Unmoderated, Parallel Computers, Clusters This is a formal Request for Discussion on the creation of a new newsgroup called "comp.parallel.mpi" (unmoderated). This announcement is cross-posted to newsgroups whose readers may have interest in the discussion about the new group; follow-up discussion will take place in "news.groups". This RFD has been posted to news.announce.newgroups, comp.parallel, and comp.parallel.pvm on April 4, 1994. Suggestions, comments or problems may also be emailed to me at . REQUEST FOR DISCUSSION (THIS IS NOT A VOTE) ------------------------------------------- NEWSGROUP comp.parallel.mpi (unmoderated) PURPOSE The newsgroup 'comp.parallel.mpi' would be a forum for discussing issues related to the use and interpretation of MPI (Message Passing Interface), an informal standard for writing message-passing programs. MPI was created by the MPI Forum, a group of University, Industry, and National Laboratory representatives including significant international representation. The MPI standard can be obtained from netlib by sending the message "send draft-final.ps from mpi" to netlib@ornl.gov. Alternatively, one can anonymous ftp to netlib2.cs.utk.edu. RATIONALE Discussion of MPI questions currently has no single home. Discussion has appeared in newsgroups such as comp.parallel comp.parallel.pvm A single group will centralize discussion and target the intended audience of such posts more accurately. It is also expected that there will be a wide audience for this newsgroup as a result of the participation by many vendors, national labs, and universities in the development of MPI. CONTENT Appropriate discussion in the group will include both general MPI issues, platform-specific questions, and discussion comparing MPI to other systems. Some examples: General MPI Issues Answers to novices' questions Clarification of the standard document Exchange of public-domain MPI source code Implementation of various applications etc. Platform-specific issues Implementations available Peculiarities and bugs found in various implementations etc. Comparison to other systems P4, Express, LINDA, PVM, etc NAME comp.parallel.mpi seems to be most appropriate. MPI is to be used on many different architectures and thus doesn't fit in any one comp.sys.* hierarchy. It is not a programming language, so comp.lang.* is out; NOT MODERATED As the new group will be for informal discussion and the quick exchange of ideas, no moderator will be necessary. NEWSGROUP CREATION The proposed discussion period will be from the date of original posting to May 20, 1994 (excess of 45 days). Please post all discussion in news.groups. If a consensus is reached by the end of the discussion period, a CFV (Call for Votes) will be posted at that time. The voting period will last for 21 days. From owner-mpi-comm@CS.UTK.EDU Thu May 5 08:26:50 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id IAA05304; Thu, 5 May 1994 08:26:50 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.8s-UTK) id IAA11223; Thu, 5 May 1994 08:25:29 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 5 May 1994 08:25:28 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from pobox.cscs.ch by CS.UTK.EDU with SMTP (cf v2.8s-UTK) id IAA11209; Thu, 5 May 1994 08:25:25 -0400 Received: from agno-fddi.cscs.ch by pobox.cscs.ch with SMTP inbound id <19071-0@pobox.cscs.ch>; Thu, 5 May 1994 14:25:19 +0200 Received: from laax.serd.cscs.ch by serd.cscs.ch (5.0/SMI-SVR4) id AA15093; Thu, 5 May 1994 14:25:12 --100 Received: by laax.serd.cscs.ch (5.0/SMI-SVR4) id AA08467; Thu, 5 May 1994 14:25:24 --100 From: fritsche@serd.cscs.ch (Josef Fritscher) Message-Id: <9405051225.AA08467@laax.serd.cscs.ch> Subject: BLACS on top of MPI To: mpi-comm@CS.UTK.EDU, dongarra@CS.UTK.EDU Date: Thu, 5 May 1994 14:25:22 +0200 (MET DST) Reply-To: fritsche@serd.cscs.ch X-Mailer: ELM [version 2.4 PL23] Content-Type: text Hello, I'd like to ask if anyone is working on a BLACS implementation using the MPI standard. Any pointers are highly appreciated. Regards, and thanks in advance, Josef -- Josef Fritscher Email: fritsche@cscs.ch Centro Svizzero di Calcolo Scientifico Voice: +41 (91) 50 8259 Via Cantonale Fax: +41 (91) 50 6711 6928 Manno, Switzerland ------------------------------------------------------------------------------- A smell of burning fills the startled air - The Electrician is no longer there! (Hilaire Belloc) From owner-mpi-comm@CS.UTK.EDU Thu May 5 09:02:50 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id JAA05401; Thu, 5 May 1994 09:02:49 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.8s-UTK) id JAA13792; Thu, 5 May 1994 09:02:41 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 5 May 1994 09:02:39 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from dasher.cs.utk.edu by CS.UTK.EDU with ESMTP (cf v2.8s-UTK) id JAA13784; Thu, 5 May 1994 09:02:38 -0400 From: Jack Dongarra Received: by dasher.cs.utk.edu (cf v2.9c-UTK) id JAA26494; Thu, 5 May 1994 09:01:41 -0400 Date: Thu, 5 May 1994 09:01:41 -0400 Message-Id: <199405051301.JAA26494@dasher.cs.utk.edu> To: fritsche@serd.cscs.ch, mpi-comm@CS.UTK.EDU Subject: Re: BLACS on top of MPI In-Reply-To: Mail from 'fritsche@serd.cscs.ch (Josef Fritscher)' dated: Thu, 5 May 1994 14:25:22 +0200 (MET DST) Josef, We do not yet have an implementation of the BLACS based on MPI. We are expecting to have this by the end of summer. Regards, Jack From owner-mpi-comm@CS.UTK.EDU Thu May 5 09:05:24 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id JAA05407; Thu, 5 May 1994 09:05:24 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.8s-UTK) id JAA13982; Thu, 5 May 1994 09:05:32 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 5 May 1994 09:05:31 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with ESMTP (cf v2.8s-UTK) id JAA13975; Thu, 5 May 1994 09:05:30 -0400 Received: (from walker@localhost) by rios2.EPM.ORNL.GOV (8.6.8/8.6.6) id JAA22159; Thu, 5 May 1994 09:05:25 -0400 Message-Id: <199405051305.JAA22159@rios2.EPM.ORNL.GOV> To: fritsche@serd.cscs.ch Cc: mpi-comm@CS.UTK.EDU Subject: Re: BLACS on top of MPI In-reply-to: (Your message of Thu, 05 May 94 14:25:22 O.) <9405051225.AA08467@laax.serd.cscs.ch> Date: Thu, 05 May 94 09:05:25 -0500 From: "David W. Walker" Josef, Implementing the BLACS with MPI is something we (at Oak Ridge National Lab and Univ. Tennessess, Knoxville) intend to do, and we have started an effeort to do this. Because of communicators, MPI will permeate all levels of MPI. Regards, David From owner-mpi-comm@CS.UTK.EDU Thu May 5 09:27:23 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id JAA05502; Thu, 5 May 1994 09:27:23 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.8s-UTK) id JAA16711; Thu, 5 May 1994 09:26:57 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 5 May 1994 09:26:56 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with ESMTP (cf v2.8s-UTK) id JAA16704; Thu, 5 May 1994 09:26:55 -0400 Received: (from walker@localhost) by rios2.EPM.ORNL.GOV (8.6.8/8.6.6) id JAA17344; Thu, 5 May 1994 09:26:47 -0400 Message-Id: <199405051326.JAA17344@rios2.EPM.ORNL.GOV> To: fritsche@serd.cscs.ch cc: mpi-comm@CS.UTK.EDU Subject: Re: BLACS on top of MPI Date: Thu, 05 May 94 09:26:46 -0500 From: "David W. Walker" Josef, Here's what I meant to say in the last sentence of my previous message: Because of communicators, MPI will permeate all levels of ScaLAPACK. David From owner-mpi-comm@CS.UTK.EDU Tue May 17 10:08:49 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id KAA19950; Tue, 17 May 1994 10:08:48 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id KAA02245; Tue, 17 May 1994 10:06:12 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 17 May 1994 10:06:11 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from hub by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id KAA02232; Tue, 17 May 1994 10:05:58 -0400 Received: from tycho.co.uk (tycho.meiko.co.uk) by hub with SMTP id AA22654 (5.65c/IDA-1.4.4 for mpi-comm@cs.utk.edu); Tue, 17 May 1994 15:04:40 +0100 Received: by tycho.co.uk (5.0/SMI-SVR4) id AA18557; Tue, 17 May 1994 15:04:02 +0000 Date: Tue, 17 May 1994 15:04:02 +0000 From: jim@meiko.co.uk (James Cownie) Message-Id: <9405171404.AA18557@tycho.co.uk> To: mpi-comm@CS.UTK.EDU Subject: PACK/UNPACK People, What are the rules about pack and unpack ? IN particular my recollection was that we agreed that code which did this pack ( ... MPI_INT, ...) send (... MPI_PACKED ...) receive (... MPI_INT ...) is valid. So in other words the combination of packing and sending with MPI_PACKED is transparent to the receiver, who sees the same thing as if the data had simply been sent with the type signature used in the pack. Page 85 (20 et seq) of the standard talks about this, but is less than clear to me. (Though it clearly states that the converse case of send, receive packed, unpack is valid...) -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West 130C Baker Avenue Ext. Bristol BS12 4SD Concord England MA 01742 Phone : +44 454 616171 +1 508 371 0088 FAX : +44 454 618188 +1 508 371 7516 E-Mail: jim@meiko.co.uk or jim@meiko.com WWW : http://www.meiko.com:8080/welcome.html From owner-mpi-comm@CS.UTK.EDU Wed May 18 05:32:55 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id FAA26456; Wed, 18 May 1994 05:32:55 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id FAA10029; Wed, 18 May 1994 05:32:00 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 18 May 1994 05:31:59 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from gmdzi.gmd.de by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id FAA10020; Wed, 18 May 1994 05:31:48 -0400 Received: from f1neuman.gmd.de (f1neuman) by gmdzi.gmd.de with SMTP id AA14079 (5.65c8/IDA-1.4.4 for ); Wed, 18 May 1994 11:31:38 +0200 Received: by f1neuman.gmd.de (AIX 3.2/UCB 5.64/4.03) id AA16990; Wed, 18 May 1994 11:31:09 +0200 Date: Wed, 18 May 1994 11:31:09 +0200 From: Rolf.Hempel@gmd.de (Rolf Hempel) Message-Id: <9405180931.AA16990@f1neuman.gmd.de> To: mpi-comm@CS.UTK.EDU Subject: PACK/UNPACK Cc: gmap10@f1neuman.gmd.de Dear Friends, I agree with Jim that the explanation on page 85 is not clear enough. As far as I remember, at the Knoxville meeting we decided that PACK generates the same "byte stream" as the corresponding direct SEND would do. So, Jim's code example should be valid. In most applications, however, I would expect to see a combination of SEND with RECEIVE/UNPACK, because that's what you have to do if you port existing PVM/PARMACS/... code to MPI. I guess that's the reason why this situation has been described explicitely on page 85. - Rolf From owner-mpi-comm@CS.UTK.EDU Wed May 18 10:57:57 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id KAA27764; Wed, 18 May 1994 10:57:57 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id KAA23332; Wed, 18 May 1994 10:56:55 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 18 May 1994 10:56:53 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id KAA23324; Wed, 18 May 1994 10:56:52 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA15788; Wed, 18 May 94 09:56:33 CDT Date: Wed, 18 May 94 09:56:33 CDT From: Tony Skjellum Message-Id: <9405181456.AA15788@Aurora.CS.MsState.Edu> To: Rolf.Hempel@gmd.de, mpi-comm@CS.UTK.EDU Subject: Re: PACK/UNPACK Cc: gmap10@f1neuman.gmd.de Rolf, Rolf, I think I disagree. I think we explicitly said that MPI_PACKED messages had to be received with MPI_PACKED type also. In fact, I remember that we voted to that effect. I recall that Marc proposed the strategy still currently in the book. The whole discussion centered around the desire to optimize protocols, and it was pointed out (probably by Jim) that one wants to make sure one is getting some real optimizability from a specific design choice. In many cases, no optimizability results, as we discussed at length. What is in the draft is usable, but I do think we departed from the actual committee decision here in some degree. Does anyone have the minutes? Please correct me if I am mistaken. The implications of the current strategy are, in my opinion, i) the receiver does not need to know if the sender used PACK/SEND or did a single SEND with a datatype... that sounds good, a good portability requirement from PVM, etc, etc. ii) The discussion of packing units implies that if two packs of type MPI_INT are done, then it is equivalent to a single longer pack, in a single packing unit. This requires the implementation to track things like alignment. Still doable. iii) The cryptic discussion of multiple packing units really means that one can take MPI_PACKED buffers b1, b2 and so on and further MPI_PACK them, even hierarchically. That feature sounds good, but the receiver definitely needs to know what the sender is doing in that case. I think we would be better off with regular derived datatyped at this point. The draft kindly does not require b1 concat b2 to be a packing unit itself. -Tony From owner-mpi-comm@CS.UTK.EDU Wed May 18 11:52:08 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id LAA28099; Wed, 18 May 1994 11:52:08 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id LAA00531; Wed, 18 May 1994 11:52:05 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 18 May 1994 11:52:05 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from hub by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id LAA00522; Wed, 18 May 1994 11:51:52 -0400 Received: from tycho.co.uk (tycho.meiko.co.uk) by hub with SMTP id AA26231 (5.65c/IDA-1.4.4 for mpi-comm@CS.UTK.EDU); Wed, 18 May 1994 16:46:56 +0100 Received: by tycho.co.uk (5.0/SMI-SVR4) id AA20151; Wed, 18 May 1994 16:46:16 +0000 Date: Wed, 18 May 1994 16:46:16 +0000 From: jim@meiko.co.uk (James Cownie) Message-Id: <9405181546.AA20151@tycho.co.uk> To: tony@Aurora.CS.MsState.Edu Cc: Rolf.Hempel@gmd.de, mpi-comm@CS.UTK.EDU In-Reply-To: <9405181456.AA15788@Aurora.CS.MsState.Edu> (message from Tony Skjellum on Wed, 18 May 94 09:56:33 CDT) Subject: Re: PACK/UNPACK > What is in the draft is usable, but I do think we departed from the actual > committee decision here in some degree. Does anyone have the minutes? > Please correct me if I am mistaken. My notes (which are a little sketchy) have Y N A Pack/unpack take a communicator 13:0:1 Can you mix regular send with packed rx 10:1:6 and converse Add new packed type 9:1:7 Enforce packed type for packed buffers 5:3:9 So my understanding of this is that :- 1) A packed buffer MUST be sent as MPI_PACKED 2) Anything which is going to be unpacked must be received MPI_PACKED 3) A combination of pack, send MPI_PACKED is indistinguishable at the receiver from a send with a datatype whose type signature is the concatenation of the ones used in the packing operation(s). 4) The discussion on packing units says (in effect) :- A set of pack operations forms a packing unit provided that the user does not adjust the position argument between calls. A single pack unit can be transferred as above. You cannot take more than one pack unit and send them all in one message and then treat that as a single pack unit at the far end. (So you can't receive this concatenation with a normal typed receive and expect to get sense). What I don't understand (yet) is how to transfer a concatenation of pack units. MPI_PACKED is definitely wrong, so I guess you use MPI_BYTE, and it is up to the MPI system to ensure that there is enough info in the packed message to be able to interpret it correctly when required. If you like a pack unit is a record. You can send a single record and receive it as a normal message with matching type signature, or you can do a normal send and receive that as a single record. But if you send multiple records in a single message, then you need to separate the records to unpack each one independently. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West 130C Baker Avenue Ext. Bristol BS12 4SD Concord England MA 01742 Phone : +44 454 616171 +1 508 371 0088 FAX : +44 454 618188 +1 508 371 7516 E-Mail: jim@meiko.co.uk or jim@meiko.com WWW : http://www.meiko.com:8080/welcome.html From owner-mpi-comm@CS.UTK.EDU Wed May 18 13:15:35 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id NAA28566; Wed, 18 May 1994 13:15:35 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id NAA08196; Wed, 18 May 1994 13:15:05 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 18 May 1994 13:15:02 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id NAA08178; Wed, 18 May 1994 13:14:58 -0400 From: Received: from WATSON by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 1585; Wed, 18 May 94 13:14:57 EDT Received: from YKTVMH by watson.vnet.ibm.com with "VAGENT.V1.0" id 8508; Wed, 18 May 1994 13:14:53 EDT Received: from snir.watson.ibm.com by yktvmh.watson.ibm.com (IBM VM SMTP V2R3) with TCP; Wed, 18 May 94 13:14:52 EDT Received: from localhost by snir.watson.ibm.com (AIX 3.2/UCB 5.64/930311) id AA25537; Wed, 18 May 1994 13:13:32 -0400 Message-Id: <9405181713.AA25537@snir.watson.ibm.com> To: jim@meiko.co.uk (James Cownie) Subject: Re: PACK/UNPACK Cc: mpi-comm@CS.UTK.EDU, tony@cs.msstate.edu, rolf.hempel@gmd.de, frankeh@watson.ibm.com In-Reply-To: (Your message of Wed, 18 May 1994 16:46:16 -0000.) Date: Wed, 18 May 94 13:14:52 EDT : -) :-) :-) *** (-: (-: (-: Date: Wed, 18 May 1994 13:13:17 -0400 From: Marc Snir I think I partially agree with you, except the issue you have about concatenations of packing units. This is easier to argue from the view-point of a hypothetical implementation. Assume that a normal message, created by a regular send, contains some header identifying the type of machine that sends the message, so that the the recipient can do the required conversions, followed by data. More generally, the message could contain some formatting information (e.g., type signature) Recall that a packing unit is created by a sequence of "related" PACK calls, starting with position=0, and each incrementing the value of position returned by the previous call. A packing unit is consumed by a sequence of similarly "related" UNPACK calls. Normal send: header is prepended to data and message is sent. Send with type MPI_PACKED: no header is prepended; the header was already created by the call to PACK, and occurs in (each of the) packing units, created by a sequence of related PACK calls. Normal receive: the header is consumed by the receive operation. Receive with type MPI_PACKED: the header is passed, as is, and stored in the receive buffer; it will be consumed by the call(s) to MPI_UNPACK. Each sequence of related calls to MPI_UNPACK that consumes one packing unit, will consume one header. This means that 1. A packing unit, created by a sequence of related PACK calls, and sent with type MPI_PACKED, can be received by a normal receive. 2. A packing unit sent with type MPI_PACKED can be received with type MPI_PACKED and be unpacked by a sequence of related UNPACK calls. 3. A message sent by a normal sent can be received with type MPI_PACKED and unpakced by a sequence of related UNPACK calls. 4. Of course, a message sent by a normal send can be received with a normal send. Now, how about a sequence of bytes which is the concatenation of several packing units? Such can be Sent with type MPI_PACKED (no header added). Received with type MPI_PACKED (sequence of bytes in message body is stored, as is, in receive buffer). Each packing unit can be separately unpacked by a sequence of related UNPACK calls. Conclusion: a) A message that was created by one or several sequences of related PACK calls is sent with type MPI_PACKED. b) A message that is going to be unpacked by one or several sequences of related UNPACK calls is received with type MPI_PACKED. c) A Message created by one sequence of related PACK calls, and sent with type MPI_PACKED is identical to a message sent with a regular send. I don't understand why you came to the conclusion it is wrong to send the concatenation of several packing units with type MPI_PACKED. From owner-mpi-comm@CS.UTK.EDU Tue May 31 18:09:13 1994 Received: from cs.cs.utk.edu by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id SAA12893; Tue, 31 May 1994 18:09:13 -0400 Received: from localhost by cs.cs.utk.edu with SMTP (cf v2.9s-UTK) id SAA09368; Tue, 31 May 1994 18:06:36 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 31 May 1994 18:06:28 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by cs.cs.utk.edu with SMTP (cf v2.9s-UTK) id SAA09361; Tue, 31 May 1994 18:06:26 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA06718; Tue, 31 May 94 17:05:03 CDT Date: Tue, 31 May 94 17:05:03 CDT From: Tony Skjellum Message-Id: <9405312205.AA06718@Aurora.CS.MsState.Edu> To: rdippold@qualcomm.com Subject: Call for Votes : comp.parallel.mpi Cc: mpi-comm@CS.UTK.EDU Dear Sir, we wish to enlist your help in taking the vote on comp.parallel.mpi. We are enclosing the RFD (which was originally posted April 4, 1994) at the end of this letter (the RFD can also be obtained through anonymous ftp at aurora.cs.msstate.edu in pub/mpi/comp.parallel.mpi.rfd). We posted the RFD to news.announce.newgroups news.groups comp.parallel.pvm comp.parallel and request that you post the CFV to these newsgroups in addition to standard posting locations. For full disclosure, we also request that you post the CFV to the following mailing lists: mpi-comm@cs.utk.edu dwf@lanl.gov (David Forslund's moderated distributed computing list) mpi-users@mcs.anl.gov More information about MPI can be obtained by reading the MPI FAQ which is available by anonymous ftp at aurora.cs.msstate.edu in pub/mpi/faq/mpi-faq.ascii. The FAQ is also available through the WWW at URL: http://www.cs.msstate.edu/dist_computing/mpi-faq.html Furthermore, the URL for the full online manual of the standard, and reference implementation is as follows: http://www.mcs.anl.gov/mpi/ We would like voting to start on June 10, and end on June 30. Thank you, Anthony Skjellum =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >Path: nntp.msstate.edu!gatech!swrinde!cs.utexas.edu!uunet!bounce-back >From: tony@aurora.cs.msstate.edu (Tony Skjellum) >Newsgroups: news.announce.newgroups,news.groups,comp.parallel,comp.parallel.pvm >Subject: RFD: comp.parallel.mpi >Followup-To: news.groups >Date: 4 Apr 1994 20:09:49 -0400 >Organization: Mississippi State University >Lines: 88 >Sender: tale@uunet.uu.net >Approved: tale@uunet.uu.net, bigrigg@cs.pitt.edu >Message-ID: >NNTP-Posting-Host: rodan.uu.net >Xref: nntp.msstate.edu news.announce.newgroups:4147 news.groups:77787 comp.parallel:7901 comp.parallel.pvm:1633 This is a formal Request for Discussion on the creation of a new newsgroup called "comp.parallel.mpi" (unmoderated). This announcement is cross-posted to newsgroups whose readers may have interest in the discussion about the new group; follow-up discussion will take place in "news.groups". This RFD has been posted to news.announce.newgroups, comp.parallel, and comp.parallel.pvm on April 4, 1994. Suggestions, comments or problems may also be emailed to me at . REQUEST FOR DISCUSSION (THIS IS NOT A VOTE) ------------------------------------------- NEWSGROUP comp.parallel.mpi (unmoderated) PURPOSE The newsgroup 'comp.parallel.mpi' would be a forum for discussing issues related to the use and interpretation of MPI (Message Passing Interface), an informal standard for writing message-passing programs. MPI was created by the MPI Forum, a group of University, Industry, and National Laboratory representatives including significant international representation. The MPI standard can be obtained from netlib by sending the message "send draft-final.ps from mpi" to netlib@ornl.gov. Alternatively, one can anonymous ftp to netlib2.cs.utk.edu. RATIONALE Discussion of MPI questions currently has no single home. Discussion has appeared in newsgroups such as comp.parallel comp.parallel.pvm A single group will centralize discussion and target the intended audience of such posts more accurately. It is also expected that there will be a wide audience for this newsgroup as a result of the participation by many vendors, national labs, and universities in the development of MPI. CONTENT Appropriate discussion in the group will include both general MPI issues, platform-specific questions, and discussion comparing MPI to other systems. Some examples: General MPI Issues Answers to novices' questions Clarification of the standard document Exchange of public-domain MPI source code Implementation of various applications etc. Platform-specific issues Implementations available Peculiarities and bugs found in various implementations etc. Comparison to other systems P4, Express, LINDA, PVM, etc NAME comp.parallel.mpi seems to be most appropriate. MPI is to be used on many different architectures and thus doesn't fit in any one comp.sys.* hierarchy. It is not a programming language, so comp.lang.* is out; NOT MODERATED As the new group will be for informal discussion and the quick exchange of ideas, no moderator will be necessary. NEWSGROUP CREATION Please post all discussion in news.groups. If a consensus is reached by the end of the discussion period, a CFV (Call for Votes) will be posted at that time. The voting period will last for 21 days. -- . . . . . . . . . "There is no lifeguard at the gene pool." - C. H. Baldwin - - - Anthony Skjellum, MSU/ERC, (601)325-8435; FAX: 325-8997; tony@cs.msstate.edu From owner-mpi-comm@CS.UTK.EDU Fri Jun 3 14:36:51 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id OAA10170; Fri, 3 Jun 1994 14:36:51 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id OAA11397; Fri, 3 Jun 1994 14:34:16 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 3 Jun 1994 14:34:15 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by cs.cs.utk.edu with SMTP (cf v2.9s-UTK) id OAA11390; Fri, 3 Jun 1994 14:34:13 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA08744; Fri, 3 Jun 94 13:32:15 CDT Date: Fri, 3 Jun 94 13:32:15 CDT From: Tony Skjellum Message-Id: <9406031832.AA08744@Aurora.CS.MsState.Edu> To: mpi-comm@CS.UTK.EDU Subject: Conference to include MPI emphasis Our second annual Scalable Libraries Conference will feature an MPI emphasis... -Tony Skjellum cf, *** PRELIMINARY ANNOUNCEMENT + CALL FOR PAPERS & POSTERS *** Scalable Parallel Libraries Conference II (SPLC94) including MPI Applications Day and Multicomputer Toolbox Developers' & Users' Meeting October 12-14, 1994 National Science Foundation Engineering Research Center for Computational Field Simulation, Mississippi State, Mississippi The Scalable Libraries Conference series is designed to . complement other, larger, general conferences in high performance computing, . bring together key researchers from national laboratories, industry, and academia to effect technology transfer of high performance computing software and effect an interchange of ideas, . include vendors as attendees, but not include promotional activities, . provide invited talks on topics involving application of scalable libraries in applications, developments in scalable libraries, and related issues (such as message-passing technology), . provide for contributed talks and posters to allow further interchange of ideas, . introduce potential users to the work done at our center on the scalable libraries, a specific example of technology we hope to transfer, . provide user of scalable libraries with an opportunity to impact research in the area (i.e., complain/suggest needs/requirements), . There will be no parallel sessions (except that posters will be displayed for at least one full day), . Support the promulgation of the MPI standard, and its follow-ons, including application development and user training, . A proceedings of the all papers and posters will be made; the paper deadline will be at the conference. [Publisher: IEEE Press] ---------------------------------------------------------------------------- Note: The Proceedings of the 1993 conference are available from IEEE Computer Society Press. ---------------------------------------------------------------------------- Paper: A proposal to present a paper should consist of a title plus a seven-hundred-fifty word abstract of the work. It is expected that papers be controversial, up-to-the-minute, and cutting-edge. Include the name(s), complete address(es), phone number(s), e-mail(s), of the author(s). Deadline: July 20, 1994 Notification of acceptance: August 1, 1994 It is STRONGLY preferred that proposal to present be sent by e-mail to splc94-conf@cs.msstate.edu. Hardcopy proposals can be sent to the address cited below under poster submissions. Papers marked "MPI Applications Day" will be considered for special sessions on Thursday, October 13, which will be devoted to MPI-based applications, and related talks and posters. ---------------------------------------------------------------------------- Posters: A proposal to present a poster should consist of a title plus a three-hundred word abstract of the work. It is expected that work-in-progress and completed work will be proposed for posters, so that up-to-date material will appear in the posters. Include the name(s), complete address(es), phone number(s), e-mail(s), of the author(s). Deadline: July 20, 1994 Notification of acceptance: August 1, 1994 It is STRONGLY preferred that proposal to present be sent by e-mail to splc94-conf@cs.msstate.edu. Hardcopy proposals can be sent to: Anthony Skjellum NSF Engineering Research Center Attn: SCL Conference PO Box 6176 Mississippi State, MS 39762 Posters marked "MPI Applications Day" will be considered for poster session on Thursday, October 13, which will be devoted to MPI-based applications, and related talks and posters. ---------------------------------------------------------------------------- Conference fee: $225 for all attendees. Attendance will be limited due to space requirements (approximately 100). Fee includes cost of proceedings (to be distributed after the conference), and some meals. Shuttle service also provided to/from airport during limited times. Fees payable by check to Mississippi State University. PREREGISTRATION IS REQUIRED. ON-SITE REGISTRATION CANNOT BE GUARANTEED. REGISTRANTS ARE STRONGLY ENCOURAGED TO STAY FOR THE ENTIRE THREE-DAY CONFERENCE, TO IMPROVE THE EXCHANGE OF INFORMATION. For further information, contact: Anthony Skjellum, tony@cs.msstate.edu, phone: 601-325-8435 fax: 601-325-8997 Organizing Committee: Anthony Skjellum, Mississippi State University, NSF Engineering Research Center for Computational Field Simulation + Computer Science Department, Donna S. Reese, Mississippi State University, NSF Engineering Research Center for Computational Field Simulation + Computer Science Department, Andrew Lumsdaine, University of Notre Dame, Dept. of Computer Science and Engineering Ewing Lusk, Argonne National Laboratory ---------------------------------------------------------------------------- Hotel Accomodations: Hotel accomodations are available at several hotels close to the NSF Engineering Research Center. The following are reasonable choices: The State House Hotel, (601) 323-2000 The Best Western Motel, (601) 324-5555 The Holiday Inn, (601) 323-6161 Airport: Starkville is served by the nearby Columbus, Mississippi Airport, (GTR), which provides connections through three larger cities: (American/American Eagle) through Nashville, Tennessee, (Delta/ASA) through Atlanta, Georgia, (Northwest Airlink) through Memphis, Tennessee GTR is approximately a 20-minute drive from Starkville, and several rental agencies are on-site. Correct directions (better than rental agencies) will be provided to registered attendees. Shuttle to hotels on October 11/12, and to airport on October 14. Also, shuttle to/from hotels will be provided each morning, and Wednesday, Thursday afternoons. ---------------------------------------------------------------------------- From owner-mpi-comm@CS.UTK.EDU Sun Jun 5 13:40:03 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id NAA22749; Sun, 5 Jun 1994 13:40:03 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id NAA02465; Sun, 5 Jun 1994 13:38:14 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sun, 5 Jun 1994 13:38:12 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by cs.cs.utk.edu with SMTP (cf v2.9s-UTK) id NAA02458; Sun, 5 Jun 1994 13:38:10 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA09718; Sun, 5 Jun 94 12:35:57 CDT Date: Sun, 5 Jun 94 12:35:57 CDT From: Tony Skjellum Message-Id: <9406051735.AA09718@Aurora.CS.MsState.Edu> To: dwf@lanl.gov, turcotte@bulldog.wes.army.mil Subject: fyi Cc: mpi-comm@CS.UTK.EDU Dear colleagues, I am doing a survey of MPI applications underway, or completed, or even seriously planned and just getting started, for a paper that I am writing, to be completed in the next few weeks. I will make this paper available by anonymous ftp when it is done (the goal is to be done by June 22). The following are the goals of the paper, and also the format for contributions... 1) List MPI-based applications, who's working on them, and experiences thus far. Specifically, - authors - short blurb of application area (4-10 sentences) - goals of application project (high speed, cheap cycles, new physics, etc) (2-4 sentences) - proper citation(s) to the project, and techniques it is based on, and/or ftp locations to get relevant articles 2) Determine anectodal information about process of converting from other systems (about 3-4 sentences each, max) - amount of effort - motivation for conversion - degree of satisfaction 3) Determine which features of MPI have proven most helpful in applications development and/or conversion, such as - communicators - inter-communicators - topologies - pack/unpack capabilities - datatypes (4-10 sentences) 4) Determine which parts of MPI are most understandable and most confusing for application developers (4-10 sentences) 5) Get early indications of performance results (such as on IBM SP/1, and Paragon) (as much space as reasonably needed to display, including any tables or graphs, in Latex or EPS format) 6) Get indications of which MPI implementation or implementations have been used in conjunction with the applications 7) Miscellenaous feedback from application programmers and users, if applicable. For instance, desire for other/different features. (4-10 sentences) If you are working on one or more applications, please respond in each of the seven categories listed above, for each application. Please include your full name, one or more citations (bibtex or ASCII) that can be used properly to describe your activity, and your institution, e-mail/phone information, in case I need more information. I would like to present a seven-part description of each representative application, following the above format. I will assume that it is OK for me to quote the material submitted to me in whole or in part, within this survey paper (editing for conciseness, as necessary), without any further grant of permission on your part or that of your institution, and I will reference all the contributors in the acknowledgements. Please be sure to secure any needed permissions before sending me the information, because the paper I am writing will be distributed without restrictions. I would appreciate e-mail responses, but my postal address is as follows: Anthony Skjellum NSF Engineering Research Center Mississippi State University PO Box 6176 Mississippi State, MS 39762 Thanks very much, Tony Skjellum From owner-mpi-comm@CS.UTK.EDU Thu Jun 9 18:46:11 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id SAA10761; Thu, 9 Jun 1994 18:46:10 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id SAA06563; Thu, 9 Jun 1994 18:43:18 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 9 Jun 1994 18:43:17 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from SSD.intel.com by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id SAA06556; Thu, 9 Jun 1994 18:43:15 -0400 Received: from tualatin.SSD.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA25050; Thu, 9 Jun 94 15:42:44 PDT Received: by tualatin.SSD.intel.com (4.1/SMI-4.1) id AA01955; Thu, 9 Jun 94 15:42:42 PDT Date: Thu, 9 Jun 94 15:42:42 PDT Message-Id: <9406092242.AA01955@tualatin.SSD.intel.com> From: Bob Knighten To: mpi-comm@CS.UTK.EDU Subject: MPI and SPEC HPSC Reply-To: knighten@ssd.intel.com (Bob Knighten) Some news on the MPI front. SPEC, the benchmark people, recently created a "High Performance Steering Committee" to develop benchmarks for the high performance computing arena. The latest meeting was June 7&8. Here is an interesting note from the minutes: Motion (940008): Intel proposed " The message passing version of a candidate benchmark code must use PVM or MPI libraries to be considered for the first release." HP seconded. For Against Abstain IBM Intel Sun NEC None DEC FAI HP KAI (UMn/UIL) Motion 940008 carried: 8-0-1 -- Bob -- Robert L. Knighten | knighten@ssd.intel.com Intel Supercomputer Systems Division | CO1-05 | 15201 N.W. Greenbrier Pkwy. | (503) 629-4315 Beaverton, Oregon 97006 | (503) 629-9147 [FAX] From owner-mpi-comm@CS.UTK.EDU Tue Jun 14 19:10:00 1994 Received: from CS.UTK.EDU by netlib2 with ESMTP (cf v2.8t-netlib) id TAA24603; Tue, 14 Jun 1994 19:09:59 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id TAA03758; Tue, 14 Jun 1994 19:07:37 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 14 Jun 1994 19:07:36 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id TAA03742; Tue, 14 Jun 1994 19:07:34 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA06125; Tue, 14 Jun 94 18:06:24 CDT Date: Tue, 14 Jun 94 18:06:24 CDT From: Tony Skjellum Message-Id: <9406142306.AA06125@Aurora.CS.MsState.Edu> To: rdippold@qualcomm.com Subject: Re: CFV Ron, No, I have not heard about this at all. That is of course distressing since an analogous group already exists. There were no negative comments when the RFD was put out, nor when the previous group (comp.parallel.pvm) was posed. I did not hear of any such delays when that group was formed. RFD time seems like it was the appropriate time to complain, not now. Of course, we do not care if the name is changed to comp.parallel.message-passing-interface, but, I object strenuously to this delay, otherwise, as we have left lots of time for people to respond to the RFD in the first place. -Tony From rdippold@qualcomm.com Tue Jun 14 17:53:49 1994 Received: from happy.qualcomm.com by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA06101; Tue, 14 Jun 94 17:53:48 CDT Received: (rdippold@localhost) by happy.qualcomm.com (8.6.9/QC-BSD-2.4) id PAA07877 for Tony Skjellum ; Tue, 14 Jun 1994 15:54:54 -0700 Date: Tue, 14 Jun 94 15:54:53 PDT From: Ron Dippold To: Tony Skjellum Subject: CFV Message-Id: Status: R has group-advice@uunet.uu.net contacted you yet about the CFV? They've held it up for discussion on the name (acronyms are generally considered evil), and I wanted to be sure they'd let you know about it. From owner-mpi-comm@CS.UTK.EDU Wed Jun 15 10:29:26 1994 Received: from CS.UTK.EDU by netlib2 with ESMTP (cf v2.8t-netlib) id KAA29023; Wed, 15 Jun 1994 10:29:26 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id KAA21552; Wed, 15 Jun 1994 10:27:28 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 15 Jun 1994 10:27:27 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id KAA21534; Wed, 15 Jun 1994 10:27:25 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA06340; Wed, 15 Jun 94 09:26:12 CDT Date: Wed, 15 Jun 94 09:26:12 CDT From: Tony Skjellum Message-Id: <9406151426.AA06340@Aurora.CS.MsState.Edu> To: mpi-comm@CS.UTK.EDU Subject: CFV delayed, but coming The call for votes on comp.parallel.mpi was delayed, but is coming today, according to net gods. - TOny From owner-mpi-comm@CS.UTK.EDU Wed Jun 15 16:50:33 1994 Received: from CS.UTK.EDU by netlib2 with ESMTP (cf v2.8t-netlib) id QAA02658; Wed, 15 Jun 1994 16:50:33 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id QAA29843; Wed, 15 Jun 1994 16:49:20 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 15 Jun 1994 16:49:19 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with ESMTP (cf v2.9s-UTK) id QAA29836; Wed, 15 Jun 1994 16:49:18 -0400 Received: from godzilla (godzilla.mcs.anl.gov [140.221.5.136]) by antares.mcs.anl.gov (8.6.4/8.6.4) with ESMTP id PAA14428 for ; Wed, 15 Jun 1994 15:49:15 -0500 From: William Gropp Date: Wed, 15 Jun 1994 15:49:13 -0500 Message-Id: <199406152049.PAA09577@godzilla> To: mpi-comm@CS.UTK.EDU Subject: Errors in May 5 version What is the process for correcting errors at this point? I sent several but I have received few comments (all have agreed). (1) The May 5 version has: int MPI_Get_count( MPI_Status status, MPI_Datatype datatype, int *count ) int MPI_Get_elements( MPI_Status status, MPI_Datatype datatype, int *count ) int MPI_Test_cancelled( MPI_Status status, int *flag ) These should all be int MPI_Get_count( MPI_Status *status, MPI_Datatype datatype, int *count ) int MPI_Get_elements( MPI_Status *status, MPI_Datatype datatype, int *count ) int MPI_Test_cancelled( MPI_Status *status, int *flag ) (2) The various MPI_Type_size, extent, etc. functions in the May 5 version assume that the difference between 2 pointers to the same storage area in C can be represented by an int (as opposed to, for example, a long or a long long). I believe this is in error; while MPI_Aint was used for both these lengths and the addresses, we could assume that MPI_Aint was large enough for both purposes; choosing the C type "int" for the extents and sizes can make it impossible to return a correct value for some situations. (3) In 3.2.2, the sentance fragment MPI_DOUBLE_COMPLEX for double precision complex in Fortran declared to be of type DOUBLE PRECISION; should be MPI_DOUBLE_COMPLEX for double precision complex in Fortran declared to be of type DOUBLE COMPLEX; ---------------------------------------------------------------------------- Question for the MPIF: What is the proper way to make these changes? I would claim that these are all "clearly in error" but particularly the first one, the bindings for MPI_Get_count etc. is correct but unintended. Bill From owner-mpi-comm@CS.UTK.EDU Wed Jun 15 18:01:35 1994 Received: from CS.UTK.EDU by netlib2 with ESMTP (cf v2.8t-netlib) id SAA03084; Wed, 15 Jun 1994 18:01:35 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id SAA06596; Wed, 15 Jun 1994 18:00:42 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 15 Jun 1994 18:00:40 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id SAA06549; Wed, 15 Jun 1994 18:00:36 -0400 From: Received: from WATSON by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 9403; Wed, 15 Jun 94 18:00:07 EDT Received: from YKTVMV by watson.vnet.ibm.com with "VAGENT.V1.0" id 5157; Wed, 15 Jun 1994 18:00:06 EDT Received: from snir.watson.ibm.com by yktvmv.watson.ibm.com (IBM VM SMTP V2R3) with TCP; Wed, 15 Jun 94 18:00:05 EDT Received: from localhost by snir.watson.ibm.com (AIX 3.2/UCB 5.64/930311) id AA29798; Wed, 15 Jun 1994 18:00:03 -0400 Message-Id: <9406152200.AA29798@snir.watson.ibm.com> To: William Gropp Cc: dongarra@CS.UTK.EDU, otto@cse.ogi.edu, mpi-comm@CS.UTK.EDU Subject: Re: Errors in May 5 version In-Reply-To: (Your message of Wed, 15 Jun 1994 15:49:13 CDT.) Date: Wed, 15 Jun 94 18:00:05 EDT : -) :-) :-) *** (-: (-: (-: Date: Wed, 15 Jun 1994 18:00:02 -0400 From: Marc Snir I suggest the following process: Major changes should be discussed now only if the text is inconsistent, and would require approval by the forum; syntactic corrections can be done by the chapter author(s). The chapter author forwards the correction to the erata keeper (Steve?). An MPI erata is maintained on netlib and updated at reasonable frequency (monthly?). No new full version of the MPI document before it gets published as a book. It would be useful, if possible, to put in the journal article a pointer to the erata. From owner-mpi-comm@CS.UTK.EDU Wed Jun 15 18:06:52 1994 Received: from CS.UTK.EDU by netlib2 with ESMTP (cf v2.8t-netlib) id SAA03102; Wed, 15 Jun 1994 18:06:52 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id SAA07070; Wed, 15 Jun 1994 18:06:00 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 15 Jun 1994 18:05:59 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from merckx.cse.ogi.edu by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id SAA07060; Wed, 15 Jun 1994 18:05:54 -0400 Received: by merckx.cse.ogi.edu (1.37.109.4/16.2) id AA16136; Wed, 15 Jun 94 15:05:51 -0700 Date: Wed, 15 Jun 94 15:05:51 -0700 From: Steve Otto Message-Id: <9406152205.AA16136@merckx.cse.ogi.edu> To: gropp@mcs.anl.gov, snir@watson.ibm.com Subject: Re: Errors in May 5 version Cc: dongarra@CS.UTK.EDU, mpi-comm@CS.UTK.EDU, otto@cse.ogi.edu I like Marc's suggestions, they seem sensible. He writes: It would be useful, if possible, to put in the journal article a pointer to the erata. I will ask the MIT Press person right now if this will be easy. --Steve Otto From owner-mpi-comm@CS.UTK.EDU Wed Jun 15 18:33:11 1994 Received: from CS.UTK.EDU by netlib2 with ESMTP (cf v2.8t-netlib) id SAA03159; Wed, 15 Jun 1994 18:33:11 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id SAA10079; Wed, 15 Jun 1994 18:32:27 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 15 Jun 1994 18:32:26 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with ESMTP (cf v2.9s-UTK) id SAA10061; Wed, 15 Jun 1994 18:32:10 -0400 Received: from godzilla (godzilla.mcs.anl.gov [140.221.5.136]) by antares.mcs.anl.gov (8.6.4/8.6.4) with ESMTP id RAA18334; Wed, 15 Jun 1994 17:32:06 -0500 From: William Gropp Date: Wed, 15 Jun 1994 17:32:04 -0500 Message-Id: <199406152232.RAA14965@godzilla> To: otto@cse.ogi.edu CC: snir@watson.ibm.com, dongarra@CS.UTK.EDU, mpi-comm@CS.UTK.EDU, otto@cse.ogi.edu In-reply-to: <9406152205.AA16136@merckx.cse.ogi.edu> (message from Steve Otto on Wed, 15 Jun 94 15:05:51 -0700) Subject: Re: Errors in May 5 version I agree with Marc's suggestions; I'd like to discuss them with respect to the 3 bugs I submitted. (1) Bindings. Presumably these are syntatic errors caused by the automatic process by which the bindings were constructed (Rusty could confirm this). By Marc's reasoning, this is a syntactic correction. (2) int replaced MPI_Aint. The text is incorrect, so a change is needed. Since a previous version was correct (by using MPI_Aint), I'm not sure that discussion is required, but perhaps that is the safest course. Marc, since this is your chapter, what do you prefer? (3) DOUBLE PRECISION instead of DOUBLE COMPLEX. A clear example of a simple error. Steve, I believe you are the chapter editor for terms and conventions. There is one other issue. The online, WWW version should be correct since many people are using it as a reference manual. There are three obvious possibilities: (a) The WWW version is up-to-date, with corrections (b) The WWW version is not up-to-date, but has pointers to errata (c) The WWW version is not up-to-date and reflects the last published version; it is necessarily incorrect. (I'll admit that the descriptions may seem a little loaded but I believe that they are precise and will reflect the way the WWW version is viewed, since while paper documentation is expected to be out-of-date, online documentation is not (even though it often is).) We can maintain any and all of these. Which should we do? My preference is for (a), possibly merged with (b). In other words, include the corrections along with some indication that this reflects a correction from the errata. Comments? Bill From owner-mpi-comm@CS.UTK.EDU Thu Jun 16 15:40:11 1994 Received: from CS.UTK.EDU by netlib2 with ESMTP (cf v2.8t-netlib) id PAA15555; Thu, 16 Jun 1994 15:40:10 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id PAA00787; Thu, 16 Jun 1994 15:37:50 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 16 Jun 1994 15:37:49 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from noc.usfca.edu by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id PAA00773; Thu, 16 Jun 1994 15:37:34 -0400 Received: by noc.usfca.edu (5.65/DEC-Ultrix/4.3) id AA01320; Thu, 16 Jun 1994 12:38:30 -0700 Received: by mobydick.usfca.edu (4.1/SMI-4.1) id AA07958; Thu, 16 Jun 94 12:32:25 PDT Date: Thu, 16 Jun 94 12:32:25 PDT From: peter@mobydick.usfca.edu (Peter Pacheco) Message-Id: <9406161932.AA07958@mobydick.usfca.edu> To: gropp@mcs.anl.gov, otto@cse.ogi.edu Subject: Re: Errors in May 5 version Cc: dongarra@CS.UTK.EDU, mpi-comm@CS.UTK.EDU, snir@watson.ibm.com Bill Gropp writes that we have three possibilities > (a) The WWW version is up-to-date, with corrections > (b) The WWW version is not up-to-date, but has pointers to errata > (c) The WWW version is not up-to-date and reflects the last published > version; it is necessarily incorrect. I agree with Bill that option (a) is preferred, although I think that differences from the published hardcopy should be marked. An obvious question in this connection: does anyone have the time to keep it up to date? Peter Pacheco From owner-mpi-comm@CS.UTK.EDU Wed Jun 22 08:17:47 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id IAA00458; Wed, 22 Jun 1994 08:17:47 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id IAA02574; Wed, 22 Jun 1994 08:15:02 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 22 Jun 1994 08:15:00 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from nlrgup.nlr.nl by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id IAA02565; Wed, 22 Jun 1994 08:14:57 -0400 Received: from uxmain.nlr.nl by nlrgup.nlr.nl with SMTP id AA00783 (5.67b+/IDA-1.5 for ); Wed, 22 Jun 1994 14:14:23 +0200 Disclaimer: "The National Aerospace Laboratory NLR DOES NOT ACCEPT ANY FINANCIAL COMMITMENT derived from this message." Received: by uxmain.nlr.nl (5.61/1.34) id AA04992; Wed, 22 Jun 94 14:14:21 +0200 From: Klaas Jan Wierenga Message-Id: <9406221214.AA04992@uxmain.nlr.nl> Subject: subscribe To: mpi-comm@CS.UTK.EDU Date: Wed, 22 Jun 1994 14:14:21 +0200 (DST) X-Mailer: ELM [version 2.4 PL22] Content-Type: text From owner-mpi-comm@CS.UTK.EDU Wed Jun 22 09:58:12 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id JAA01126; Wed, 22 Jun 1994 09:58:12 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id JAA12203; Wed, 22 Jun 1994 09:56:37 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 22 Jun 1994 09:56:36 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from gmdzi.gmd.de by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id JAA12192; Wed, 22 Jun 1994 09:56:21 -0400 Received: from f1neuman.gmd.de (f1neuman) by gmdzi.gmd.de with SMTP id AA28483 (5.65c8/IDA-1.4.4 for ); Wed, 22 Jun 1994 15:56:12 +0200 Received: by f1neuman.gmd.de (AIX 3.2/UCB 5.64/4.03) id AA17158; Wed, 22 Jun 1994 15:56:13 +0200 Date: Wed, 22 Jun 1994 15:56:13 +0200 From: Rolf.Hempel@gmd.de (Rolf Hempel) Message-Id: <9406221356.AA17158@f1neuman.gmd.de> To: mpi-comm@CS.UTK.EDU Subject: data representation Cc: gmap10@f1neuman.gmd.de, zimmermann@gmd.de Hi folks, during our work on an automatic PARMACS-->MPI translator the following question came up: how can an MPI-process find out whether another process runs on a processor with the same data representation or not? In the first case I can just send a PARMACS buffer as type MPI_BYTE without bothering about its contents, whereas in the second case I have to do something special for every segment of the message. (A message segment contains data of one type only.) For performance reasons I don't want to go the second way in the 95% of cases where the representations match. I hoped for finding an inquiry function in the Environmental Management section. The function MPI_GET PROCESSOR_NAME does not help, because it gives different strings for different processors, not for different types of processors. Any brilliant idea? The only obvious way seems to be to send a test message and look if the received message matches the original. However, this method is not very precise, and complicated to use. - Rolf From owner-mpi-comm@CS.UTK.EDU Wed Jun 22 12:36:00 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id MAA03210; Wed, 22 Jun 1994 12:36:00 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id MAA00332; Wed, 22 Jun 1994 12:34:47 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 22 Jun 1994 12:34:46 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id MAA00311; Wed, 22 Jun 1994 12:34:42 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA07893; Wed, 22 Jun 94 11:33:48 CDT Date: Wed, 22 Jun 94 11:33:48 CDT From: Tony Skjellum Message-Id: <9406221633.AA07893@Aurora.CS.MsState.Edu> To: mpi-comm@CS.UTK.EDU, Rolf.Hempel@gmd.de Subject: Re: data representation Cc: gmap10@f1neuman.gmd.de, zimmermann@gmd.de Rolf, MPI assumes that the information on process types will be managed with groups or with communicators. Specifically, a property of a communicator could be its conversion protocol among its members. So, your PARMACS translator should not need to address this issue. It must be addressed by implementations of MPI that you use with your applications thereafter. -Tony ----- Begin Included Message ----- From owner-mpi-comm@CS.UTK.EDU Wed Jun 22 08:59:55 1994 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 22 Jun 1994 09:56:36 EDT Date: Wed, 22 Jun 1994 15:56:13 +0200 From: Rolf.Hempel@gmd.de (Rolf Hempel) To: mpi-comm@CS.UTK.EDU Subject: data representation Cc: gmap10@f1neuman.gmd.de, zimmermann@gmd.de Content-Length: 1018 Hi folks, during our work on an automatic PARMACS-->MPI translator the following question came up: how can an MPI-process find out whether another process runs on a processor with the same data representation or not? In the first case I can just send a PARMACS buffer as type MPI_BYTE without bothering about its contents, whereas in the second case I have to do something special for every segment of the message. (A message segment contains data of one type only.) For performance reasons I don't want to go the second way in the 95% of cases where the representations match. I hoped for finding an inquiry function in the Environmental Management section. The function MPI_GET PROCESSOR_NAME does not help, because it gives different strings for different processors, not for different types of processors. Any brilliant idea? The only obvious way seems to be to send a test message and look if the received message matches the original. However, this method is not very precise, and complicated to use. - Rolf ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Wed Jun 22 14:50:21 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id OAA03902; Wed, 22 Jun 1994 14:50:21 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id OAA10909; Wed, 22 Jun 1994 14:48:35 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 22 Jun 1994 14:48:26 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from merckx.cse.ogi.edu by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id OAA10873; Wed, 22 Jun 1994 14:48:23 -0400 Received: by merckx.cse.ogi.edu (1.37.109.4/16.2) id AA23040; Wed, 22 Jun 94 11:48:20 -0700 Date: Wed, 22 Jun 94 11:48:20 -0700 From: Steve Otto Message-Id: <9406221848.AA23040@merckx.cse.ogi.edu> To: mpi-comm@CS.UTK.EDU Subject: Errata for MPI Spec I am implementing an errata system as per Marc's suggestions; they seem reasonable. I will assemble an errata. --Steve >: -) :-) :-) *** (-: (-: (-: >Date: Wed, 15 Jun 1994 18:00:02 -0400 >From: Marc Snir > >I suggest the following process: > > >Major changes should be discussed now only if the text is >inconsistent, and would require approval by the forum; syntactic >corrections can be done by the chapter author(s). The chapter >author forwards the correction to the erata keeper (Steve?). An MPI >erata is maintained on netlib and updated at reasonable frequency >(monthly?). No new full version of the MPI document before it gets >published as a book. It would be useful, if possible, to put in the >journal article a pointer to the erata. From owner-mpi-comm@CS.UTK.EDU Thu Jun 23 08:50:19 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id IAA00417; Thu, 23 Jun 1994 08:50:18 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id IAA04994; Thu, 23 Jun 1994 08:47:12 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 23 Jun 1994 08:47:10 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from gmdzi.gmd.de by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id IAA04985; Thu, 23 Jun 1994 08:47:05 -0400 Received: from f1neuman.gmd.de (f1neuman) by gmdzi.gmd.de with SMTP id AA06183 (5.65c8/IDA-1.4.4 for ); Thu, 23 Jun 1994 08:36:27 +0200 Received: by f1neuman.gmd.de (AIX 3.2/UCB 5.64/4.03) id AA17478; Thu, 23 Jun 1994 08:36:29 +0200 Date: Thu, 23 Jun 1994 08:36:29 +0200 From: Rolf.Hempel@gmd.de (Rolf Hempel) Message-Id: <9406230636.AA17478@f1neuman.gmd.de> To: mpi-comm@CS.UTK.EDU Subject: Again: data representation Cc: gmap10@f1neuman.gmd.de, zimmermann@gmd.de I really don't want to bother you all with my problem of data representation, but Tony's reply doesn't solve it. Therefore, I try to explain the issue again. Of course I know that MPI takes care of changing the data representation between processors, if necessary. For this to work, however, MPI needs a detailed specification of the the message contents (e.g. 4 integers, followed by 10 doubles). In case the representations match, it is save to send the message as a string of bytes with type MPI_BYTES (on a machine with 4 bytes integers the above example would result in 96 bytes). In a PARMACS application, the user creates the buffer himself. Additionally, he specifies the contents of the buffer, as given in the above example. If I translate this into MPI, I can save a lot of extra work if I know that the message is going to a processor with the same data representation. I just send the message as MPI_BYTES with the given length. If the representations don't match, I either have to repack the buffer segments using the pack/unpack routines, specifying for each segment the corresponding MPI basic data type, or I have to build a derived datatype for the whole message and then send the original buffer with this type. I expect that both ways create a substantial overhead, which should be avoided if the representations match. In the first case, the message transmission requires two additional copies of the whole message. I don't know how much time is spent in creating the derived datatype for the second solution, but I hesitate to create a derived datatype for every single message. To summarize, it's not for reasons of functionality that I need to know whether the data representations match, it's just for performance. I hope that I could convince you that in terms of performance this knowledge really does make a difference. So, does anybody have an idea how I can get this piece of information? - Rolf From owner-mpi-comm@CS.UTK.EDU Thu Jun 23 18:20:18 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id SAA04963; Thu, 23 Jun 1994 18:20:17 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id SAA18080; Thu, 23 Jun 1994 18:19:24 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 23 Jun 1994 18:19:23 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from merckx.cse.ogi.edu by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id SAA18073; Thu, 23 Jun 1994 18:19:21 -0400 Received: by merckx.cse.ogi.edu (1.37.109.4/16.2) id AA21683; Thu, 23 Jun 94 15:19:16 -0700 Date: Thu, 23 Jun 94 15:19:16 -0700 From: Steve Otto Message-Id: <9406232219.AA21683@merckx.cse.ogi.edu> To: mpi-comm@CS.UTK.EDU Subject: Re: Again: data representation I'll have a go; Rolf raises some good issues. >I really don't want to bother you all with my problem of data >representation, but Tony's reply doesn't solve it. Therefore, I try >to explain the issue again. >Of course I know that MPI takes care of changing the data representation >between processors, if necessary. For this to work, however, MPI needs >a detailed specification of the the message contents (e.g. 4 integers, yes, you should specify the type signature >followed by 10 doubles). In case the representations match, it is save >to send the message as a string of bytes with type MPI_BYTES (on a >machine with 4 bytes integers the above example would result in 96 >bytes). yes, it is safe to do this, but you are circumventing the recommended procedure. If the machine architectures are "matching" then the MPI implementation should be smart enough to know that representation conversion is not necessary and it won't do it. But, you don't specify BYTE as the type! and, there are a few other issues... >In a PARMACS application, the user creates the buffer himself. >Additionally, he specifies the contents of the buffer, as given in the >above example. If I translate this into MPI, I can save a lot of extra >work if I know that the message is going to a processor with the same >data representation. I just send the message as MPI_BYTES with the given >length. If the representations don't match, I either have to repack the >buffer segments using the pack/unpack routines, specifying for each >segment the corresponding MPI basic data type, or I have to build a OK, you want to do a direct translation from PARMACS, which has packing/unpacking. And yes, if you use MPI's pack/unpack facilities, it is true that rep. conversion may "needlessly" occur since at pack time MPI does not necessarily know whether the architectures are matching or not. But note that, if one is on a homogeneous system, MPI could easily avoid rep conversion, even for pack and unpack. This could also be done on a group-by-group basis, since one specifies a communicator with these calls, so if the process group is homogeneous, we could again make this optimization. >derived datatype for the whole message and then send the original >buffer with this type. I expect that both ways create a substantial >overhead, which should be avoided if the representations match. In the >first case, the message transmission requires two additional copies of >the whole message. I don't know how much time is spent in creating >the derived datatype for the second solution, but I hesitate to create >a derived datatype for every single message. Yes, creating a derived datatype does entail some overhead, but note that it need not make additional copies of the data (that is, it need not assemble the data into a contiguous buffer of memory). So now, what about the overhead of constructing the derived datatype description itself (internal to MPI)? OK, well, all you need to do is to use the datatype for multiple sends and recvs, and then you will have amortized the cost. This should make the majority of applications happy. >To summarize, it's not for reasons of functionality that I need to >know whether the data representations match, it's just for performance. >I hope that I could convince you that in terms of performance this >knowledge really does make a difference. yes, but the way in which one gets performance in MPI may be different than in PARMACS --- in MPI, the recommended procedure is to create derived datatypes one time, then use them for lots of communications. Another optimization with a similar spirit is to use "persistent communication requests" (that is, a channel-like construct) >So, does anybody have an idea how I can get this piece of information? > >- Rolf --Steve Otto From owner-mpi-comm@CS.UTK.EDU Fri Jun 24 23:25:21 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id XAA12654; Fri, 24 Jun 1994 23:25:20 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id XAA15647; Fri, 24 Jun 1994 23:23:13 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 24 Jun 1994 23:23:10 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with ESMTP (cf v2.9s-UTK) id XAA15598; Fri, 24 Jun 1994 23:23:07 -0400 Received: from mcs.anl.gov (donner.mcs.anl.gov [140.221.5.134]) by antares.mcs.anl.gov (8.6.4/8.6.4) with ESMTP id WAA16071 for ; Fri, 24 Jun 1994 22:23:04 -0500 Message-Id: <199406250323.WAA16071@antares.mcs.anl.gov> To: mpi-comm@CS.UTK.EDU Subject: vote on mpi newsgroup Date: Fri, 24 Jun 1994 22:23:03 -0500 From: Rusty Lusk Tony Skjellum has officially proposed a newsgroup dedicated to discussion of all aspects of MPI (See the CONTENT section below.) The last call for votes has appeared. Establishment of newsgroups such as this one is far from automatic (We need at least a 100-vote margin.) and so if you have not done so already, please send I vote YES on comp.parallel.mpi to voting@qualcomm.com. Rusty Lusk ______________________________________________________________________ In article , rdippold@qualcomm.com (Ron "Asbestos" Dippold) writes: LAST CALL FOR VOTES (of 2) unmoderated group comp.parallel.mpi Newsgroups line: comp.parallel.mpi Message Passing Interface (MPI). Votes must be received by 23:59:59 UTC, 6 July 1994. After this CFV appears on news.announce.newgroups it will be sent to the mailing lists , , and (David Forslund's moderated distributed computing list). This vote is being conducted by a neutral third party. For voting questions only contact rdippold@qualcomm.com. For questions about the proposed group contact Tony Skjellum . CHARTER The newsgroup 'comp.parallel.mpi' would be a forum for discussing issues related to the use and interpretation of MPI (Message Passing Interface), an informal standard for writing message-passing programs. MPI was created by the MPI Forum, a group of University, Industry, and National Laboratory representatives including significant international representation. The MPI standard can be obtained from netlib by sending the message "send draft-final.ps from mpi" to netlib@ornl.gov. Alternatively, one can anonymous ftp to netlib2.cs.utk.edu. More information about MPI can be obtained by reading the MPI FAQ which is available by anonymous ftp at aurora.cs.msstate.edu in pub/mpi/faq/mpi-faq.ascii. The FAQ is also available through the WWW at URL: http://www.cs.msstate.edu/dist_computing/mpi-faq.html Furthermore, the URL for the full online manual of the standard, and reference implementation is as follows: http://www.mcs.anl.gov/mpi/ RATIONALE Discussion of MPI questions currently has no single home. Discussion has appeared in newsgroups such as comp.parallel comp.parallel.pvm A single group will centralize discussion and target the intended audience of such posts more accurately. It is also expected that there will be a wide audience for this newsgroup as a result of the participation by many vendors, national labs, and universities in the development of MPI. CONTENT Appropriate discussion in the group will include both general MPI issues, platform-specific questions, and discussion comparing MPI to other systems. Some examples: General MPI Issues Answers to novices' questions Clarification of the standard document Exchange of public-domain MPI source code Implementation of various applications etc. Platform-specific issues Implementations available Peculiarities and bugs found in various implementations etc. Comparison to other systems P4, Express, LINDA, PVM, etc HOW TO VOTE Send MAIL to: voting@qualcomm.com Just Replying should work if you are not reading this on a mailing list. Your mail message should contain one of the following statements: I vote YES on comp.parallel.mpi I vote NO on comp.parallel.mpi You may also ABSTAIN in place of YES/NO - this will not affect the outcome. Anything else may be rejected by the automatic vote counting program. The votetaker will respond to your received ballots with a personal acknowledge- ment by mail - if you do not receive one within several days, try again. It's your responsibility to make sure your vote is registered correctly. One vote counted per person, no more than one per account. Addresses and votes of all voters will be published in the final voting results list. comp.parallel.mpi Bounce List - No need to revote ------------------------------------------------------------------------------ Bertrand.Le_Cun@inria.fr le-cun bertrand jean-marie godson@uranium.cambridge.scr.slb.com Anthony Godson hideaki@sol.crj.cray.com Hideaki Moriyama From owner-mpi-comm@CS.UTK.EDU Tue Jun 28 18:47:28 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id SAA17025; Tue, 28 Jun 1994 18:47:28 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id SAA02857; Tue, 28 Jun 1994 18:43:16 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 28 Jun 1994 18:43:15 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id SAA02850; Tue, 28 Jun 1994 18:43:14 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA26600; Tue, 28 Jun 94 17:41:21 CDT Date: Tue, 28 Jun 94 17:41:21 CDT From: Tony Skjellum Message-Id: <9406282241.AA26600@Aurora.CS.MsState.Edu> To: dwf@lanl.gov, mpi-comm@CS.UTK.EDU, mpi-users@mcs.anl.gov Subject: Availability of UNIFY 0.9.1 Availability of UNIFY Version 0.9.1 The UNIFY system, which patches PVM 3.2, 3.2.6, or 3.3, provides a subset of MPI within the PVM environment, without sacrificing the PVM calls already available. This system currently works with the network versions of PVM, and has been tested with Sun and SGI platforms. It is easily installed. The motivations for UNIFY are as follows: * Permit an easy learning curve for MPI to existing PVM users * Provide a dual-API mechanism for mixed PVM and MPI applications * Aid with the migration of codes from PVM syntax to MPI syntax * Provide a testbed for further MPI development UNIFY is based on completely independent MPI code. Version 0.9.1 provides a subset. Subsequent version will provide the entire MPI specification, and make use of the Argonne/Mississippi State implementation. Currently, UNIFY has the following limits: * SPMD model only * Static process creation * C Interface only Who will use UNIFY? * Anyone who wants to learn MPI without reading the entire standards document * Anyone who wants to migrate applications to MPI * Anyone who wants to learn parallel programming, and is interested in comparing strategies for message-passing syntax and semantics We will aggressively update Unify as users send their requests, bug reports, and suggestions. The following mailing addresses may be used: unify@erc.msstate.edu (suggestions, questions) unify-bugs@erc.msstate.edu (problem reports) The software is available free, for all uses, from the following ftp site: site: ftp.erc.msstate.edu directory: unify/ Read the file "README" to get started. UNIFY comes with a tutorial manual that explains the MPI calls that it supports, explains installation, and provides a number of figures to help with intuition about message-passing. Sincerely, Paula Vaughan Anthony Skjellum Donna Reese NSF Engineering Research Center for Computational Field Simulation Mississippi State University From owner-mpi-comm@CS.UTK.EDU Fri Jul 1 16:49:13 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id QAA20719; Fri, 1 Jul 1994 16:49:13 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id QAA20484; Fri, 1 Jul 1994 16:46:34 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 1 Jul 1994 16:46:32 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from mailhost.lanl.gov by CS.UTK.EDU with ESMTP (cf v2.9s-UTK) id QAA20475; Fri, 1 Jul 1994 16:46:29 -0400 Received: from wrangler.lanl.gov by mailhost.lanl.gov (8.6.8.1/1.2) id OAA15713; Fri, 1 Jul 1994 14:46:25 -0600 Received: from twinkie.lanl.gov by wrangler.lanl.gov (4.1/SMI-4.1) id AA03493; Fri, 1 Jul 94 14:46:25 MDT Date: Fri, 1 Jul 94 14:46:25 MDT From: rbarrett@wrangler.lanl.gov (Richard Barrett) Message-Id: <9407012046.AA03493@wrangler.lanl.gov> To: mpi-comm@CS.UTK.EDU subscribe From owner-mpi-comm@CS.UTK.EDU Sun Jul 10 16:32:33 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id QAA04874; Sun, 10 Jul 1994 16:32:33 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id QAA28057; Sun, 10 Jul 1994 16:29:30 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sun, 10 Jul 1994 16:29:30 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id QAA28050; Sun, 10 Jul 1994 16:29:28 -0400 Message-Id: <199407102029.QAA28050@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 0171; Sun, 10 Jul 94 16:29:24 EDT Date: Sun, 10 Jul 94 15:50:48 EDT From: "Marc Snir ((914) 945-3204)" To: mpi-comm@CS.UTK.EDU Subject: enquiry for machine type Reply-To: SNIR@watson.ibm.com The requirement for an Environmental Enquiry that allows to find whether two nodes use the same data representation is reasonable, irrespective of the specific use that Rolf has in mind, for Parmacs emulation. More generally, users may want enquiry functions to find the type of node on which a process runs. This would help in situations where the user is willing to run on any available machine on a network, but may want to use different code on different machine types. It is not reasonable to assume that the MPI forum will develop its own nomenclature for data representations (size of each basic language data type, big vs little endian, floating point format, character codes, etc.). First because other standards try to handle this issue and, second, because a complete characterization of the data representation used by a process (which depends on the underlying machine architecture and on the compiler) is likely to be quite lengthy. We agreed that vendors can attach implementation specific predefined attributes that are associated with MPI_COMM_WORLD. I would suggest the following: Associate a predefined attribute key MPI_PROC_TYPE with MPI_COMM_WORLD. The value of this attribute encodes the "type" of the executing process, i.e. the type of data representation used by this process. The meaning of the values of this attribute is implementation dependent, except that if two processes return the same value, then they use the same data representation (i.e., data can be moved untyped between these processes). Implementers of MPI for homogeneous SPMD systems just need to always return the same value (0?) with this attribute. Implementers of MPI on heterogeneous systems may need some work. If this suggestion sounds reasonable, then the various MPI implementers may accept it now as "common practice", and we can, at a latter time, add this to the standard. From owner-mpi-comm@CS.UTK.EDU Sun Jul 10 21:07:36 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id VAA05399; Sun, 10 Jul 1994 21:07:36 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id VAA16304; Sun, 10 Jul 1994 21:06:26 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sun, 10 Jul 1994 21:06:25 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id VAA16297; Sun, 10 Jul 1994 21:06:23 -0400 Message-Id: <199407110106.VAA16297@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 1147; Sun, 10 Jul 94 21:06:21 EDT Date: Sun, 10 Jul 94 20:28:42 EDT From: "Marc Snir ((914) 945-3204)" To: mpi-comm@CS.UTK.EDU cc: frankeh, gropp at mcs.anl.gov, otto at cse.ogi.edu Subject: Correction of Bill Gropp Reply-To: SNIR@watson.ibm.com Difference between 2 pointers to the same storage area should be of type MPI_Aint, rather than int. This is the assumption we go by in the definition of MPI_Type_hvector and MPI_Tpe_hindexed. If we want to be consistent, then we should use the same type MPI_Aint in the functions that return such difference; thus, MPI_Type_extent(..., MPI_Aint* extent) MPI_Type_size(..., MPI_Aint* size) MPI_Type_lb(..., MPI_Aint* displacement) MPI_Type_ub(..., MPI_Aint *displacement) Unless somebody objects, I would take this to be a bug in the current text, which will be corrected as outlined above. A similar issue: Should the arguments POSITION and OUTCOUNT in MPI_PACK be of type int (as they are now) or of type MPI_Aint? I.e., do we want to allow for a buffer with more than 2 Gbytes? Finally, a syntactic correction: MPI_PACK(..., OUTCOUNT,...) should be MPI_PACK(..., OUTSIZE,...), similar to MPI_UNPACK(..., INSIZE,...). OUTCOUNT is not a count of elements inserted in the buffer, but the total buffer size. From owner-mpi-comm@CS.UTK.EDU Mon Jul 11 02:03:37 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id CAA07333; Mon, 11 Jul 1994 02:03:36 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id CAA10093; Mon, 11 Jul 1994 02:02:48 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 11 Jul 1994 02:02:47 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from gmdzi.gmd.de by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id CAA10084; Mon, 11 Jul 1994 02:02:44 -0400 Received: from f1neuman.gmd.de (f1neuman) by gmdzi.gmd.de with SMTP id AA04263 (5.65c8/IDA-1.4.4 for ); Mon, 11 Jul 1994 08:02:40 +0200 Received: by f1neuman.gmd.de (AIX 3.2/UCB 5.64/4.03) id AA17493; Mon, 11 Jul 1994 08:02:45 +0200 Date: Mon, 11 Jul 1994 08:02:45 +0200 From: Rolf.Hempel@gmd.de (Rolf Hempel) Message-Id: <9407110602.AA17493@f1neuman.gmd.de> To: mpi-comm@CS.UTK.EDU Subject: Re: enquiry for machine type Cc: falk.zimmermann@gmd.de, gmap10@f1neuman.gmd.de Marc's suggestion to add a predefined attribute MPI_PROC_TYPE to MPI_COMM_WORLD, which in two processes is set to the same value if and only if the data representations match, would solve my problem. Therefore, I would very much appreciate if this would find its way into the standard at some later time. If the implementers agree to put it into their implementations earlier, so much the better. - Rolf From owner-mpi-comm@CS.UTK.EDU Mon Jul 11 16:05:17 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id QAA12380; Mon, 11 Jul 1994 16:05:17 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id QAA25069; Mon, 11 Jul 1994 16:03:46 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 11 Jul 1994 16:03:40 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with ESMTP (cf v2.9s-UTK) id QAA25061; Mon, 11 Jul 1994 16:03:37 -0400 Received: from mcs.anl.gov (donner.mcs.anl.gov [140.221.5.134]) by antares.mcs.anl.gov (8.6.4/8.6.4) with ESMTP id PAA22748; Mon, 11 Jul 1994 15:03:34 -0500 Message-Id: <199407112003.PAA22748@antares.mcs.anl.gov> To: mpi-comm@CS.UTK.EDU Subject: Correction on status argument to 3 routines cc: otto@cse.ogi.edu, frankeh@watson.ibm.com, gropp@antares.mcs.anl.gov Date: Mon, 11 Jul 1994 15:03:31 -0500 From: Rusty Lusk To go along with Marc's recent posting, I would like to take the C binding error on three MPI routines as another "official" bug in the current text. The input status argument on MPI_Get_count, MPI_Get_elements, and MPI_Test_cancel should be passed as a pointer, not as a struct. This error crept in when we decided to make the MPI_Status object a non-opaque type. There was not much response when I posted this before, but what there was supported treating this as an error that could be fixed without collective action. Specifically: ___________________________________________________________________________ Erratum: The C bindings for MPI_Get_count, MPI_Get_elements, and MPI_Test_cancelled should be as follows: int MPI_Get_count(MPI_Status *status, MPI_Datatype datatype, int *count) int MPI_Get_elements(MPI_Status *status, MPI_Datatype datatype, int *count) int MPI_Test_cancelled(MPI_Status *status, int *flag) instead of: int MPI_Get_count(MPI_Status status, MPI_Datatype datatype, int *count) int MPI_Get_elements(MPI_Status status, MPI_Datatype datatype, int *count) int MPI_Test_cancelled(MPI_Status status, int *flag) ___________________________________________________________________________ We have made this change, along with Marc's on MPI_Aint, in the upcoming release of our portable implementation, and in the "Using MPI" book. Rusty and Bill and Tony From owner-mpi-comm@CS.UTK.EDU Mon Jul 11 16:31:44 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id QAA12574; Mon, 11 Jul 1994 16:31:44 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id QAA27662; Mon, 11 Jul 1994 16:30:05 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 11 Jul 1994 16:30:02 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from gaudi.nas.nasa.gov by CS.UTK.EDU with ESMTP (cf v2.9s-UTK) id QAA27624; Mon, 11 Jul 1994 16:29:56 -0400 Received: from localhost (fineberg@localhost) by gaudi.nas.nasa.gov (8.6.8.1/NAS.5.b) with SMTP id NAA04498; Mon, 11 Jul 1994 13:29:49 -0700 Message-Id: <199407112029.NAA04498@gaudi.nas.nasa.gov> X-Authentication-Warning: gaudi.nas.nasa.gov: Host localhost didn't use HELO protocol To: mpi-comm@CS.UTK.EDU cc: fineberg@nas.nasa.gov Subject: Re: Correction on status argument to 3 routines In-reply-to: Your message of "Mon, 11 Jul 1994 15:03:31 CDT." <199407112003.PAA22748@antares.mcs.anl.gov> Date: Mon, 11 Jul 1994 13:29:49 -0700 From: "Samuel A. Fineberg" In message <199407112003.PAA22748@antares.mcs.anl.gov>Rusty Lusk writes > >To go along with Marc's recent posting, I would like to take the C binding >error on three MPI routines as another "official" bug in the current text. ... > >We have made this change, along with Marc's on MPI_Aint, in the upcoming >release of our portable implementation, and in the "Using MPI" book. > I have not been following the bug discussions very closely, but I would like to see a summary of the bugs in the current standard document. Has someone made such a list, and if so, could you please mail it out and put it on the net somewhere. Thanks, Sam ----- Samuel A. Fineberg Address: Numerical Aerodynamic Simulation Computer Sciences Corporation NASA Ames Research Center M/S 258-6 Applied Technology Division Moffett Field, CA 94035-1000 e-mail: fineberg@nas.nasa.gov Phone: (415)604-4319(voice) (415)604-4377(FAX) See my home page at URL: "ftp://ftp.netcom.com/pub/fineberg/home-page.html" From owner-mpi-comm@CS.UTK.EDU Tue Jul 12 09:41:56 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id JAA21063; Tue, 12 Jul 1994 09:41:55 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id JAA19445; Tue, 12 Jul 1994 09:39:57 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 12 Jul 1994 09:39:50 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id JAA19438; Tue, 12 Jul 1994 09:39:48 -0400 Received: from WATSON by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 1629; Tue, 12 Jul 94 09:39:51 EDT Received: from YKTVMV by watson.vnet.ibm.com with "VAGENT.V1.0" id 5287; Tue, 12 Jul 1994 09:39:50 EDT Received: from bubba.watson.ibm.com by yktvmv.watson.ibm.com (IBM VM SMTP V2R3) with TCP; Tue, 12 Jul 94 09:39:50 EDT Received: from localhost by bubba.watson.ibm.com (AIX 3.2/UCB 5.64/930311) id AA24018; Tue, 12 Jul 1994 09:39:45 -0400 Message-Id: <9407121339.AA24018@bubba.watson.ibm.com> To: Rusty Lusk Cc: mpi-comm@CS.UTK.EDU, otto@cse.ogi.edu, gropp@antares.mcs.anl.gov Subject: Re: Correction on status argument to 3 routines In-Reply-To: Your message of Mon, 11 Jul 1994 15:03:31 CDT. <199407112003.PAA22748@antares.mcs.anl.gov> Date: Tue, 12 Jul 1994 09:39:45 -0400 From: "Hubertus Franke" Since you are willing to make a few changes let me put you on the trail of another one. I extract this out of the MPI-F include files. #define MPI_DUP_FN mpi_dup_fn /* note the draft provides MPI_NULL_FN, however this violates * the prototype differences in usage for copy and delete function * In agreement with the public domain libary developers we * distinguish these functions #define MPI_NULL_FN */ #define MPI_NULL_COPY_FN ... #define MPI_NULL_DELETE_FN ... I've discussed this with Nathan Doss and I believe this has been incorporated into the public domain version as well. -- Hubertus From owner-mpi-comm@CS.UTK.EDU Tue Jul 12 11:12:41 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id LAA21814; Tue, 12 Jul 1994 11:12:40 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id LAA28866; Tue, 12 Jul 1994 11:12:00 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 12 Jul 1994 11:11:59 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from Phoenix.ERC.MsState.Edu by CS.UTK.EDU with ESMTP (cf v2.9s-UTK) id LAA28858; Tue, 12 Jul 1994 11:11:57 -0400 Received: from Cheers.ERC.MsState.Edu by Phoenix.ERC.MsState.Edu using ESMTP (8.6.8.1/6.0s-FWP); id KAA23188; Tue, 12 Jul 1994 10:12:42 -0500 From: Tony Skjellum Received: by Cheers.ERC.MsState.Edu (8.6.8.1/6.0c-FWP); id KAA24529; Tue, 12 Jul 1994 10:10:54 -0500 Date: Tue, 12 Jul 1994 10:10:54 -0500 Message-Id: <199407121510.KAA24529@Cheers.ERC.MsState.Edu> To: mpi-comm@CS.UTK.EDU Subject: comp.parallel.mpi passes Cc: mpi-users@mcs.anl.gov Please see note below. It was 506:14, with one abstention. Please ask your USENET site administrators to begin carrying comp.parallel.mpi. - Regards, Tony Skjellum PS I have suppressed the voting list in what follows, for brevity, but you can read it on news.announce.newgroups. Path: nntp.msstate.edu!emory!swrinde!news.dell.com!tadpole.com!uunet!bounce-back ~From: rdippold@qualcomm.com (Ron "Asbestos" Dippold) ~Newsgroups: news.announce.newgroups,news.groups,comp.parallel.pvm,comp.parallel ~Subject: RESULT: comp.parallel.mpi passes 506:14 Supersedes: Followup-To: news.groups ~Date: 11 Jul 1994 23:49:54 -0400 Organization: Usenet Volunteer Votetakers ~Lines: 617 ~Sender: tale@uunet.uu.net Approved: tale@uunet.uu.net, parallel@hubcap.clemson.edu Message-ID: ~References: NNTP-Posting-Host: rodan.uu.net ~Xref: nntp.msstate.edu news.announce.newgroups:4584 news.groups:86017 comp.parallel.pvm:2184 comp.parallel:8578 RESULT unmoderated group comp.parallel.mpi passes 506:14 There were 506 YES votes and 14 NO votes, for a total of 520 valid votes. There was 1 abstain. For group passage, YES votes must be at least 2/3 of all valid (YES and NO) votes. There also must be at least 100 more YES votes than NO votes. There is a five day discussion period after these results are posted. If no serious allegations of voting irregularities are raised, the moderator of news.announce.newgroups will create the group shortly thereafter. Newsgroups line: comp.parallel.mpi Message Passing Interface (MPI). This vote is being conducted by a neutral third party. For voting questions only contact rdippold@qualcomm.com. For questions about the proposed group contact Tony Skjellum . CHARTER The newsgroup 'comp.parallel.mpi' would be a forum for discussing issues related to the use and interpretation of MPI (Message Passing Interface), an informal standard for writing message-passing programs. MPI was created by the MPI Forum, a group of University, Industry, and National Laboratory representatives including significant international representation. The MPI standard can be obtained from netlib by sending the message "send draft-final.ps from mpi" to netlib@ornl.gov. Alternatively, one can anonymous ftp to netlib2.cs.utk.edu. More information about MPI can be obtained by reading the MPI FAQ which is available by anonymous ftp at aurora.cs.msstate.edu in pub/mpi/faq/mpi-faq.ascii. The FAQ is also available through the WWW at URL: http://www.cs.msstate.edu/dist_computing/mpi-faq.html Furthermore, the URL for the full online manual of the standard, and reference implementation is as follows: http://www.mcs.anl.gov/mpi/ RATIONALE Discussion of MPI questions currently has no single home. Discussion has appeared in newsgroups such as comp.parallel comp.parallel.pvm A single group will centralize discussion and target the intended audience of such posts more accurately. It is also expected that there will be a wide audience for this newsgroup as a result of the participation by many vendors, national labs, and universities in the development of MPI. CONTENT Appropriate discussion in the group will include both general MPI issues, platform-specific questions, and discussion comparing MPI to other systems. Some examples: General MPI Issues Answers to novices' questions Clarification of the standard document Exchange of public-domain MPI source code Implementation of various applications etc. Platform-specific issues Implementations available Peculiarities and bugs found in various implementations etc. Comparison to other systems P4, Express, LINDA, PVM, etc comp.parallel.mpi Final Vote Ack From owner-mpi-comm@CS.UTK.EDU Wed Jul 13 13:41:51 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id NAA02587; Wed, 13 Jul 1994 13:41:51 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id NAA12963; Wed, 13 Jul 1994 13:39:08 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 13 Jul 1994 13:39:07 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from antares10.mcs.anl.gov by CS.UTK.EDU with ESMTP (cf v2.9s-UTK) id NAA12931; Wed, 13 Jul 1994 13:39:03 -0400 Received: from mcs.anl.gov (donner.mcs.anl.gov [140.221.5.134]) by antares10.mcs.anl.gov (8.6.4/8.6.4) with ESMTP id MAA05673; Wed, 13 Jul 1994 12:38:50 -0500 Message-Id: <199407131738.MAA05673@antares10.mcs.anl.gov> To: mpi-users@antares10.mcs.anl.gov, mpi-comm@CS.UTK.EDU cc: lusk@antares10.mcs.anl.gov Subject: New release of portable MPI implementation Date: Wed, 13 Jul 1994 12:38:46 -0500 From: Rusty Lusk A new version of the Argonne - Mississippi State portable MPI implementation is available by anonymous ftp from info.mcs.anl.gov, in the directory pub/mpi. Take the file mpich-Jul13.tar.Z. See the README file in the distribution for details of building, installing, and running the new version. A copy of the README is also in the ftp directory. New in the July 13, 1994 version: Tested on the Meiko CS2 (configure -device=ch_p4 -arch=meiko) Tested on the SGI Onyx Enhanced upshot that can read ParaGraph log files Better installation procedures (make install PREFIX=/usr/local/mpi). The examples subdirectory will contain a location-independent Makefile that can be copied and modified for user programs. Better support for sites without Fortran (configure .... -nof77) An early version of a portable command for launching mpi programs (in util/mpirun). It knows about workstation networks, the Meiko CS-2, the Intel Paragon, and the IBM SP1. MPI executables built with the ch_p4 device can be started on clusters by the next release of DQS, which is currently being tested. The "Errata" in the May 5th draft recently posted by Steve Otto have been incorporated: the input status argument to the C versions of MPI_Get_count, MPI_Get-elements, and MPI_Test_cancelled is now passed as a pointer, and several arguments specified as ints have changed to MPI_Aints. For details, see the Errata posted on netlib. Problem reports to mpi-bugs@mcs.anl.gov. Nathan Doss Bill Gropp Rusty Lusk Tony Skjellum From owner-mpi-comm@CS.UTK.EDU Wed Jul 13 14:19:00 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id OAA02903; Wed, 13 Jul 1994 14:18:59 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id OAA17579; Wed, 13 Jul 1994 14:17:58 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 13 Jul 1994 14:17:56 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id OAA17562; Wed, 13 Jul 1994 14:17:54 -0400 Received: from WATSON by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 3001; Wed, 13 Jul 94 14:17:56 EDT Received: from YKTVMV by watson.vnet.ibm.com with "VAGENT.V1.0" id 4789; Wed, 13 Jul 1994 14:17:55 EDT Received: from snir.watson.ibm.com by yktvmv.watson.ibm.com (IBM VM SMTP V2R3) with TCP; Wed, 13 Jul 94 14:17:53 EDT Received: from localhost by snir.watson.ibm.com (AIX 3.2/UCB 5.64/930311) id AA17666; Wed, 13 Jul 1994 14:17:47 -0400 Message-Id: <9407131817.AA17666@snir.watson.ibm.com> To: mpi-comm@CS.UTK.EDU Cc: simonc@epcc.ed.ac.uk, frankeh@watson.ibm.com Subject: Suggestions of S R Chapple on section 3.7.5. Reply-To: snir@watson.ibm.com Date: Wed, 13 Jul 1994 14:17:47 -0400 From: Marc Snir :-) :-) :-) *** (-: (-: (-: I got some time ago a note from S R Chapple that points at some problems in the definition of the behavior of MPI_{TEST|WAIT}{ |ANY|SOME|ALL} when supplied with a list of handles that are all null or inactive (there is no debate about the "normal" behavior of these functions.) Let's call this situation, for short, "empty list". Let's call the status that signals no message (tag = MPI_ANY_TAG, source = MPI_ANY_SOURCE, count = 0) the "null status". The changes Chapple suggests: 1. TEST and TESTANY should return flag = TRUE on an empty list (rather than current flag = FALSE). In addition, TEST will return a null status (currently, it does not set the status). Note that TESTANY will return index = MPI_UNDEFINED on an empty list. Thus, for TEST flag=TRUE and status = 'null' indicates that there is no pending communication to wait; flag=TRUE and status != 'null' indicates that a communication completed; and flag = FALSE indicates that the communication has not complete. For TESTANY, flag = FALSE inidcates that no communication completed; flag = TRUE and index = MPI_UNDEFINED will indicate that there is no communication to wait for; and flag = TRUE and index != MPI_UNDEFINED will indicate that a communication completed. Reasons for change: Need to distinguish TESTANY returning (with flag = FALSE, index = MPI_UNDEFINED) because none of the pending communications completed, and TESTANY returning (now with flag = TRUE, index = MPI_UNDEFINED) because there is no communication to complete. Similarly, need to distinguish TEST returning (with flag = FALSE) because the pending communication has not completed, and TEST returning (now with flag = TRUE, status = "null") because there is no pending communication. With the change, busy waiting on list of communications would go like while(1) { /* busy wait until flag=1 */ while (!{MPI_Testany(count, reqs, &idx, &flag, &status); flag}) ; if(idx == MPI_UNDEFINED) break; /* no more requests */ do_something(reqs, &idx, &status); /* else handle completed communication*/ } 2. WAITSOME and TESTSOME should return outcount = MPI_UNDEFINED on an empty list (rather than outcount = 0). Reason for change: Need to distinguish TESTSOME returning (with outcount = 0) because no communication has completed, and TESTSOME returning (with outcount = MPI_UNDEFINED) because there is no operation to complete. WAITSOME is modified in order to be consistent with TESTSOME. Thus, a loop that deals for all communications in a list could be written as while(1) { MPI_Waitsome(incnt, reqs, &outcnt, indxs, statuses); if (outcnt == MPI_UNDEFINED) break; for(i=0; i To: mpi-comm@CS.UTK.EDU Cc: simonc@epcc.ed.ac.uk, frankeh@watson.ibm.com Subject: Another proposal by Chapple Reply-To: snir@watson.ibm.com Date: Wed, 13 Jul 1994 14:32:48 -0400 From: Marc Snir :-) :-) :-) *** (-: (-: (-: Chapple proposes to add to MPI a "null status pointer". If a communication function that receives a status (array of statuses) argument is passed, instead, the value MPI_STATUS_NULL, then this function need not return a status for the completed communication(s). This allows to save the cost of setting status information when this information is not needed, at the expense of an additional test. This proposal needs to be refined somewhat (e.g., status and array_of_statuses are not of the same type). Since this is additional functionality, not a correction of the current definition, I would suggest to postpone discussion of this suggestion to a later date. From owner-mpi-comm@CS.UTK.EDU Fri Jul 15 11:44:55 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id LAA22152; Fri, 15 Jul 1994 11:44:54 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id LAA13345; Fri, 15 Jul 1994 11:42:43 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jul 1994 11:42:42 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from super by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id LAA13338; Fri, 15 Jul 1994 11:42:41 -0400 Received: from b125.super.org by super (4.1/SMI-4.1) id AA07496; Fri, 15 Jul 94 11:42:40 EDT Date: Fri, 15 Jul 94 11:42:40 EDT From: lederman@super.org (Steve Huss-Lederman) Message-Id: <9407151542.AA07496@super> Received: by b125.super.org (4.1/SMI-4.1) id AA04680; Fri, 15 Jul 94 11:42:40 EDT To: mpi-comm@CS.UTK.EDU Subject: clarification of reduce with complex datatypes Hello MPIers, The discussion of how to do reduce on complex values seems to have slowed down on mpi-core. I propose the following clarification to the MPI standard text (I call it a clarification instead of a change since the current text is vague). I don't think (hope :-) that this requires a vote. I believe it is consistent with the discussion. I am posting to mpi-comm to make sure everyone interested will see this. As always, I welcome comments. Steve ---------------------------------------------------------------------- Proposed insertion of text on p. 113 line 40 of April 21 version: MPI_MAX and MPI_MIN on complex datatype values use the modulus (absolute value) to determine the ordering. The result returned in recvbuf is the complex value for which the maximum or minimum occurred. \rationale{ Unlike other datatypes, the maximum/minimum of a complex value is of a different datatype (floating point) and value than the input. Since the maximum/minimum can easily be determined from the complex value but the inverse operation is not unique, it was decided to return the complex value. As a result, the user is not returned the actual maximum/minimum for complex values as is true for the other datatypes. } A further consideration with complex arguments is what value to return when the modulus is the same for two input arguments. For other datatypes, the output value would be the same if the maximum/minimum were equal. For this case with complex values, MPI_MAX and MPI_MIN are defined as: If $|a| = |b|$ then $\rm{Max}(a,b) = a$ and $\rm{Min}(a,b) = a$. Thus, the first entry (lowest rank) with the maximum/minimum modulus will be returned. This definition means that MPI_MAXLOC and MPI_MINLOC will return the same value as MPI_MAX and MPI_MIN when two input values have the same modulus. ---------------------------------------------------------------------- Change to p. 112 line 20: All predefined operations are also assumed to be commutative should become All predefined operations are also assumed to be commutative with the exception of max and min on complex datatypes. ---------------------------------------------------------------------- Insertion to p. 115 line 17: If MPI_MAXLOC or MPI_MINLOC is applied to a complex value then $u > v$ becomes $ |u| < |v|$ and similarly for $u = v$ and $u < v$ in the above definitions. ---------------------------------------------------------------------- From owner-mpi-comm@CS.UTK.EDU Fri Jul 15 12:14:56 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id MAA23214; Fri, 15 Jul 1994 12:14:56 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id MAA15847; Fri, 15 Jul 1994 12:14:05 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jul 1994 12:14:03 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from hub.meiko.co.uk by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id MAA15838; Fri, 15 Jul 1994 12:14:01 -0400 Received: from tycho.co.uk (tycho.meiko.co.uk) by hub.meiko.co.uk with SMTP id AA16918 (5.65c/IDA-1.4.4 for mpi-comm@CS.UTK.EDU); Fri, 15 Jul 1994 17:13:48 +0100 Received: by tycho.co.uk (5.0/SMI-SVR4) id AA05620; Fri, 15 Jul 1994 17:13:41 +0000 Date: Fri, 15 Jul 1994 17:13:41 +0000 From: jim@meiko.co.uk (James Cownie) Message-Id: <9407151613.AA05620@tycho.co.uk> To: lederman@super.org Cc: mpi-comm@CS.UTK.EDU In-Reply-To: <9407151542.AA07496@super> (lederman@super.org) Subject: Re: clarification of reduce with complex datatypes Steve, I think you are making an unnecessary distinction when you say " As a result, the user is not returned the actual maximum/minimum for complex values as is true for the other datatypes." If we just say that we always return the input element which has the maximum or minimum value, then complex is no different from any other type. The difference is just that the comparison is based on a function of the input value (in this case its modulus), rather than the input value itself. This doesn't actually change anything, because for all the other types the input element which has the max or min value IS identically the max or min value. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West 130C Baker Avenue Ext. Bristol BS12 4SD Concord England MA 01742 Phone : +44 454 616171 +1 508 371 0088 FAX : +44 454 618188 +1 508 371 7516 E-Mail: jim@meiko.co.uk or jim@meiko.com WWW : http://www.meiko.com:8080/welcome.html From owner-mpi-comm@CS.UTK.EDU Fri Jul 15 12:31:46 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id MAA23908; Fri, 15 Jul 1994 12:31:46 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id MAA17555; Fri, 15 Jul 1994 12:31:08 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jul 1994 12:31:07 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from epcc.ed.ac.uk by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id MAA17546; Fri, 15 Jul 1994 12:31:02 -0400 Date: Fri, 15 Jul 94 17:30:43 BST Message-Id: <16258.9407151630@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: clarification of reduce with complex datatypes To: lederman@super.org (Steve Huss-Lederman), mpi-comm@CS.UTK.EDU In-Reply-To: Steve Huss-Lederman's message of Fri, 15 Jul 94 11:42:40 EDT Reply-To: lyndon@epcc.ed.ac.uk So we are saying that we take the max/min for complex according to modulus thereof. Why dont we just not offer max/min of complex in MPI. In no other case do we offer max/min of function(data), just max/min of data. The user can easily store the modulus in a real (or double) array, and do max/min on that. 1c Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Fri Jul 15 14:33:25 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id OAA24932; Fri, 15 Jul 1994 14:33:25 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id OAA24997; Fri, 15 Jul 1994 14:31:56 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jul 1994 14:31:50 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id OAA24988; Fri, 15 Jul 1994 14:31:48 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA21592; Fri, 15 Jul 94 13:27:22 CDT Date: Fri, 15 Jul 94 13:27:22 CDT From: Tony Skjellum Message-Id: <9407151827.AA21592@Aurora.CS.MsState.Edu> To: lederman@super.org, mpi-comm@CS.UTK.EDU, lyndon@epcc.ed.ac.uk Subject: Re: clarification of reduce with complex datatypes I concur with Lyndon's suggestion! -Tony ----- Begin Included Message ----- From owner-mpi-comm@CS.UTK.EDU Fri Jul 15 11:29:00 1994 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jul 1994 12:31:07 EDT Date: Fri, 15 Jul 94 17:30:43 BST From: L J Clarke Subject: Re: clarification of reduce with complex datatypes To: lederman@super.org (Steve Huss-Lederman), mpi-comm@CS.UTK.EDU Reply-To: lyndon@epcc.ed.ac.uk Content-Length: 609 So we are saying that we take the max/min for complex according to modulus thereof. Why dont we just not offer max/min of complex in MPI. In no other case do we offer max/min of function(data), just max/min of data. The user can easily store the modulus in a real (or double) array, and do max/min on that. 1c Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Mon Jul 18 19:41:23 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id TAA17480; Mon, 18 Jul 1994 19:41:23 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id TAA28948; Mon, 18 Jul 1994 19:37:31 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 18 Jul 1994 19:37:30 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from merckx.cse.ogi.edu by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id TAA28941; Mon, 18 Jul 1994 19:37:24 -0400 Received: by merckx.cse.ogi.edu (1.37.109.4/16.2) id AA28606; Mon, 18 Jul 94 16:37:18 -0700 Date: Mon, 18 Jul 94 16:37:18 -0700 From: Steve Otto Message-Id: <9407182337.AA28606@merckx.cse.ogi.edu> To: mpi-comm@CS.UTK.EDU Subject: complex. sigh. In a blinding flash of clarity, Jim Cownie said: >This doesn't actually change anything, because for all the other types >the input element which has the max or min value IS identically the >max or min value. So it just isn't such a big deal, and no, we aren't changing the standard. We also hear: >So we are saying that we take the max/min for complex according to >modulus thereof. Why dont we just not offer max/min of complex in MPI. >In no other case do we offer max/min of function(data), just max/min of >data. The user can easily store the modulus in a real (or double) >array, and do max/min on that. Ah...no. Think like a Fortran person. COMPLEX is fundamental and a Fortran person wants the operation. What is this business about this one being so special because it uses a "function"? They *all* use a function, it's just that in some cases the function looks a little simpler than in other cases. No big deal. What we have decided upon is natural for type COMPLEX. And remember that there is no C datatype corresponding to type MPI_COMPLEX. --Steve Otto From owner-mpi-comm@CS.UTK.EDU Mon Jul 18 20:03:29 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id UAA17511; Mon, 18 Jul 1994 20:03:29 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id UAA00288; Mon, 18 Jul 1994 20:02:00 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 18 Jul 1994 20:01:58 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from almaden.ibm.com by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id UAA00280; Mon, 18 Jul 1994 20:01:56 -0400 Received: from ALMADEN by almaden.ibm.com (IBM VM SMTP V2R2) with BSMTP id 2758; Mon, 18 Jul 94 16:58:52 PDT Received: by ALMADEN (XAGENTA 3.0) id 6285; Mon, 18 Jul 1994 16:58:51 -0700 Received: by vincennes.almaden.ibm.com (AIX 3.2/UCB 5.64/4.03) id AA13965; Mon, 18 Jul 1994 17:01:37 -0700 Message-Id: <9407190001.AA13965@vincennes.almaden.ibm.com> To: mpi-comm@CS.UTK.EDU Date: Mon, 18 Jul 94 17:01:33 -0800 From: "Eric Leu" Please remove my name from your mailing list (leu@almaden.ibm.com) Thanks in advance -Eric Leu From owner-mpi-comm@CS.UTK.EDU Tue Jul 19 12:10:06 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id MAA24384; Tue, 19 Jul 1994 12:10:05 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id MAA22248; Tue, 19 Jul 1994 12:08:22 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 19 Jul 1994 12:08:16 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id MAA22239; Tue, 19 Jul 1994 12:08:14 -0400 Received: from WATSON by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 7033; Tue, 19 Jul 94 12:08:06 EDT Received: from YKTVMV by watson.vnet.ibm.com with "VAGENT.V1.0" id 9507; Tue, 19 Jul 1994 12:08:06 EDT Received: from snir.watson.ibm.com by yktvmv.watson.ibm.com (IBM VM SMTP V2R3) with TCP; Tue, 19 Jul 94 12:08:06 EDT Received: from localhost by snir.watson.ibm.com (AIX 3.2/UCB 5.64/930311) id AA28838; Tue, 19 Jul 1994 12:08:00 -0400 Message-Id: <9407191608.AA28838@snir.watson.ibm.com> To: mpi-comm@CS.UTK.EDU Subject: Changes in {WAIT | TEST}{ | ANY | SOME | ALL} with empty list Reply-To: snir@watson.ibm.com Date: Tue, 19 Jul 1994 12:07:59 -0400 From: Marc Snir :-) :-) :-) *** (-: (-: (-: I have received so far only one answer. I would approciate more feedback to my note, both on substance (whether changes are warranted) and on form (whether this is a correction that we can do in the errata or a substantive change that need some form of vote). From owner-mpi-comm@CS.UTK.EDU Tue Jul 19 16:49:45 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id QAA26023; Tue, 19 Jul 1994 16:49:45 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id QAA18563; Tue, 19 Jul 1994 16:48:20 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 19 Jul 1994 16:48:07 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from research.CS.ORST.EDU by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id QAA18531; Tue, 19 Jul 1994 16:48:02 -0400 Received: from sycamore.CS.ORST.EDU by research.CS.ORST.EDU (4.1/1.30) id AA27957; Tue, 19 Jul 94 13:47:49 PDT From: pancake@chert.CS.ORST.EDU (Cherri Pancake) Received: by sycamore.CS.ORST.EDU (4.1/CS-Client) id AA07409; Tue, 19 Jul 94 13:47:48 PDT Date: Tue, 19 Jul 94 13:47:48 PDT Message-Id: <9407192047.AA07409@sycamore.CS.ORST.EDU> To: mpi-comm@CS.UTK.EDU, otto@cse.ogi.edu Subject: Re: complex. sigh. Hear, hear. Steve Otto is right on the mark with: > Think like a Fortran person. COMPLEX is fundamental > and a Fortran person wants the operation. The suggestion that we not offer max/min of COMPLEX will - set a dangerous precedent for we-almost-apply-this-to-all-data types-but-not-quite - force programmers to roll their own, since they really need this Cherri Pancake From owner-mpi-comm@CS.UTK.EDU Tue Jul 19 18:10:39 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id SAA27350; Tue, 19 Jul 1994 18:10:39 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id SAA26157; Tue, 19 Jul 1994 18:09:45 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 19 Jul 1994 18:09:43 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id SAA26150; Tue, 19 Jul 1994 18:09:40 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA05815; Tue, 19 Jul 94 17:04:41 CDT Date: Tue, 19 Jul 94 17:04:41 CDT From: Tony Skjellum Message-Id: <9407192204.AA05815@Aurora.CS.MsState.Edu> To: all@erc, mpi-comm@CS.UTK.EDU, mpi-users@mcs.anl.gov Subject: comp.parallel.mpi This newsgroup is now up and running, as of today, at least where we are! -Tony Skjellum From owner-mpi-comm@CS.UTK.EDU Tue Jul 19 19:14:53 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id TAA28692; Tue, 19 Jul 1994 19:14:52 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id TAA29945; Tue, 19 Jul 1994 19:13:44 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 19 Jul 1994 19:13:42 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from research.CS.ORST.EDU by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id TAA29937; Tue, 19 Jul 1994 19:13:38 -0400 Received: from sycamore.CS.ORST.EDU by research.CS.ORST.EDU (4.1/1.30) id AA29236; Tue, 19 Jul 94 16:13:30 PDT From: pancake@chert.CS.ORST.EDU (Cherri Pancake) Received: by sycamore.CS.ORST.EDU (4.1/CS-Client) id AA07563; Tue, 19 Jul 94 16:13:29 PDT Date: Tue, 19 Jul 94 16:13:29 PDT Message-Id: <9407192313.AA07563@sycamore.CS.ORST.EDU> To: mpi-comm@CS.UTK.EDU Subject: Re: Suggestions of S R Chapple on section 3.7.5. I agree that the suggested change would improve the precision with which the TEST/etc. functions can be used. As per Marc's second note, I think the changes are substantive and must go to the larger forum for review. Cherri Pancake From owner-mpi-comm@CS.UTK.EDU Tue Jul 19 19:15:00 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id TAA28697; Tue, 19 Jul 1994 19:15:00 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id TAA29954; Tue, 19 Jul 1994 19:14:03 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 19 Jul 1994 19:14:01 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from gw1.fsl.noaa.gov by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id TAA29947; Tue, 19 Jul 1994 19:13:56 -0400 Received: by gw1.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA18555; Tue, 19 Jul 94 23:12:48 GMT Received: by nipmuc.fsl.noaa.gov (4.1/SMI-4.1) id AA00766; Tue, 19 Jul 94 17:13:28 MDT Message-Id: <9407192313.AA00766@nipmuc.fsl.noaa.gov> To: mpi-comm@CS.UTK.EDU Subject: Re: complex. sigh. Date: Tue, 19 Jul 1994 17:13:28 -0600 From: Leslie Hart (NOAA/FSL) I'm no expert on ANSI Fortran, buy my Sun compiler will not let me put a variable declared as complex in the MAX intrinsic. Are we proposing to support something that the native language would not define? program t1 complex a, b, c a = (1, 1) b = (2, -1) c = max (a, b) print *, a, b, c end "f.f", line 5: Error: bad argument type to intrinsic "max" Leslie From owner-mpi-comm@CS.UTK.EDU Tue Jul 19 20:09:55 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id UAA29168; Tue, 19 Jul 1994 20:09:55 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id UAA02679; Tue, 19 Jul 1994 20:08:31 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 19 Jul 1994 20:08:27 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from noc.usfca.edu by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id UAA02671; Tue, 19 Jul 1994 20:08:23 -0400 Received: by noc.usfca.edu (5.65/DEC-Ultrix/4.3) id AA10656; Tue, 19 Jul 1994 17:10:52 -0700 Received: by mobydick.usfca.edu (4.1/SMI-4.1) id AA24385; Tue, 19 Jul 94 17:01:17 PDT Date: Tue, 19 Jul 94 17:01:17 PDT From: peter@mobydick.usfca.edu (Peter Pacheco) Message-Id: <9407200001.AA24385@mobydick.usfca.edu> To: hart@nipmuc.fsl.noaa.gov, mpi-comm@CS.UTK.EDU Subject: Re: complex. sigh. It's not clear to me that if w and z are complex, then "max(z,w)" should necessarily return the argument with the largest modulus. I can easily imagine that someone would want the argument with the largest real part. This would generalize max applied to real values, since, for example, max(-2,1) is 1. (Of course, with max(z,w) = argument with larger modulus, max(-2,1) is -2.) Incidentally, Leslie's program won't compile on our RS 6000 or our nCUBE -- neither fortran compiler accepts complex arguments to max. So I vote that the action of reduce with complex arguments and the various max/min operations be undefined. Peter From owner-mpi-comm@CS.UTK.EDU Tue Jul 19 20:19:13 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id UAA29181; Tue, 19 Jul 1994 20:19:13 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id UAA03038; Tue, 19 Jul 1994 20:18:17 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 19 Jul 1994 20:18:11 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from SSD.intel.com by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id UAA03030; Tue, 19 Jul 1994 20:18:06 -0400 Received: from tualatin.SSD.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA11090; Tue, 19 Jul 94 17:17:28 PDT Received: by tualatin.SSD.intel.com (4.1/SMI-4.1) id AA05476; Tue, 19 Jul 94 17:17:27 PDT Date: Tue, 19 Jul 94 17:17:27 PDT Message-Id: <9407200017.AA05476@tualatin.SSD.intel.com> From: Bob Knighten To: hart@nipmuc.fsl.noaa.gov Cc: mpi-comm@CS.UTK.EDU Subject: Re: complex. sigh. In-Reply-To: <9407192313.AA00766@nipmuc.fsl.noaa.gov> References: <9407192313.AA00766@nipmuc.fsl.noaa.gov> Reply-To: knighten@ssd.intel.com (Bob Knighten) Neither Fortran 77 nor Fortran 90 suport COMPLEX arguments for the MAX or MIN intrinsic. I have no objection to the proposal, but I was puzzled by the claim that this is fundamental in Fortran and equally surprised by the description of the proposed rule for breaking "ties" as a "stable sort requirement". MAX and MIN do NOT have a natural meaning for the COMPLEX type. -- Bob -- Robert L. Knighten | knighten@ssd.intel.com Intel Supercomputer Systems Division | CO1-05 | 15201 N.W. Greenbrier Pkwy. | (503) 629-4315 Beaverton, Oregon 97006 | (503) 629-9147 [FAX] From owner-mpi-comm@CS.UTK.EDU Tue Jul 19 23:14:16 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id XAA00633; Tue, 19 Jul 1994 23:14:16 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id XAA18144; Tue, 19 Jul 1994 23:13:24 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 19 Jul 1994 23:13:23 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id XAA18137; Tue, 19 Jul 1994 23:13:22 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA06646; Tue, 19 Jul 94 22:08:22 CDT Date: Tue, 19 Jul 94 22:08:22 CDT From: Tony Skjellum Message-Id: <9407200308.AA06646@Aurora.CS.MsState.Edu> To: mpi-comm@CS.UTK.EDU Subject: Dissolve this mailing list? David Walker expressed interest at the last MPI meeting to dissolve this mailing list (and possibly others) when comp.parallel.mpi had formed. This has happened now, and is the logical place for most if not all of the MPI reflector discussions. Any other thoughts? Thanks, Tony From owner-mpi-comm@CS.UTK.EDU Wed Jul 20 04:41:20 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id EAA02165; Wed, 20 Jul 1994 04:41:20 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id EAA15611; Wed, 20 Jul 1994 04:40:21 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 20 Jul 1994 04:40:19 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from hub by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id EAA15579; Wed, 20 Jul 1994 04:40:09 -0400 Received: from tycho.co.uk (tycho.meiko.co.uk) by hub with SMTP id AA19097 (5.65c/IDA-1.4.4 for mpi-comm@CS.UTK.EDU); Wed, 20 Jul 1994 09:39:50 +0100 Received: by tycho.co.uk (5.0/SMI-SVR4) id AA02018; Wed, 20 Jul 1994 09:39:29 +0000 Date: Wed, 20 Jul 1994 09:39:29 +0000 From: jim@meiko.co.uk (James Cownie) Message-Id: <9407200839.AA02018@tycho.co.uk> To: knighten@ssd.intel.com Cc: hart@nipmuc.fsl.noaa.gov, mpi-comm@CS.UTK.EDU In-Reply-To: <9407200017.AA05476@tualatin.SSD.intel.com> (message from Bob Knighten on Tue, 19 Jul 94 17:17:27 PDT) Subject: Re: complex. sigh. > Neither Fortran 77 nor Fortran 90 suport COMPLEX arguments for the MAX or MIN > intrinsic. Nor for the MINVAL and MAXVAL intrinsics in F90 (which apply to arrays, rather than scalars). On these grounds there is no need to make this extension (which seems to be contentious). -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West 130C Baker Avenue Ext. Bristol BS12 4SD Concord England MA 01742 Phone : +44 454 616171 +1 508 371 0088 FAX : +44 454 618188 +1 508 371 7516 E-Mail: jim@meiko.co.uk or jim@meiko.com WWW : http://www.meiko.com:8080/welcome.html From owner-mpi-comm@CS.UTK.EDU Wed Jul 20 05:24:05 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id FAA02708; Wed, 20 Jul 1994 05:24:04 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id FAA22340; Wed, 20 Jul 1994 05:23:07 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 20 Jul 1994 05:23:06 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from epcc.ed.ac.uk by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id FAA22231; Wed, 20 Jul 1994 05:22:55 -0400 Date: Wed, 20 Jul 94 10:22:40 BST Message-Id: <23820.9407200922@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Dissolve this mailing list? To: Tony Skjellum , mpi-comm@CS.UTK.EDU In-Reply-To: Tony Skjellum's message of Tue, 19 Jul 94 22:08:22 CDT Reply-To: lyndon@epcc.ed.ac.uk > David Walker expressed interest at the last MPI meeting to dissolve this mailing > list (and possibly others) when comp.parallel.mpi had formed. This has > happened now, and is the logical place for most if not all of the MPI > reflector discussions. Any other thoughts? Thanks, Tony Just one thought that it take people a while to get used to changes. So the question is whether we can configure the mail address to behave differently: (a) dont send to list of people on mail list (b) post message to comp.parallel.mpi (c) advisory reply to message sender that should post to comp.parallel.mpi instead of send to mpi-comm@cs.utk.edu Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Wed Jul 20 05:45:57 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id FAA02768; Wed, 20 Jul 1994 05:45:57 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id FAA24468; Wed, 20 Jul 1994 05:45:12 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 20 Jul 1994 05:45:11 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from hub by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id FAA24453; Wed, 20 Jul 1994 05:45:07 -0400 Received: from tycho.co.uk (tycho.meiko.co.uk) by hub with SMTP id AA20476 (5.65c/IDA-1.4.4 for mpi-comm@CS.UTK.EDU); Wed, 20 Jul 1994 10:44:28 +0100 Received: by tycho.co.uk (5.0/SMI-SVR4) id AA02064; Wed, 20 Jul 1994 10:44:07 +0000 Date: Wed, 20 Jul 1994 10:44:07 +0000 From: jim@meiko.co.uk (James Cownie) Message-Id: <9407200944.AA02064@tycho.co.uk> To: lyndon@epcc.ed.ac.uk Cc: tony@Aurora.CS.MsState.Edu, mpi-comm@CS.UTK.EDU In-Reply-To: <23820.9407200922@subnode.epcc.ed.ac.uk> (message from L J Clarke on Wed, 20 Jul 94 10:22:40 BST) Subject: Re: Dissolve this mailing list? > David Walker expressed interest at the last MPI meeting to dissolve this mailing > list (and possibly others) when comp.parallel.mpi had formed. This has > happened now, and is the logical place for most if not all of the MPI > reflector discussions. Any other thoughts? Thanks, Tony The reach of e-mail is probably larger than news, so there may well be people who can subscribe to the mail, but do not have access to news groups. We should be careful before dis-enfranchising this community. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West 130C Baker Avenue Ext. Bristol BS12 4SD Concord England MA 01742 Phone : +44 454 616171 +1 508 371 0088 FAX : +44 454 618188 +1 508 371 7516 E-Mail: jim@meiko.co.uk or jim@meiko.com WWW : http://www.meiko.com:8080/welcome.html From owner-mpi-comm@CS.UTK.EDU Wed Jul 20 07:59:42 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id HAA03150; Wed, 20 Jul 1994 07:59:42 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id HAA05674; Wed, 20 Jul 1994 07:58:57 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 20 Jul 1994 07:58:51 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from convex.convex.com by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id HAA05666; Wed, 20 Jul 1994 07:58:45 -0400 Received: from mozart.convex.com by convex.convex.com (5.64/1.35) id AA17120; Wed, 20 Jul 94 06:52:08 -0500 Received: from localhost by mozart.convex.com (8.6.4/1.28) id GAA15271; Wed, 20 Jul 1994 06:59:13 -0500 From: joelw@mozart.convex.com (Joel Williamson) Message-Id: <199407201159.GAA15271@mozart.convex.com> Subject: Re: complex. sigh. To: jim@meiko.co.uk (James Cownie) Date: Wed, 20 Jul 94 6:59:12 CDT Cc: knighten@ssd.intel.com, hart@nipmuc.fsl.noaa.gov, mpi-comm@CS.UTK.EDU In-Reply-To: <9407200839.AA02018@tycho.co.uk>; from "James Cownie" at Jul 20, 94 9:39 am X-Mailer: ELM [version 2.3 PL11] James Cownie writes: > > > Neither Fortran 77 nor Fortran 90 suport COMPLEX arguments for the MAX or MIN > > intrinsic. > > Nor for the MINVAL and MAXVAL intrinsics in F90 (which apply to > arrays, rather than scalars). > > On these grounds there is no need to make this extension (which seems > to be contentious). There's an interesting article in (I think) this month's Discover magazine, discussing metrics that reduce to a single number. In the introduction it is pointed out that there is no way to order the points on a plane, and that any attempt to map the plane to a line where ordering makes sense is simply a matter of taste. Joel Williamson From owner-mpi-comm@CS.UTK.EDU Wed Jul 20 09:55:57 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id JAA03675; Wed, 20 Jul 1994 09:55:57 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id JAA16786; Wed, 20 Jul 1994 09:54:44 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 20 Jul 1994 09:54:43 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from super by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id JAA16778; Wed, 20 Jul 1994 09:54:38 -0400 Received: from b125.super.org by super (4.1/SMI-4.1) id AA17820; Wed, 20 Jul 94 09:54:33 EDT Date: Wed, 20 Jul 94 09:54:33 EDT From: lederman@super.org (Steve Huss-Lederman) Message-Id: <9407201354.AA17820@super> Received: by b125.super.org (4.1/SMI-4.1) id AA06484; Wed, 20 Jul 94 09:54:30 EDT To: mpi-comm@CS.UTK.EDU Subject: reduce on complex datatype Hello All, The issue of reduce on the complex datatype has gotten more involved and contentious then I originally expected. I see two basic points of view (I'm not sure how many support each): #1 Proposal: Allow reduce on the complex datatype and define it to sort by the modulus and return the max entry. Ties are broken by returning entry with lowest rank. This is basically the proposal I put out after the first round of comments. Arguments for: -This seems the most natural definition to most and is consistant with the other reduce functions. -MPI supports reduce on all datatypes instead of having an exception for complex. -It clarifies the text already in the standard MPI document -Some users want this and will use it. #2 Proposal: Drop the definition of reduce on complex datatypes Arguments for: -There is more than one possible definition of what reduce does on a complex type. Sort by modulus, real, etc.; return modulus, entry, etc. -This only applies to Fortran binding of MPI (since complex only exists there) and standard FORTRAN does not support MIN, MAX, MAXLOC, MINLOC on the FORTRAN complex datatype. -You can get the same functionality by creating a second floating point (real,double) array arry and making its entries the modulus of the complex array. Then simply take the MAXLOC,MINLOC of the second array and use the index returned to get the value from the first array. What to do: The correct action is not clear to me. p. 113 and 115 of the April 15 draft allows for reduce on the complex datatype (though it is not well defined). I viewed #1 as a clarification of the text (which most seem to support if you can reduce on complex values). #2 is removing functionality that exists in the current draft (I don't view this as a clarification; I think it is a change). I think both positions have merit and each is supported by some fraction of those participating in the discussion. I don't know how the resolution is done and what the rules really are. I don't recall clearly defining how such changes would occur. I think this applies to several other items that are being discussed here. I think the time has come to define how we vote on items such as this since there are no planned meetings. I don't think a single person should be making these decisions. I'll stick my neck out and throw out a proposal for voting. This is just to get the discussion started. It clearly needs refining and input from others. Proposal for votes to decide issues: -A vote should be called when a proposal has been made that changes the current draft and seems reasonable after a discussion period. A vote is also taken when a clarification is not clearly decided after a discussion period. -I think mpi-core is the logical place for all messages to be posted. I have no problem with them also being posted to comp.parallel.mpi but since we may not want the whole world involved in the votes, I think the minimum is a post to mpi-core. -A clear written proposal of the choices and what a vote would mean MUST be posted. This proposal MUST include the exact text that would go into the standard document and where it would be placed. -A discussion period on the vote begins. Since people are away at times, I think it should last a reasonable period of time. At least 2 weeks; maybe a month. -Any modification to the original proposal MUST be clearly written out in the same mannor as the original proposal. This essential begins a new discussion period and delays any possible vote. -Someone needs to call a formal vote and moderate proposals if there is contention. To be consistant with the the meetings, the primary author of the chapter (or someone the author requested to act for him/her) would do this. If this cannot happen then Steve Otto could do this as the master editor. -Once a formal vote is called, the voting rules are the same as at the meetings. Only one vote per institution. The 2 out of 3 meetings rule is a little strange here. How about you can only vote if you formally voted (or could have) in 2 of the last 5 meetings. This will hopefully include anyone who was active in the last half of the MPIF. Someone should create a e-mail drop box to collect votes. The voting period would be 2 weeks. Some adoption rules would apply as at the meeting: majority carries. -I don't think we need to vote on something twice at this point like we did at the meetings. Comments? If we agree on how to vote, I will post a proposed vote on reduce for complex values. Steve From owner-mpi-comm@CS.UTK.EDU Wed Jul 20 11:11:05 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id LAA04116; Wed, 20 Jul 1994 11:11:04 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id LAA24244; Wed, 20 Jul 1994 11:09:40 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 20 Jul 1994 11:09:33 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id LAA24235; Wed, 20 Jul 1994 11:09:30 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA08356; Wed, 20 Jul 94 10:03:48 CDT Date: Wed, 20 Jul 94 10:03:48 CDT From: Tony Skjellum Message-Id: <9407201503.AA08356@Aurora.CS.MsState.Edu> To: lyndon@epcc.ed.ac.uk, jim@meiko.co.uk Subject: Re: Dissolve this mailing list? Cc: tony@Aurora.CS.MsState.Edu, mpi-comm@CS.UTK.EDU I agree that we should be careful and not hasty :-) -Tony ----- Begin Included Message ----- From jim@meiko.co.uk Wed Jul 20 04:40:00 1994 Date: Wed, 20 Jul 1994 10:44:07 +0000 From: jim@meiko.co.uk (James Cownie) To: lyndon@epcc.ed.ac.uk Cc: tony@Aurora.CS.MsState.Edu, mpi-comm@CS.UTK.EDU Subject: Re: Dissolve this mailing list? Content-Length: 815 > David Walker expressed interest at the last MPI meeting to dissolve this mailing > list (and possibly others) when comp.parallel.mpi had formed. This has > happened now, and is the logical place for most if not all of the MPI > reflector discussions. Any other thoughts? Thanks, Tony The reach of e-mail is probably larger than news, so there may well be people who can subscribe to the mail, but do not have access to news groups. We should be careful before dis-enfranchising this community. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West 130C Baker Avenue Ext. Bristol BS12 4SD Concord England MA 01742 Phone : +44 454 616171 +1 508 371 0088 FAX : +44 454 618188 +1 508 371 7516 E-Mail: jim@meiko.co.uk or jim@meiko.com WWW : http://www.meiko.com:8080/welcome.html ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Wed Jul 20 14:54:37 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id OAA05924; Wed, 20 Jul 1994 14:54:36 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id OAA15026; Wed, 20 Jul 1994 14:53:19 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 20 Jul 1994 14:53:17 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id OAA15013; Wed, 20 Jul 1994 14:53:15 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA06070; Tue, 19 Jul 94 18:11:55 CDT Date: Tue, 19 Jul 94 18:11:55 CDT From: Tony Skjellum Message-Id: <9407192311.AA06070@Aurora.CS.MsState.Edu> To: mpi-comm@CS.UTK.EDU, otto@cse.ogi.edu, pancake@chert.CS.ORST.EDU Subject: Re: complex. sigh. ----- Begin Included Message ----- From owner-mpi-comm@CS.UTK.EDU Tue Jul 19 15:49:28 1994 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 19 Jul 1994 16:48:07 EDT From: pancake@chert.CS.ORST.EDU (Cherri Pancake) Date: Tue, 19 Jul 94 13:47:48 PDT To: mpi-comm@CS.UTK.EDU, otto@cse.ogi.edu Subject: Re: complex. sigh. Content-Length: 382 Hear, hear. Steve Otto is right on the mark with: > Think like a Fortran person. COMPLEX is fundamental > and a Fortran person wants the operation. The suggestion that we not offer max/min of COMPLEX will - set a dangerous precedent for we-almost-apply-this-to-all-data types-but-not-quite - force programmers to roll their own, since they really need this Cherri Pancake ----- End Included Message ----- Various expositors, Absolute value min and max make sense for complex, just by generalizing the idea of modulus to be the usual modulus for complex. If this is what Steve means, fine. If we mean to leave that out, I think we are making a mistake. However, regular complex min/max do not make sense. There is no mathematical operation for min(c1,c2) where c1 and c2 are complex. Why provide an operation for symmetry when there is no mathematical analog? This seems empty to me. min and max are a statement of the ordering of the real numbers or integers! Why should MPI define a meaning? (You can argue than minrealpart, mincomplexpart, minabsrealpart, minabscomplexpart, and max versions of these 4 are meaningful, however). I do not think that users will want operations that do not make sense mathematically regardless of symmetry that might provide!?!?!?! -Tony Skjellum From owner-mpi-comm@CS.UTK.EDU Wed Jul 20 15:28:30 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id PAA06541; Wed, 20 Jul 1994 15:28:29 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id PAA18479; Wed, 20 Jul 1994 15:27:39 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 20 Jul 1994 15:27:38 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with ESMTP (cf v2.9s-UTK) id PAA18472; Wed, 20 Jul 1994 15:27:37 -0400 Received: (from walker@localhost) by rios2.EPM.ORNL.GOV (8.6.8/8.6.6) id PAA21572; Wed, 20 Jul 1994 15:27:36 -0400 From: David Walker Message-Id: <199407201927.PAA21572@rios2.EPM.ORNL.GOV> To: mpi-comm@CS.UTK.EDU Subject: Re: complex. sigh Date: Wed, 20 Jul 94 15:27:36 -0500 TONY WROTE: > Absolute value min and max make sense for complex, just by generalizing > the idea of modulus to be the usual modulus for complex. If this is what > Steve means, fine. If we mean to leave that out, I think we are making > a mistake. > > However, regular complex min/max do not make sense. There is no > mathematical operation for min(c1,c2) where c1 and c2 are complex. Why > provide an operation for symmetry when there is no mathematical > analog? This seems empty to me. min and max are a statement of the > ordering of the real numbers or integers! Why should MPI define a > meaning? (You can argue than minrealpart, mincomplexpart, minabsrealpart, > minabscomplexpart, and max versions of these 4 are meaningful, however). > > I do not think that users will want operations that do not make sense > mathematically regardless of symmetry that might provide!?!?!?! > > - -Tony Skjellum So we should provide MINMODULUS, MINREALPART, MINCOMPLEXPART, MINABSREALPART, MINABSCOMPLEXPART and max variants as Tony suggests. These may only take a complex number, and exist only for the Fortran language binding. David From owner-mpi-comm@CS.UTK.EDU Wed Jul 27 13:17:21 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id NAA06085; Wed, 27 Jul 1994 13:17:21 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id NAA15735; Wed, 27 Jul 1994 13:12:17 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 27 Jul 1994 13:12:14 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id NAA15727; Wed, 27 Jul 1994 13:12:11 -0400 Received: from WATSON by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 0711; Wed, 27 Jul 94 13:12:02 EDT Received: from YKTVMV by watson.vnet.ibm.com with "VAGENT.V1.0" id 8897; Wed, 27 Jul 1994 13:12:02 EDT Received: from snir.watson.ibm.com by yktvmv.watson.ibm.com (IBM VM SMTP V2R3) with TCP; Wed, 27 Jul 94 13:12:01 EDT Received: from localhost by snir.watson.ibm.com (AIX 3.2/UCB 5.64/930311) id AA29400; Wed, 27 Jul 1994 13:11:57 -0400 Message-Id: <9407271711.AA29400@snir.watson.ibm.com> To: mpi-comm@CS.UTK.EDU Subject: min of complex (Tony's and David's messages) Reply-To: snir@watson.ibm.com Date: Wed, 27 Jul 1994 13:11:56 -0400 From: Marc Snir :-) :-) :-) *** (-: (-: (-: Let's not be carried away. MIN of complex can be either left out of MPI, or if in, be defined to be an MPI reduction operation, which means an associative binary operation that returns a value of the same type as the inputs. The only reasonable definition of MIN of complex is the one Steve and I proposed. Whether one should include it at all is a good question. Note that Fortran defines MIN, MAX, MINLOC and MAXLOC only for types INTEGER and REAL. Note also that a user can always provide it's own user-defined MIN, MAX, MINLOC and MAXLOC for COMPLEX. Both are arguments against inclusion. I, personally, vote for keeping it out. If we decide to put it in, then we should use the text that was sent be Steve, also need to put in the corrsponding versions of MINLOC and MAXLOC. From owner-mpi-comm@CS.UTK.EDU Wed Jul 27 14:11:37 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id OAA06334; Wed, 27 Jul 1994 14:11:37 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id OAA19048; Wed, 27 Jul 1994 14:09:17 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 27 Jul 1994 14:09:15 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id OAA19032; Wed, 27 Jul 1994 14:09:12 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA09559; Wed, 27 Jul 94 13:02:57 CDT Date: Wed, 27 Jul 94 13:02:57 CDT From: Tony Skjellum Message-Id: <9407271802.AA09559@Aurora.CS.MsState.Edu> To: mpi-comm@CS.UTK.EDU, snir@watson.ibm.com Subject: Re: min of complex (Tony's and David's messages) Marc, as you recall, I argued against including MIN/MAX of complex, because it is not a mathematical concept for Complex #'s. I actually did not advocate including MINREAL, MAXREAL, MINCOMPLEX, etc, but just pointed these out, to demonstrate that one could go quite far in a different direction (not necessarily good). User-defined ops can do all this, and they can also do whatever your definition of MIN/MAX was for complex, wlog. However, since user-defined operations have the same power, but not necessarily the same performance as built-in MPI operations, there must be a performance argument for a real situation levied before we consider this anymore. Following that, we should set up a vote on a real proposal (sorry for the pun), and resolve the matter (as we did as a formal, standing committee). -Tony From owner-mpi-comm@CS.UTK.EDU Wed Oct 5 13:57:47 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id NAA14463; Wed, 5 Oct 1994 13:57:47 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id NAA20821; Wed, 5 Oct 1994 13:52:10 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 5 Oct 1994 13:52:06 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from theory.tc.cornell.edu by CS.UTK.EDU with ESMTP (cf v2.9s-UTK) id NAA20807; Wed, 5 Oct 1994 13:52:02 -0400 Received: (from xshen@localhost) by theory.tc.cornell.edu (8.6.9/8.6.6) id NAA59714; Wed, 5 Oct 1994 13:51:53 -0400 Message-Id: <199410051751.NAA59714@theory.tc.cornell.edu> To: mpi-comm@CS.UTK.EDU Subject: MPI Forum discussion list Reply-to: xshen@TC.Cornell.EDU Date: Wed, 05 Oct 94 13:51:53 -0400 From: Shennon Shen Hi, I am a staff member at Cornell Theory Center. I am porting application code to MPI and testing some feature on our SP2.I am interested in being on this mailing list If it is possible Thank you! Xianneng Shen Strategic Application Consultant Cornell THeory Center Engineering & Theory Center Building Ithaca, NY 14853-3801 Tel:(607)254-8788 From owner-mpi-comm@CS.UTK.EDU Fri Oct 21 05:40:13 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id FAA06401; Fri, 21 Oct 1994 05:40:13 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id FAA01362; Fri, 21 Oct 1994 05:37:40 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 21 Oct 1994 05:37:39 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from lorraine.loria.fr by CS.UTK.EDU with ESMTP (cf v2.9s-UTK) id FAA01345; Fri, 21 Oct 1994 05:37:35 -0400 Received: from galois.loria.fr (galois.loria.fr [152.81.4.133]) by lorraine.loria.fr (8.6.9/8.6.9) with ESMTP id IAA25169; Fri, 21 Oct 1994 08:13:15 +0100 Received: from localhost (localhost.loria.fr [127.0.0.1]) by galois.loria.fr (8.6.4/8.6.4) with ESMTP id IAA06617; Fri, 21 Oct 1994 08:13:16 +0100 Message-Id: <199410210713.IAA06617@galois.loria.fr> To: mpi-comm@CS.UTK.EDU cc: dillon@loria.fr Subject: subscribe X-Mailer: exmh version 1.2 1/14/94 Date: Fri, 21 Oct 1994 08:13:14 +0100 From: Eric Dillon SUBSCRIBE dillon@loria.fr From owner-mpi-comm@CS.UTK.EDU Thu Nov 3 13:46:33 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id NAA15261; Thu, 3 Nov 1994 13:46:33 -0500 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id NAA16866; Thu, 3 Nov 1994 13:42:58 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 3 Nov 1994 13:42:57 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from dasher.cs.utk.edu by CS.UTK.EDU with ESMTP (cf v2.9s-UTK) id NAA16860; Thu, 3 Nov 1994 13:42:55 -0500 From: Jack Dongarra Received: by dasher.cs.utk.edu (cf v2.9c-UTK) id NAA02368; Thu, 3 Nov 1994 13:42:54 -0500 Date: Thu, 3 Nov 1994 13:42:54 -0500 Message-Id: <199411031842.NAA02368@dasher.cs.utk.edu> To: mpi-comm@CS.UTK.EDU Subject: MPI B-O-F meeting at SC94 It looks like the MPI BOF session at SC94 will take place on Wed. Nov. 16 at 5:30 pm, location to be determined. Below is a brief agenda for the meeting follows. MPI BOF at SC94 Introduction How We Got to this Point (Dongarra 5 minutes) Overview of MPI and What's Missing (Walker 10 minutes) Current Implementations (Lusk 15 minutes) Future MPI-2 (Skjellum 15 minutes) Discussion Future meetings The MPI session will be immediately followed by an MPI-I/O session scheduled by Nitzberg. This is an activity involving IBM, NASA Ames and LLNL. Marc Snir has more information. Send me any comments and/or additional topics for discussion. Jack From owner-mpi-comm@CS.UTK.EDU Fri Dec 9 09:20:12 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.8t-netlib) id JAA16085; Fri, 9 Dec 1994 09:20:12 -0500 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id JAA21967; Fri, 9 Dec 1994 09:16:55 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 9 Dec 1994 09:16:53 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from lorraine.loria.fr by CS.UTK.EDU with ESMTP (cf v2.9s-UTK) id JAA21952; Fri, 9 Dec 1994 09:16:47 -0500 Received: from galois.loria.fr (galois.loria.fr [152.81.4.133]) by lorraine.loria.fr (8.6.9/8.6.9) with ESMTP id PAA01751 for ; Fri, 9 Dec 1994 15:16:33 +0100 Received: from localhost (localhost.loria.fr [127.0.0.1]) by galois.loria.fr (8.6.4/8.6.4) with ESMTP id PAA15040 for ; Fri, 9 Dec 1994 15:16:44 +0100 Message-Id: <199412091416.PAA15040@galois.loria.fr> To: mpi-comm@CS.UTK.EDU Subject: subscribe X-Mailer: exmh version 1.2 1/14/94 Date: Fri, 09 Dec 1994 15:16:41 +0100 From: Eric Dillon subscribe dillon@loria.fr From owner-mpi-comm@CS.UTK.EDU Fri Dec 16 10:47:41 1994 Received: from CS.UTK.EDU by netlib2 with ESMTP (cf v2.8t-netlib) id KAA23350; Fri, 16 Dec 1994 10:47:41 -0500 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id KAA03923; Fri, 16 Dec 1994 10:44:37 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 16 Dec 1994 10:44:36 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from redpoint.regent.e-technik.tu-muenchen.de by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id KAA03890; Fri, 16 Dec 1994 10:44:26 -0500 Received: by redpoint.regent.e-technik.tu-muenchen.de id <4727>; Fri, 16 Dec 1994 16:43:33 +0100 X-Mailer: exmh version 1.5 11/22/94 To: mpi-comm@CS.UTK.EDU Subject: subscribe Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Fri, 16 Dec 1994 16:43:23 +0100 Sender: Andreas Ganz From: Andreas.Ganz@regent.e-technik.tu-muenchen.de Message-Id: <94Dec16.164333met.4727@redpoint.regent.e-technik.tu-muenchen.de> --------------------------------------------------------------------------- Andreas Ganz asg@regent.e-technik.tu-muenchen.de Institute of Electronic Design Automation o__ Technical University of Munich _,>/'_ D-80290, Germany (_) \(_) Tel. +49 89 2105-3662 From owner-mpi-comm@CS.UTK.EDU Wed Apr 12 20:33:36 1995 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id UAA26228; Wed, 12 Apr 1995 20:33:36 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id UAA13519; Wed, 12 Apr 1995 20:31:39 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 12 Apr 1995 20:31:37 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from kanga.cse.nd.edu by CS.UTK.EDU with ESMTP (cf v2.9s-UTK) id UAA13508; Wed, 12 Apr 1995 20:31:35 -0400 From: Received: from owl.cse.nd.edu (hooper.cse.nd.edu [129.74.25.119]) by kanga.cse.nd.edu (8.6.12/8.6.12) with ESMTP id TAA10491 for ; Wed, 12 Apr 1995 19:31:33 -0500 Received: (lums@localhost) by owl.cse.nd.edu (8.6.10/8.6.9) id TAA27500; Wed, 12 Apr 1995 19:31:31 -0500 Date: Wed, 12 Apr 1995 19:31:31 -0500 Message-Id: <199504130031.TAA27500@owl.cse.nd.edu> To: mpi-comm@CS.UTK.EDU Subject: MPI Developers Conference -- Preliminary Announcement and CFP Reply-to: Andrew.Lumsdaine@nd.edu *** PRELIMINARY ANNOUNCEMENT AND CALL FOR PAPERS *** MPI Developers Conference June 22-23, 1995 University of Notre Dame Notre Dame, IN 46556 The MPI Developers Conference is a conference for developers of applications which use the Message Passing Interface (MPI) standard and is intended to support the continued development and use of MPI and its extensions. The conference will provide a forum for developers from national laboratories, industry, and academia who are using MPI to present their ideas about, and experiences with, MPI. In addition, a full-day pre-conference tutorial on MPI will be offered on June 21, 1995. Since this conference is intended to be cutting-edge (and, if possible, controversial), we welcome abstracts for presented papers, oral presentations (i.e., talks with no full paper), panel discussions, and software demonstrations. Abstracts should be approximately 500 words (+- 250) in length and sent by email (in ASCII, LaTeX, PostScript, or HTML format), if possible. The deadline for submitting abstracts is May 29, 1995. Abstracts will be reviewed and authors notified of their acceptance by June 5, 1995. Final versions of accepted presented papers will be due June 21, 1995 and should be provided in a form suitable for reading by WWW browsers (HTML preferred). In keeping with the cutting-edge nature of the conference, the conference proceedings will be made available on the World Wide Web. Please submit abstracts (in ASCII, LaTeX, PostScript, or HTML format) to mpidc95@lsc.cse.nd.edu Topics include (but are not limited to): o Applications using MPI o Implementations of MPI o Extensions to MPI o Why MPI is better/worse than other message passing systems o Message passing versus shared memory o The future of message passing o Software issues Tutorial: An MPI tutorial will be offered on June 21, 1995. This tutorial will be unique -- it will not be just an extended lecture about MPI. The tutorial will be conducted in Notre Dame's SPARC-based instructional cluster where Each tutorial attendee will have use of a SPARC-10 workstation throughout the tutorial. The tutorial lectures will be interspersed with a series of exercises requiring each tutorial attendee to write, debug, and run MPI-based programs using C or Fortran. Because of the fixed number of SPARCs available in the cluster, tutorial attendance is limited to 20. Instructors: Anthony Skjellum, William Gropp, Nathan Doss, Andrew Lumsdaine Software Demonstrations: Conference attendees who wish to provide software demonstrations instead of, in addition to, or in conjunction with a paper are encouraged to do so. A SPARC-based instructional cluster will be reserved for this purpose. Please indicate on your abstract submission if you are interested in doing a demo and what your computational requirements will be. Social: The following social events will be held in the Center for Continuing Education at Notre Dame: Reception, evening of June 21 Lunch, June 22 & 23 Banquet, evening of June 22 Exact times will be announced with the preliminary program for the conference. Registration: A registration form is available at http://www.cse.nd.edu/mpidc95/Registration.ps Early registration (before June 2, 1995) is $150 for all attendees. After June 2, the registration fee will be $200. On-site registration will be accepted. The conference fees include an evening reception on June 21, lunch on June 22 & 23, and a banquet on June 22. The tutorial fee is $150 for conference attendees, $225 for non-conference attendees. Tutorial attendance is limited to the first 20 respondents. Mail your completed registration form with payment to: MPI Developers Conference Center for Continuing Education P.O. Box 1008 University of Notre Dame Notre Dame, IN 46556 All fees are payable by credit card (M/C or Visa) or check to the University of Notre Dame. Credit card registrations may be faxed to (219) 631-8083 (Web registration is not supported at this time because of concerns about the present state of secure transactions). Transportation: Notre Dame is served by the South Bend airport (officially known as the Michiana Regional Transportation Center). The airport is about 20 minutes from Notre Dame. Taxi and bus service are available. Most major airlines fly into South Bend. Delta, USAir, and Northwest offer jet service (DC-9 and 737) while United and American provide commuter service. Notre Dame is about a two hour drive from Chicago O'Hare airport. A map and directions to Notre Dame from the South Bend airport and from O'Hare airport will be made available on the conference Web page (http://www.cse.nd.edu/mpidc95/). Hotel Accomodations: A block of rooms has been reserved at the Holiday Inn University Area (219) 272-6600. Shuttle service will be available at specified times during the conference. If you would like us make a reservation for you, please indicate so on your registration form. Organizing Committee: Andrew Lumsdaine, University of Notre Dame, Department of Computer Science and Engineering Greg Burns, Ohio Supercomputer Center William Gropp, Argonne National Laboratory Ewing Lusk, Argonne National Laboratory Anthony Skjellum, Mississippi State University, NSF Engineering Research Center for Computational Field Simulation + Computer Science Department For further information, see: http://www.cse.nd.edu/mpidc95/ or contact Andrew Lumsdaine Department of Computer Science and Engineering University of Notre Dame Notre Dame, IN 46556 Phone: (219) 631-8716 Fax: (219) 631-9260 Internet: lumsdaine.1@nd.edu From owner-mpi-core@mcs.anl.gov Thu May 18 12:16:27 1995 Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id MAA16523; Thu, 18 May 1995 12:16:26 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id LAA15303 for mpi-comm-archive@netlib2.cs.utk.edu; Thu, 18 May 1995 11:16:35 -0500 Date: Thu, 18 May 1995 11:16:35 -0500 Message-Id: <199505181616.LAA15303@antares.mcs.anl.gov> To: mpi-comm-archive@netlib2.cs.utk.edu From: Majordomo@mcs.anl.gov Subject: Welcome to mpi-comm Reply-To: Majordomo@mcs.anl.gov -- Welcome to the mpi-comm mailing list! If you ever want to remove yourself from this mailing list, send the following command in email to "mpi-comm-request@mcs.anl.gov": unsubscribe Or you can send mail to "Majordomo@mcs.anl.gov" with the following command in the body of your email message: unsubscribe mpi-comm mpi-comm-archive@netlib2.cs.utk.edu Here's the general information for the list you've subscribed to, in case you don't already have it: [Last updated on: Wed May 17 21:33:45 1995] This is the MPI Forum discussion list, for all parts of MPI and MPI-2. Email participation from those not attending meetings is welcome. To unsubscribe, send unsubscribe mpi-comm to majordomo@mcs.anl.gov. From owner-mpi-comm@CS.UTK.EDU Thu May 18 12:20:03 1995 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id MAA16556; Thu, 18 May 1995 12:19:58 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id MAA02350; Thu, 18 May 1995 12:14:03 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 18 May 1995 12:14:02 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with ESMTP (cf v2.9s-UTK) id MAA02338; Thu, 18 May 1995 12:14:00 -0400 Received: from mcs.anl.gov (donner.mcs.anl.gov [140.221.5.134]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id LAA15220 for ; Thu, 18 May 1995 11:13:57 -0500 Message-Id: <199505181613.LAA15220@antares.mcs.anl.gov> To: mpi-comm@CS.UTK.EDU Subject: mpi-comm mailing list Date: Thu, 18 May 1995 11:13:56 -0500 From: Rusty Lusk Dear mpi-comm, David and I are moving the mailing lists, in order to place them under the control of the majordomo list manager, which will let people subsucribe and unsubscribe themselves from lists, find out who is on them, etc. You should soon get a message welcoming you to the "new" mpi-comm mailing list, since we are automatically subscribing everyone on mpi-comm. The full address of the mpi-comm mailing list will now be mpi-comm@mcs.anl.gov instead of mpi-comm@cs.utk.edu. After that, you should get (on the new list) a set of subcommittee mailing lists that you can join. Let us know if there are any questions, problems, or confusions. David Walker and Rusty Lusk From mpi-comm-human@mcs.anl.gov Thu May 18 14:32:37 1995 Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id OAA17831; Thu, 18 May 1995 14:32:37 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id NAA19795 for mpi-comm-out; Thu, 18 May 1995 13:31:03 -0500 Received: from mcs.anl.gov (donner.mcs.anl.gov [140.221.5.134]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id NAA19788; Thu, 18 May 1995 13:30:56 -0500 Message-Id: <199505181830.NAA19788@antares.mcs.anl.gov> To: mpi-core@antares.mcs.anl.gov, mpi-comm@antares.mcs.anl.gov Subject: Subcommittee discussion lists Date: Thu, 18 May 1995 13:30:55 -0500 From: Rusty Lusk Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk You are getting this note because you are subscribed to either of the following lists mpi-comm@mcs.anl.gov MPI Forum discussion list, all comments welcome mpi-core@mcs.anl.gov MPI forum meeting attendees list The MPI-2 Forum has tentatively formed subcommittees to work on various projects within MPI-2. We are not automatically subscribing anyone to these lists. If you want to participate in the discussion on a particular topic, please send mail to majordomo.mcs.anl.gov containing subscribe The lists and their topics are: mpi-dynamic MPI Forum discussion list for dynamic process management mpi-1sided MPI Forum discussion list for one-sided communication mpi-coll MPI Forum discussion list for collective operations mpi-external MPI Forum discussion list for external interfaces mpi-bind MPI Forum discussion list for C++ and Fortran-90 bindings Other subcommittees might appear. These are the ones agreed on at the first MPI-2 meeting. In MPI-1, the most specific discussions took place on the the subcommittee lists, where draft proposals were put together. We apologize for the barrage of mail today on organizing the mailing lists. This should be the last message on this topic. Regards, Rusty Lusk and David Walker From owner-mpi-comm@CS.UTK.EDU Fri Jun 9 19:43:13 1995 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id TAA26786; Fri, 9 Jun 1995 19:43:13 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id TAA06191; Fri, 9 Jun 1995 19:39:40 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 9 Jun 1995 19:39:38 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from macsch.com by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id TAA06177; Fri, 9 Jun 1995 19:39:36 -0400 Received: from [161.34.71.49] by macsch.com (5.61/MSC-950526) id AA16700; Fri, 9 Jun 95 16:39:35 -0700 Received: by warsaw.local (4.1/SunOS) id AA19178; Fri, 9 Jun 95 16:46:33 PDT Date: Fri, 9 Jun 95 16:46:33 PDT From: sciviz@macsch.com (Sciviz CO. (controlled by Alan Caserio)) Message-Id: <9506092346.AA19178@warsaw.local> To: gropp@mcs.anl.gov, mpi-comm@CS.UTK.EDU Subject: mpi user interface Cc: sciviz@madrid.macsch.com Hello :-) I am Mark Kelly (username sciviz). I have some experience with pvm, but none with mpi, I have been asked to work on a project involving NASA, Rockwell, and the MacNeal-Schwendler Corporation, and some cfd code involving mpi. My specific area relates to job submission, load balancing, job monitoring, etc. I was wondering about a user interface (X/Motif type) on top of or integrated with mpi, do you know of anyone working in this area of mpi ? If so, please let me know who they are/how I can meet/talk/e-mail them. Thank you for all time and effort in advance, Mark Kelly (sciviz@macsch.com) Cc: dongarra@cs.utk.edu From owner-mpi-comm@CS.UTK.EDU Sat Jun 10 15:27:08 1995 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id PAA01863; Sat, 10 Jun 1995 15:27:08 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id PAA05631; Sat, 10 Jun 1995 15:24:47 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sat, 10 Jun 1995 15:24:46 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from macsch.com by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id PAA05624; Sat, 10 Jun 1995 15:24:45 -0400 Received: from [161.34.71.49] by macsch.com (5.61/MSC-950526) id AA29400; Sat, 10 Jun 95 12:24:40 -0700 Received: by warsaw.local (4.1/SunOS) id AA16504; Sat, 10 Jun 95 12:31:38 PDT Date: Sat, 10 Jun 95 12:31:38 PDT From: sciviz@macsch.com (Sciviz CO. (controlled by Alan Caserio)) Message-Id: <9506101931.AA16504@warsaw.local> To: raja@tbag.osc.edu Subject: lam, xmpi Cc: lam@tbag.osc.edu, mpi-comm@CS.UTK.EDU Raja, hello :-) This is Mark Kelly (username sciviz) again. > My specific area relates to job submission, > load balancing, job monitoring, etc. > > I was wondering about a user interface (X/Motif type) > on top of or integrated with mpi, do you know of anyone > working in this area of mpi ? If so, please let me > know who they are/how I can meet/talk/e-mail them. --- >> Take a look at LAM. It's an MPI cluster computing environment for UNIX, >> and has XMPI, a Motif GUI to monitor and control the application. The next >> version of XMPI will be out within a month and will support more features, >> including tracing. I can show you a sneak preview if needed. :-) --- Thanks so much, it is really neat. A couple of questions - 1). Does (or will) the next versions of lam/xmpi involve node selection based on load, resource criteria, etc. ? 2). If not, would you (or others in the lam project) want to incorporate some modular load scheduling/balancing routines into lam/xmpi ? Perhaps some vendors provide closed tools for managing/monitoring mpi programs, (IBM, LSF ?) but I am interested in developing a more open (multi-vendor) gui to submit mpi programs to clusters. Let me know if and how I can help ! Any and all info is greatly appreciated. Mark Kelly (sciviz@macsch.com) From mpi-comm-human@mcs.anl.gov Wed Jul 5 23:32:22 1995 Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id XAA21782; Wed, 5 Jul 1995 23:32:21 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id WAA05498 for mpi-comm-out; Wed, 5 Jul 1995 22:28:55 -0500 Received: from HARVARDA.HARVARD.EDU (harvarda.harvard.edu [128.103.60.11]) by antares.mcs.anl.gov (8.6.10/8.6.10) with SMTP id WAA05231; Wed, 5 Jul 1995 22:24:57 -0500 Message-Id: <199507060324.WAA05231@antares.mcs.anl.gov> Received: from physics1.byu.edu by HARVARDA.HARVARD.EDU (IBM VM SMTP V2R2) with TCP; Wed, 05 Jul 95 23:22:04 EDT Date: Wed, 5 Jul 1995 21:21:57 -0600 To: mpi-coll@mcs.anl.gov, mpi-comm@mcs.anl.gov, mpi-core@mcs.anl.gov, mpi-dynamic@mcs.anl.gov, mpi-external@mcs.anl.gov From: David@wishes.to.remain.anonymous.UA Subject: your romance ad in USSR pn R A #L O (Unverified) Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk ## meet women of the former USSR through romance ads ## Months ago, Olga Kosmina placed my personal ad in several papers of the former Soviet Union. Since that time I have received over 40 responses for the $50 I mailed Olga. (I believe she paid the newspapers something around $35 and kept the rest for her efforts.) I have found greater success and savings by placing my own personal romance advertisement rather than purchasing addresses through Russian "bride" catalog companies. If you are interested in placing a personal romance ad as I did, contact Olga. She has built up a list of most every newspaper and magazine in the former Soviet Union and could help direct your ad to certain areas if you wish. She writes, "please say that I place all ad _throughout_ Russia and other countries of former Soviet Union, not only Western Russia." Olga is 23 years old, has a bachelors in biology and works full-time as a florist in Kiev. She speaks, reads and writes English as well as her native languages of Russian and Ukrainian. I realize that it is a very trusting person who would put $ into an envelope and mail to a foreign country. If you would rather send a letter of inquiry first, Olga will respond to your questions. It takes about 16 days for a letter to travel from the USA to Kiev. Olga Kozmina Dekabristov Street 5 - 178 Kiev 253121 Ukraine I have found that by placing a single bill between two pieces of newsprint inside an envelope, the Ukrainian post cannot see through and does not bother to tamper. I have yet to lose a letter sent to Kiev. I am sorry that Olga does not have e-mail because it would make contact with her much easier. I am posting anonymously because of the inordinate amount of e-mail which I would receive -- inquiries as well as flames. Best Wishes, David and Olga Although Olga has never seen a newsgroup nor heard of "net-etiquette," she believes that helping others exceeds the cost of angering those who feel the net should not be used in this fashion. IHA (I humbly ask) that you not flame the postmaster of this site. peace. . . From mpi-comm-human@mcs.anl.gov Thu Jul 20 23:03:45 1995 Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id XAA12555; Thu, 20 Jul 1995 23:03:43 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id WAA05054 for mpi-comm-out; Thu, 20 Jul 1995 22:08:50 -0500 Received: from VM1.NoDak.EDU (SMTP@vm1.NoDak.edu [134.129.111.1]) by antares.mcs.anl.gov (8.6.10/8.6.10) with SMTP id WAA04877; Thu, 20 Jul 1995 22:06:11 -0500 Date: Thu, 20 Jul 1995 22:06:11 -0500 Message-Id: <199507210306.WAA04877@antares.mcs.anl.gov> Received: from physics1.byu.edu by VM1.NoDak.EDU (IBM VM SMTP V2R2) with TCP; Thu, 20 Jul 95 22:05:05 CDT To: mpi-bind@mcs.anl.gov, mpi-coll@mcs.anl.gov, mpi-comm@mcs.anl.gov, mpi-core@mcs.anl.gov, mpi-dynamic@mcs.anl.gov From: romance@in.the.former.USSR.ua Subject: letter from Olga! =) (Unverified) Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk Not to long ago, I posted a message re: meeting women of the former soviet union through romance ads. In August, Olga will travel to Moscow from her home in Kiev, Ukraine. In Moscow, Olga will have a much easier and cost efficient means to place your personal romance ad throughout Russia. Last week I received the following from Olga: "I have already sent your ad to the papers in such towns: Moscow, St. Petersburg, Vladimir, Kazan. At nearest future I will send your ad to the paper in some more 12 towns of Russia, where papers are published. Some times (in winter, spring & now) I placed your ad in other papers, but they are not most popular paper in Moscow and some large cities of Russia. Besides, I am continuing to place your ad in papers of Ukraine. I promise to place your ad in some other papers when I will come to Moscow in August. I am glad that you have received fairly many letters from Russian & Ukrainian girls and I think you will received some more ones and will find your ideal in my country soon. I thank you very much ones more for your kindness & your help. My best wishes, Olga" This isn't a scam - call it panhandling if you want. . . I sent her $40 or $50 and I've received over 45 responses. Unlike placing romance ads in the U.S., women from the former USSR respond. Although one would guess the are doing so in the hopes of American citizenship, I haven't found it so. Olga lives in Kiev, Ukraine (population 3 million) and will travel to Moscow in August to visit her father. If you were to send a letter this week, she would receive it in time. The population of Moscow is 10 million -- (3 times the size of Los Angeles.) Feel free to send a letter and ask her your questions. She will be happy to respond. Olga's address: Ukraine Kiev 253121 Dekabristov Street 5 - 178 Olga Kozmina I am posting anonymously because of the flames and volume of inquiries that would result otherwise. I think those who are truely interested will take the time to write. _____________________________________________________________________________ To: probable flamer Subject: polite note Although Olga has never seen a newsgroup nor heard of "net-etiquette," she believes that offering lonely singles the possibility of romance exceeds the cost of angering those who feel the net shouldn't be used in this fashion. IHA (I humbly ask) that you not flame the postmaster of this site. peace. . . From owner-mpi-comm@CS.UTK.EDU Thu Aug 17 01:09:32 1995 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id BAA17785; Thu, 17 Aug 1995 01:09:32 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id BAA10312; Thu, 17 Aug 1995 01:04:44 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 17 Aug 1995 01:04:43 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from theory.lcs.mit.edu by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id BAA10305; Thu, 17 Aug 1995 01:04:42 -0400 Received: from vulture.lcs.mit.edu by theory.lcs.mit.edu (5.65c/TOC-1.2S) id AA27340; Thu, 17 Aug 95 01:04:39 EDT From: rudolph@theory.lcs.mit.edu (Larry Rudolph) Received: by vulture.lcs.mit.edu (5.65c/TOC-1.2C) id AA28949; Thu, 17 Aug 95 01:04:10 EDT Date: Thu, 17 Aug 95 01:04:10 EDT Message-Id: <199508170504.AA28949@vulture.lcs.mit.edu> To: mpi-comm@CS.UTK.EDU Subject: subscribe From owner-mpi-comm@CS.UTK.EDU Wed Aug 23 06:18:56 1995 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id GAA01162; Wed, 23 Aug 1995 06:18:56 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id GAA11742; Wed, 23 Aug 1995 06:13:32 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 23 Aug 1995 06:13:31 EDT Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from basfegw.basf-corp.com by CS.UTK.EDU with ESMTP (cf v2.9s-UTK) id GAA11727; Wed, 23 Aug 1995 06:13:29 -0400 From: Received: from basfigw.basf-corp.com by basfegw.basf-corp.com with SMTP (PP); Wed, 23 Aug 1995 12:13:19 +0200 Received: from nwbz.zxd.basf-ag.de by basfigw.basf-corp.com with SMTP (PP); Wed, 23 Aug 1995 06:13:13 -0400 Received: from snow-white.zxa.basf-ag.de by nwbz.zxd.basf-ag.de with SMTP (PP); Wed, 23 Aug 1995 12:13:08 +0200 Received: by snow-white.zxa.basf-ag.de (AIX 3.1/UCB 5.61/BASF AG 91-10-31) id AA14952; Wed, 23 Aug 95 12:35:16 GMT Date: Wed, 23 Aug 95 12:35:16 GMT Message-Id: <9508231235.AA14952@snow-white.zxa.basf-ag.de> To: mpi-comm@CS.UTK.EDU Subject: help help How do I get access to the discussion list? Thanks, Rob From mpi-comm-human@mcs.anl.gov Sun Oct 1 14:57:12 1995 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id OAA23533; Sun, 1 Oct 1995 14:57:10 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id NAA16214 for mpi-comm-out; Sun, 1 Oct 1995 13:52:42 -0500 Received: from mcs.anl.gov (donner.mcs.anl.gov [140.221.5.134]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id NAA16205 for ; Sun, 1 Oct 1995 13:52:29 -0500 Message-Id: <199510011852.NAA16205@antares.mcs.anl.gov> To: mpi-comm@antares.mcs.anl.gov Subject: MPI Poll results Date: Sun, 01 Oct 1995 13:52:28 -0500 From: Rusty Lusk Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk Many of you may have already seen this, but I thought I would pass it along. Although the sample is not at all random, there are interesting results here: Those who use MPI like it. People do use the advanced features of MPI-1, although there are some ease-of-use problems in datatypes (no surprise). Most of the things they would like to have we are indeed working on for MPI-2, but not all. Thanks to Nick Nevin. Regards, Rusty MPI Poll '95 Results A poll was conducted to better understand how programmers are using the rich MPI functionality, what extensions are needed, and how to prioritize our work to better serve MPI users, and the HPC community at large. The complete results of the poll are also available on WWW at the following URL: http://www.osc.edu/Lam/mpi/mpi_poll.html There were 129 responses. The numbers given beside each answer indicate the total number of respondents who checked that answer. --------------------------------------------------------------------- Are you using MPI? (check one) (104) Currently using it ( 25) May use it in the future --------------------------------------------------------------------- Which types of decomposition are you currently using? (check all that apply) (68) Master/Slave (81) Grid Decomposition (46) Other --------------------------------------------------------------------- Which machine architectures are you currently using? (check all that apply) ( 83) NOW (Network of Workstations) ( 35) Standalone SMP (SGI, SUN, etc.) (101) MPP (T3D, Paragon, SP2, Exemplar, CS2, etc.) --------------------------------------------------------------------- Which MPI groups of functions do you use? (check all that apply) - Point-to-Point: (98) Blocking (93) Non-blocking (39) Derived Datatypes (93) Collective - Groups, Contexts, and Communicators: (54) Group Functions (54) Communicators (Intra) beyond MPI_COMM_WORLD (15) Inter-communicators ( 8) Attribute Caching - Topologies: (41) Cartesian (17) Graph - Environment: (24) Predefined Attributes (16) Error Handlers --------------------------------------------------------------------- Which derived datatype constructors do you use? (check all that apply) (39) Contiguous (39) Vector (22) Indexed (38) Struct (19) Pack --------------------------------------------------------------------- Which intra-communicator creation calls do you use? (check all that apply) (29) Comm_dup (40) Comm_create (28) Comm_split These are the verbatim responses to the question on missing MPI functionality. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? |> Dynamic task creation. One-sided communications --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? active message types, remote execution, interrupt-messages --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? > Dynamic creation of tasks for client-server programs > Parallel I/O --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? Better standard I/O facilities --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? Some simple forms of remote actions without explcit polling by the remote program: - remote memory access - fully asynchronous collective operations - active messages Full fledged multithreading is not so important. I would rather prefer to reduce the functionality of MPI in some areas. In particular, I would propose to standardize a v e r y small subset of functions which are easy to implement (efficiently). These can then serve as a basis for generic implementations of MPI. This would improve the availability of MPI on new machines and research machines. Has anybody thought about standardizing benchmarks? As an assembler handbook for a CISC computer is incomplete without a table of cycle times, I consider a library implementation without timings for the most common operations incomplete. One way to force vendors to disclose this vital infomation would be to make a set of benchmarks part of some kind of certification process. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? * For porting existing parallel progs to MPI it would be useful to be able to have processes ranked from 1->n instead of 0->n-1. There does not appear to be a way of doing this (as far as I can make out from the standard!). --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? MPI_Pprobe(int source, int tagcount, int *tagarray, MPI_Comm comm, MPI_Status *status) This will wait for any message with from a given array of tags, but given the choice between two or more messages will return the status of the message with the lowest tag-index. MPI_Ipprobe(...) As above, but does not block. MPI_Discard(int source, int tag, MPI_Comm comm) Especially when writing task-farms and pipelines, there are many cases when the existence of a message gives enough information and the content can be discarded. There are also cases where the message itself is out of date. Creating a receive buffer for a message that is not going to be read is a waste of resources, and can also be quite complicated for the programmer. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? _ _ Dynamic startup of processes, C++ binding --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? Nothing that I can think of at the moment... Parallel I/O would be nice, though. We can deal without it, (obviously), but it might make things a little easier and more streamlined. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? An easy way to send a message from one group to ALL members of another group. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? > Some facility for inter-application communication. I would like to > be able to set up an inter-communicator between two applications > that are started at different times. Specifically, I would like the > ability to create persistent servers and transient clients. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? Our current NX implementation uses hrecvx() -- whether we can reasonably convert this to multiple threads or processes is as yet unclear to us. We also start processes at initialization (only). A portable way to do this would be very useful. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? - Spawning as it is possible in PVM: one process creates another. - An asynchronous receive function should have a pointer to a function as parameter. This function is called when the message can is received (this is possible e.g. in CMMD for the CM-5). --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? Dynamic process creation. The ability for a new program to join to a running parallel application. Calls to provide host and platform information. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? >1. Support for atomicity, e.g., an atomic transaction >2. Support for a process to asynchronously send a msg to a group of processes, such as the eureka mechanism in Cray T3D --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? > 1)congestion free multi broadcasting for balanced data distribution > 2)i/o :) --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? MPI_SPAWN - ABILITY TO CREATE A PROCESS -- MAY NOT REALLY FIT IN MPI --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? Guaranteed access to stdio. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? I would like to have the facility to set up a non-blocking receive with a pointer to a handling function. Upon arival of the message, the function would be called. The facility for supplying a pointer to a function that returns a pointer to a receive buffer would be very useful. Upon arrival of the message, the function would be called and the message DMA'ed into the memory pointed to by the function (which would have access to all of the message's header information). A null pointer returned by the function would indicate that the message should be ignored. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? Dynamic creation or abortion of process; Dynamic relabeling of process; Simulat ion of Fault-tolerance. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? I need to investigate MPI further before answering this --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? - support for threads - active messages - integrated MPI-IO --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? > - multicast operation that replaces replicated pt-to-pt sends. > - an operation to create new communicators without a requirement that it be a collective call across an existing communicator (ie. like the existing comm_create but where only the members of the new communicator need to make the call - when all processes to be in the new comm have made the call, then everyone completes) --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? > It would be nice to have a standard, class-based C++ interface... --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? > SCATTER & GATHER functions to decompose 2 dim. arrarys over a grid. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? * Myself and other users of the AP1000 machine which has a broadcast network use a multicast type operation very often. It is hoped that the current proposal within the MPI-2 forum for multicast will be accepted. * Remote memory operations - like the current put/get proposal. * Interrupt driven send/recv calls - again that has been proposed within the MPI-2 forum. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? IO - this is a big pain! --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? > Advanced reduce (SUM) subroutine, where only those values different than > zero are summed up. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? > Interrupt driven handled messages, ala Intel's NX would be useful > for certain of my requirements. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? Remote operations: load, store, [op] to memory (e.g., add to memory.) As an alternative, "active messages" (which can be used to implement remote operations.) This functionality is *very* important to my applications. Without it, MPI is not terribly interesting. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? A handler based receive function, so that my code will be interrupted asynchronously and a specified handler will be run when a message arrives. This could make a big difference to efficiency (our local MPI guru is in the process of adding MPI_Hrecv(). --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? Non-blocking extension of collective communication for different buffers. For example call MPI_REDUCE(A,...) call MPI_REDUCE(B,...) I need CALL MPI_REDUCE_NONBLOCKing(A...) CALL MPI_REDUCE_NONBLOCKing(B,...) CALL MPI_BARRIER() --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? Remote procedure calls. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? |> A more efficient global sum!! --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? I'm pretty new to this game, so I may not understand all the implications, but I think a non-blocking version of MPI_Bcast() would be useful. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? Haven't run across any yet... Libraries I may want to use (mostly from PNL HPCC group) don't yet run under MPI, but they're working on it... I believe they would be happier if there were explicit shared memory (get/put) support. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? Receive-add (or more general receive-op) would be very nice for all the different types of receives. Native complex type for C/C++ is needed It would be nice to be able to do a scatter where the origin (not the destination as it seems to be now) creates the derived datatype that is used on the receiving end. After all how can you destination create the indexed derived datatype if it doesn't know what is coming. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? Addition/deletion of dynamic created processes to MPI_COMM_WORLD at runtime. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? Dynamic Process Management --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? 1) The standard for C should include complex numbers, it use to be there and then was taken out, I need this for C++ and global operations. Now I have to ifdef code for using complex or real with a data type of complex I won't need to 2) Receives should support add to location or more general merge into location rather then just copy into buffer. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? Active messages, parallel IO, task creation --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? c++ binding/support (I do know about MPI++) --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? Parallel I/O - especially disk read/writes --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? I/O --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? None right now, but in my opinion it would be interesting to add task (or thread) management functios. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? Task support like PVM (spawn, ...) --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? General high-level permutation routines (like CM-2/CM-5 CMF_send/CMF_get) --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? The ability to expand and contract the working space, or number of nodes. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? 1) A boolean function returns whether you are in the Communicator or not. 2) A mapping function to tell a rank in communicator "A" corresponding to a rank in communicator "B". --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? Those things that MPI-2 is designed to address, like I/O. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? Ability to query unused capacity of user buffer for buffered communication (so that I know whether a buffered send request will be successful). --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? a send that implies already a receive (a little bit like in data parallel programming, e.g. C*) --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? Dynamic process control! --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? Parallel I/O is definitely needed. I'm having problems using very simple I/O with flushing/seeking to EOF on an SP-2. Of course, this isn't really supported so it shouldn't work, but parallel I/O would fix it. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? full functionalityi in an implementation. To actually do what we ask not default to another mode that is safe. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? None --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? * I need some mechanism for informing programs about the interconnection topology, so the application can make partition decisions. For example, on a machine with a communication hierarchy, such as a cluster of shared-memory machines, my apps need to know which tasks are in the same box, and which are in different boxes to make partitioning decisions. Presumably the problem is general. * related to this, I need a collective communication operation that exchanges common task environmental information. For example, most of my programs begin with an Allgather of hostname/pid/ip address/... info, so every task knows common info about the other tasks. Some MPE function to provide similar environmental info, if it was possible in a relatively portable way would be ideal. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? I am in urge need for a communication between the workstation cluster and the massively parallel systems. Therefor it would be good to have MPI supporting different protocols simultaniously. Spawning processes and integrating allready running processes into an existing MPI_COMM_WORLD would be a good feature, too. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? A) dynamical creation of processes B) MPI IO standard --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? Exclusive scans (rather than inclusive ones) would be nice --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? Multi threaded funcionality, but the proposals for MPI2 may cover this missing thing (requests with handler functions) --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? C++ bindings would really help... An easy way to debug and trace ALL mpi traffic in an application. --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? Threads --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? process control -- creation (spawning) and deletion (kill) --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? parallel IO to Files --------------------------------------------------------------------- What missing MPI functionality do you need for your programs? c++ binding In a Master-Slave context one often has to use gatherv etc because there is no data to be sent from the master. This complicates the code, requiring creation of recvcounts and displs arrays which are all the same except for the first element! I would like `gatherx' (for "exclusive") etc to behave the same as gather but with no contribution from root. These are the verbatim responses to the request for general commentary on MPI. --------------------------------------------------------------------- Comments: > I use MPI for systems programming. I am building a parallel language > run-time system on top of it. I do not use it for application programming. --------------------------------------------------------------------- Comments: !Active messages are highly desirable! although not part of the current standard... --------------------------------------------------------------------- Comments: I'm not very familiar with MPI. Mostly I had used Express and PVM for parallelization. But I'm interested in using MPI for more portable programming. --------------------------------------------------------------------- Comments: > MPI is very useful. Nevertheless, It is very hard to convince > people to natively implement in MPI. Usually, MPI is one of the > transport layer (with PVM & Parmacs), so only basic functionnalities > are used. Probably, the lack of native versions (supported by > hardware provider) and associated tools is the main reason for that. > > Some points are not very clear in the standard such as the beaviour > of some functions (ex: ready mode,...) --------------------------------------------------------------------- Comments: Excellent product, not to say almost perfect. --------------------------------------------------------------------- Comments: * Currently use only the predefined communicator MPI_COMM_WORLD in all of my applications. * The MPI timing routines are very useful for having *portable* timing, however it is not clear (to me) how comparable these are from one system to another. * The various MPI_allxxx, MPI_allxxxv routines are all highly useful, however it is vital that implementations of these are heavily optimised. -> For some application codes I have set up a compile time choice between MPI collective routines OR hand written (reasonably efficient) collective routines using only MPI sends and recvs; this is necessary to avoid a performance hit on machines which have very naive implementations of collective routines. * There is a general drift here towards fortran 90. Specific bindings for F90 will therefore be needed; there do not seem to have been any efforts to address this. * In general, I find MPI far better to use than all other message passing setups with which I am familiar (PVM,Intel NX,etc.). I am now using it in *all* the application programs that I am working on to ensure portability. --------------------------------------------------------------------- Comments: Derived data types are an absolute pain to construct, and are even more difficult to debug. This is particularly true when creating types with non-standard offsets. Some means of 'viewing' the actual message content (i.e. the underlying basic values) would be invaluable. --------------------------------------------------------------------- Comments: I am encouragin all our customers/collaborators to use MPI as the standard. We are responsible for supporting some 50% of the UK Grand-Challenge applicatio ns on a 320-processor Cray T3D, 16-processor IBM SP-2 and 64-processor Intel iPSC/860. Applications in a wide area of science and engineering. I am using MPI with Fortran-90. --------------------------------------------------------------------- Comments: _ contains allmost everything one can ask for... --------------------------------------------------------------------- Comments: It would be nice to have a universal way to run MPI. I realize that some implementations are trying to use "mpirun" as a general environment, which is nice for workstations; but what about the big MPP's, like the SP2, and such? If the vendors could all get together on this too, that would be great! Then us poor programmers wouldn't have to worry about the easily-confused users getting confused! :) --------------------------------------------------------------------- Comments: We use MPI (among other things) to provide communication for HPF programs. My greatest concern is better performance, not more functionality. --------------------------------------------------------------------- Comments: The MPI_UB and MPI_LB stuff is really awkward. There should be a better solution to this. I would suggest alternate versions of the type creation calls which allow the extent of the datatype just be passed as an argument. --------------------------------------------------------------------- Comments: THERE COULD HAVE BEEN NOTHING BETTER THAN A STANDARD INTERFACE LIKE MPI. CREDITS TO ALL THOSE INVOLVED IN DEVELOPING THE STANDARD AND ALSO THOSE WHO QUICKLY MADE THE IMPLEMENTATIONS AVAILABLE. --------------------------------------------------------------------- Comments: I look forward to reading the survey results and following future developments. --------------------------------------------------------------------- Comments: > - the MPI standard has been an immense success in reducing the need for online debugging on MPP systems; most development can now proceed on a workstation (or a workstation network), and most problems debugged before testing on the much more inaccessible MPP system. --------------------------------------------------------------------- Comments: > I'm not tremendously familiar with all the MPI concepts so I'm not absolutely sure whether or not I need/will use them. Thus my nonanswers to some of the questions. --------------------------------------------------------------------- Comments: * On the whole, besides the above "difficiencies", I feel most of our users are quite happy with MPI, and it has been quite successful in porting largish scientific programs to the AP1000 that have been written in MPI. * It would be nice if it could be more rigoriously defined as to how threads and signals can be intermixed with MPI calls. As far as I can tell, this hasn't really be address by the standard or at this stage, using threads and signals will not make your application "MPI compliant". --------------------------------------------------------------------- Comments: > I use the ANU implementation of MPI for the AP 1000. The > feature that I would most like to see is the ability to run > multiple processes on a single processor (as can be done with > the native message passing software). I'm not sure if > any other implementors have done this, but it would be very > useful. --------------------------------------------------------------------- Comments: add IO :-) --------------------------------------------------------------------- Comments: Some comments can be found in my paper for MPI Developer Conference '95 http://www.cse.nd.edu/mpidc95/proceedings/papers/postscript/fang.ps --------------------------------------------------------------------- Comments: |> I am working on a code for quantum checial calculations; a direct |> self consitent fild program including electron correlation. |> I am only working on the whole processor array ( mpi_comm_world ) and I am only using 10 subroutines from mpi: MPI_INIT, MPI_COMM_RANK, MPI_COMM_SIZE, MPI_BCAST, MPI_SEND, MPI_RECV, MPI_ALLREDUCE( sum, max and min ), MPI_REDUCE( sum, max, min ), MPI_BARRIER, MPI_FINALIZE --------------------------------------------------------------------- Comments: I have had good experiences with MPI (MPICH implementation on our SP2 and RS6000 network). This is the first message passing system I've used, and I have found it to work well in parallelizing my applications. --------------------------------------------------------------------- Comments: MPI is great. It is nice (and very rare) to see a commitee create something that is MUCH better then any of the previous packages. --------------------------------------------------------------------- Comments: I may have already repleid (can't remember). This project is a finite element code for wave propagation. One of the aims was to try and evaluate how some intermediate/advanced features of MPI could help express parallelism. They can ! --------------------------------------------------------------------- Comments: I'm submitting this as a developer of our intercommunicator extensions to MPI so I may not be quite the kind of user you're wanting responses from -- Nathan --------------------------------------------------------------------- Comments: Good work.. --------------------------------------------------------------------- Comments: Keep up the great work! --------------------------------------------------------------------- Comments: MPI is great - no doubt about it. I personally think that message passing is bound to go the way of assembly language programming sometime in the future, but in the meantime - I think the forum did an excellent job, and have continued to do so. --------------------------------------------------------------------- Comments: Today I use PVM --------------------------------------------------------------------- Comments: The debugging tools of LAM are great! Thanks! --------------------------------------------------------------------- Comments: The datatype constructor seems to be a bit tedious and complicated. --------------------------------------------------------------------- Comments: MPI is a wonderful example of the benefits that can be derived from a group of people working together to generate a truly useful tool. --------------------------------------------------------------------- Comments: What I really miss from several of the MPI implementations is the ability to start up each process running inside a debugger such as xdbx. Debugging numerical code with big arrays where something is walking over something else is made much more easy if this functionality is available. So far I've only found CHIMP-MPI does this easily although I can fool LAM into doing it if I don't run my code using MPIRUN. --------------------------------------------------------------------- Comments: This answer is for a program that is currently being written. We are doing higher order finite elements for solving the wave eqaution by domain decomposition. The features I've checked are those we *think* we'll be using but we do not yet have any actual experience, or feedback, as to else we'll need. It's possible we'll need some simple form of global communications (algorithms is explicit, so no global residual, but maybe a global energy). Also it is not clear how much is to be gained by using graph topologies (in terms of expressiveness, not performance). --------------------------------------------------------------------- Comments: I just bumped into LAM and MPI...looks good. Would consider how to use it for both performance and fault tolerance in computers on aircraft (radar processing, sensor image processing, etc.). --------------------------------------------------------------------- Comments: I'm just getting started in a new project. We are planning to use MPI, but how so, we haven't decided. --------------------------------------------------------------------- Comments: I find MPI very exciting to use. It has many features not present in other message-passing systems (e.g. the BLACS). For example, MPI_Bcast is much easier than if's with MPI_Bsend and MPI_Brecv's. --------------------------------------------------------------------- Comments: There should be a stripped version of MPI for single group code, which uses only the six basic MPI calls for message passing. A majority of user codes in MPI are this type, and the less overhead in the language, the better. --------------------------------------------------------------------- Comments: Advanced enviroment for program developement, debugging and profiling required. --------------------------------------------------------------------- Comments: I hope I did not fill this out already. We are using MPI to parallelize a Four dimensional data assimilation system (.e.g. optimal interpolation). THe project is classified as one of the great challange problems. --------------------------------------------------------------------- Comments: I am not using MPI yet. I am starting to get it and then I will install it at my university network. Now I am checking ftp and www sites. --------------------------------------------------------------------- Comments: All my MPI codes so far are straightforward translations of PVM code to MPI, so I haven't made use of features like derived datatypes yet, though they look useful. --------------------------------------------------------------------- Comments: MPI is great, it does most of the important things to make life bearable. Having said that... I tried to use a derived data type once, but it didn't work quite the way I needed. I had a 3 dimensional array in Fortran, and wanted to send a collection of 'strips' of it from a number of slave processes so that they could be gathered into the same sort of structure in the master process --- however, when I tried to define a single type to represent the set of strips (using mpi_type_indexed) I found that, because of the way extent works, they were gathered with information from consecutive slaves contiguous, which is not what I wanted. To illustrate, here is a schematic representation of the final array at the root, using numbers to indicate the process each strip has come from--- wanted: 1..1..1..2..2..2..3..3..3.. got: 1..1..12..2..23..3..3 I could no -=- Nick Nevin nevin@tbag.osc.edu Ohio Supercomputer Center http://www.osc.edu/lam.html From owner-mpi-comm@CS.UTK.EDU Mon Oct 16 11:59:05 1995 Return-Path: Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id LAA03044; Mon, 16 Oct 1995 11:59:04 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id LAA06196; Mon, 16 Oct 1995 11:52:15 -0400 Received: from dlr1 by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id LAA06187; Mon, 16 Oct 1995 11:52:10 -0400 Received: by dlr1 id AA14050 (5.67b/IDA-1.5 for mpi-comm@cs.utk.edu); Mon, 16 Oct 1995 16:48:20 +0100 From: "Martin Strietzel" Message-Id: <9510161648.ZM14048@dlr1.MI.Uni-Koeln.DE> Date: Mon, 16 Oct 1995 16:48:18 +0100 X-Mailer: Z-Mail (3.2.0 06sep94) To: mpi-comm@CS.UTK.EDU Subject: subscribe mpi-comm mst@zpr.uni-koeln.de Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii subscribe mpi-comm mst@zpr.uni-koeln.de From owner-mpi-comm@CS.UTK.EDU Mon Oct 16 12:41:51 1995 Return-Path: Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id MAA04466; Mon, 16 Oct 1995 12:41:50 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id MAA10778; Mon, 16 Oct 1995 12:39:48 -0400 Received: from hermes.intel.com by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id MAA10766; Mon, 16 Oct 1995 12:39:30 -0400 Received: from ichips.intel.com by hermes.intel.com (5.65/10.0i); Mon, 16 Oct 95 09:38:56 -0700 Received: from pdx362 by ichips.intel.com (5.64+/10.0i); Mon, 16 Oct 95 09:38:27 -0700 Received: by pdx362.intel.com (AIX 3.2/UCB 5.64/10.0i); Mon, 16 Oct 1995 09:38:22 -0700 Date: Mon, 16 Oct 1995 09:38:19 -0700 (PDT) From: Garry Petrie To: mpi-comm@CS.UTK.EDU Subject: Unsubscribe Message-Id: Mime-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Please unsubscribe me from this mailer. _______________________________________________________________________________ | Garry Petrie M/S JF1-56 Phone: | | Intel Corporation 5200 NE Elam Young Pkwy 503-264-3027 day | | gpetrie@ichips.intel.com Hillsboro, Oregon 97124 503-690-5465 evenings | From mpi-comm-human@mcs.anl.gov Tue Oct 17 15:57:35 1995 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id PAA19351; Tue, 17 Oct 1995 15:57:05 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id OAA05040 for mpi-comm-out; Tue, 17 Oct 1995 14:52:39 -0500 Received: from mcs.anl.gov (donner.mcs.anl.gov [140.221.5.134]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id OAA05014; Tue, 17 Oct 1995 14:51:45 -0500 Message-Id: <199510171951.OAA05014@antares.mcs.anl.gov> To: mpi-comm@antares.mcs.anl.gov, mpi-dynamic@antares.mcs.anl.gov Subject: dynamic processes chapter Date: Tue, 17 Oct 1995 14:51:42 -0500 From: Rusty Lusk Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk Here is the draft of the dynamic processes chapter. Changes from last time (extensive) are highlighted at the beginning. Rusty, Bill S., and Al %!PS-Adobe-2.0 %%Creator: dvips 5.528 Copyright 1986, 1994 Radical Eye Software %%Title: temp.dvi %%CreationDate: Tue Oct 17 14:16:54 1995 %%Pages: 20 %%PageOrder: Ascend %%BoundingBox: 0 0 612 792 %%EndComments %DVIPSCommandLine: dvips -o temp.ps temp %DVIPSParameters: dpi=300, comments removed %DVIPSSource: TeX output 1995.10.17:1416 %%BeginProcSet: tex.pro /TeXDict 250 dict def TeXDict begin /N{def}def /B{bind def}N /S{exch}N /X{S N}B /TR{translate}N /isls false N /vsize 11 72 mul N /hsize 8.5 72 mul N /landplus90{false}def /@rigin{isls{[0 landplus90{1 -1}{-1 1} ifelse 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale isls{landplus90{VResolution 72 div vsize mul 0 exch}{Resolution -72 div hsize mul 0}ifelse TR}if Resolution VResolution vsize -72 div 1 add mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@landscape{/isls true N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder}N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail}B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{ pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get}B /ch-xoff{128 ch-data dup length 3 sub get sub}B /ch-yoff{ch-data dup length 2 sub get 127 sub}B /ch-dx{ch-data dup length 1 sub get}B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ ch-image}imagemask restore}B /D{/cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D }B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto /V matrix currentmatrix dup 1 get dup mul exch 0 get dup mul add .99 lt{/QV}{/RV}ifelse load def pop pop}N /eop{SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if pop /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put}for 65781.76 div /vsize X 65781.76 div /hsize X}N /p{show}N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V {}B /RV statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval dup(Display)eq exch 0 4 getinterval(NeXT)eq or}{pop false} ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /QV{gsave transform round exch round exch itransform moveto rulex 0 rlineto 0 ruley neg rlineto rulex neg 0 rlineto fill grestore}B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail} B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{ 3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{ 3 2 roll p a}B /bos{/SS save N}B /eos{SS restore}B end %%EndProcSet TeXDict begin 40258431 52099146 1000 300 300 (/tmp_mnt/Net/antireo/antireo6/lusk/mpi2/report2/temp.dvi) @start /Fa 4 112 df<03CC063C0C3C181C3838303870387038E070E070E070E070E0E2 C0E2C0E261E462643C380F127B9115>97 D<01F007080C08181C3838300070007000E000 E000E000E000E000E008E010602030C01F000E127B9113>99 D<1F800380038007000700 070007000E000E000E000E001C001C001C001C0038003800380038007000700070007000 E400E400E400E40068003800091D7C9C0B>108 D<01E007180C0C180C380C300E700E70 0EE01CE01CE01CE018E038E030E06060C031801E000F127B9115>111 D E /Fb 37 122 df<00E001C0038007000E000E001C001C003800380038007000700070 007000E000E000E000E000E000E000E000E000E000E000E000E000700070007000700038 00380038001C001C000E000E000700038001C000E00B2A7E9E10>40 DI44 D<001C0000003E0000003E0000002E0000006700000067000000E7800000C7800000C380 0001C3C0000183C0000181C0000381E0000381E0000700F0000700F0000600F0000E0078 000FFFF8000FFFF8001C003C001C003C0018003C0038001E0038001E0070001F0070000F 0070000F00E0000780191D7F9C1C>65 D<003FC000FFF003C0F00780300F00001E00003C 00003C0000780000780000780000F00000F00000F00000F00000F00000F00000F00000F0 0000F000007800007800007800003C00003C00001E00000F000807801803C07800FFF000 3F80151F7D9D1B>67 D69 DI<003F8001FFF003C0F80780380F 00181E00003C00003C0000780000780000780000F00000F00000F00000F00000F00000F0 0000F007F8F007F8F000387800387800387800383C00383C00381E00380F003807803803 C0F801FFF0003F80151F7D9D1C>I I I76 DII<003F000001FFE00003FFF00007C0F8000F807C001E001E003E001F003C000F007800 07807800078078000780F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F000 03C0F00003C0F80007C078000780780007807C000F803C000F003E001F001F003E000F80 7C0007C0F80003FFF00001FFE000003F00001A1F7E9D1F>II82 D<03F8000FFE001C0F00380700700300600000E00000E00000E00000E00000F000007800 007F00003FE0001FFC0007FE0001FF00001F800007800003C00003C00001C00001C00001 C00001C0C00180E00380F007007C0E001FFC0007F000121F7E9D17>IIII89 D<7FFFF07FFFF00001E00003E00003C00007C0000780000F00001F00001E00003E00003C 0000780000F80000F00001F00001E00003C00007C0000780000F80000F00001E00003E00 003C00007C0000780000FFFFF0FFFFF0141D7E9C19>I<0FC03FF07FF87038401C001C00 1C00FC0FFC3FFC781CE01CE01CE01CF07C7FFC7FDC3F1C0E127E9114>97 D<07E00FF81FFC3C1C70047000E000E000E000E000E000E000700070043C1C1FFC0FF807 E00E127E9112>99 D<07C01FE03FF078787018601CFFFCFFFCFFFCE000E000E000700070 043C1C3FFC1FF807E00E127E9112>101 D104 DI109 DI<03F0000FFC001F FE003C0F00780780700380E001C0E001C0E001C0E001C0E001C0F003C07003807807803C 0F001FFE000FFC0003F00012127F9115>I<078E1FEE3FFE7C3E781E700EE00EE00EE00E E00EE00EE00EF00E701E7C3E3FFE1FEE0F8E000E000E000E000E000E000E000E000E0F1A 7E9115>113 DI<1FC03FF07FF0F030E000E000F0007F003FC01FE000F0 003800388038F078FFF07FE01FC00D127F9110>I<1C001C001C001C001C001C00FFE0FF E01C001C001C001C001C001C001C001C001C001C001C001C001C201FF00FF007C00C187F 970F>II119 D121 D E /Fc 17 119 df<1FE0003FF0007FF800783C00300E00000E0000 0E0003FE001FFE003E0E00700E00E00E00E00E00E00E00783E007FFFE03FE7E00F83E013 127E9116>97 D<03F80FFC1FFE3C1E780C7000E000E000E000E000E000F000700778073E 0E1FFC0FF803F010127D9116>99 D<003F00007F00003F00000700000700000700000700 03C7000FF7001FFF003C1F00780F00700700E00700E00700E00700E00700E00700E00700 700F00700F003C1F001FFFE00FE7F007C7E014197F9816>I<001F00007F8000FF8001E7 8001C30001C00001C0007FFF00FFFF00FFFF0001C00001C00001C00001C00001C00001C0 0001C00001C00001C00001C00001C00001C0003FFE007FFF003FFE0011197F9816>102 D<03E3C007F7E00FFFE01C1CC0380E00380E00380E00380E00380E001C1C000FF8001FF0 001BE0003800001800001FFC001FFF003FFF807803C0E000E0E000E0E000E0E000E07001 C07C07C03FFF800FFE0003F800131C7F9116>I<7E0000FE00007E00000E00000E00000E 00000E00000E3C000EFE000FFF000F87800F03800E03800E03800E03800E03800E03800E 03800E03800E03800E03800E03807FC7F0FFE7F87FC7F01519809816>I<018003C003C0 018000000000000000007FC07FC07FC001C001C001C001C001C001C001C001C001C001C0 01C001C07FFFFFFF7FFF101A7D9916>I<7E0000FE00007E00000E00000E00000E00000E 00000E7FE00E7FE00E7FE00E0F000E1E000E3C000E78000EF0000FF0000FF8000FBC000F 1E000E0E000E07000E07807F87F0FFCFF07F87F01419809816>107 D109 D<7E3C00FEFE007FFF000F87800F03800E03800E03800E03800E03800E03800E03800E03 800E03800E03800E03807FC7F0FFE7F87FC7F01512809116>I<03E0000FF8001FFC003C 1E00780F00700700E00380E00380E00380E00380E00380F00780700700780F003C1E001F FC000FF80003E00011127E9116>I<7E3E00FEFF007FFF800F83C00F00E00E00E00E0070 0E00700E00700E00700E00700E00700E00E00F01E00F83C00FFF800EFF000E3C000E0000 0E00000E00000E00000E00000E00007FC000FFE0007FC000141B809116>I114 D<0FEC3FFC7FFCF03CE01CE01C7000 7F801FF007F8003C600EE00EF00EF81EFFFCFFF8C7E00F127D9116>I<03000007000007 00000700000700007FFF00FFFF00FFFF0007000007000007000007000007000007000007 000007010007038007038007038007870003FE0001FC0000F80011177F9616>I<7E1F80 FE3F807E1F800E03800E03800E03800E03800E03800E03800E03800E03800E03800E0380 0E03800E0F800FFFF007FBF803E3F01512809116>I<7F1FC0FF1FE07F1FC01C07001E0F 000E0E000E0E000E0E00071C00071C00071C00071C0003B80003B80003B80001F00001F0 0000E00013127F9116>I E /Fd 66 124 df<007E1F0001C1B1800303E3C00703C3C00E 03C1800E01C0000E01C0000E01C0000E01C0000E01C0000E01C000FFFFFC000E01C0000E 01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E 01C0000E01C0000E01C0000E01C0000E01C0000E01C0007F87FC001A1D809C18>11 D<007E0001C1800301800703C00E03C00E01800E00000E00000E00000E00000E0000FFFF C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01 C00E01C00E01C00E01C00E01C07F87F8151D809C17>I<007FC001C1C00303C00703C00E 01C00E01C00E01C00E01C00E01C00E01C00E01C0FFFFC00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C07F CFF8151D809C17>I<6060F0F0F8F86868080808080808101010102020404080800D0C7F 9C15>34 D<0F0000C0188000C030600380703807006027FB00E0100600E0100C00E0100C 00E0101800E0101800E0103000E0106000602060007020C00030418000188180000F0303 C00006062000060C10000C1C08001818080018380400303804006038040060380400C038 0400C03804018038040300180803001C0806000C100C000620040003C01E217E9E23>37 D<60F0F8680808081010204080050C7C9C0C>39 D<004000800100020006000C000C0018 001800300030007000600060006000E000E000E000E000E000E000E000E000E000E000E0 00E000600060006000700030003000180018000C000C00060002000100008000400A2A7D 9E10>I<800040002000100018000C000C000600060003000300038001800180018001C0 01C001C001C001C001C001C001C001C001C001C001C00180018001800380030003000600 06000C000C00180010002000400080000A2A7E9E10>I<60F0F070101010102020408004 0C7C830C>44 DI<60F0F06004047C830C>I<0001000300060006 0006000C000C000C0018001800180030003000300060006000C000C000C0018001800180 030003000300060006000C000C000C00180018001800300030003000600060006000C000 C00010297E9E15>I<03C00C301818300C300C700E60066006E007E007E007E007E007E0 07E007E007E007E007E007E007E00760066006700E300C300C18180C3007E0101D7E9B15 >I<03C00C301818300C700C600EE006E006E007E007E007E007E0076007700F300F1817 0C2707C700060006000E300C780C78187010203030C00F80101D7E9B15>57 D<60F0F0600000000000000000000060F0F06004127C910C>I<0FE03038401CE00EF00E F00EF00E000C001C0030006000C000800180010001000100010001000100000000000000 0000000003000780078003000F1D7E9C14>63 D<000600000006000000060000000F0000 000F0000000F00000017800000178000001780000023C0000023C0000023C0000041E000 0041E0000041E0000080F0000080F0000180F8000100780001FFF80003007C0002003C00 02003C0006003E0004001E0004001E000C001F001E001F00FF80FFF01C1D7F9C1F>65 DI<001F808000E061800180198007000780 0E0003801C0003801C00018038000180780000807800008070000080F0000000F0000000 F0000000F0000000F0000000F0000000F0000000F0000000700000807800008078000080 380000801C0001001C0001000E000200070004000180080000E03000001FC000191E7E9C 1E>IIII<001F808000E0618001801980070007800E000380 1C0003801C00018038000180780000807800008070000080F0000000F0000000F0000000 F0000000F0000000F0000000F000FFF0F0000F8070000780780007807800078038000780 1C0007801C0007800E00078007000B800180118000E06080001F80001C1E7E9C21>II I76 DII<00 3F800000E0E0000380380007001C000E000E001C0007003C00078038000380780003C078 0003C0700001C0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0F0 0001E0700001C0780003C0780003C0380003803C0007801C0007000E000E0007001C0003 80380000E0E000003F80001B1E7E9C20>I I82 D<07E0801C1980300580700380600180E00180E0 0080E00080E00080F00000F800007C00007FC0003FF8001FFE0007FF0000FF80000F8000 07C00003C00001C08001C08001C08001C0C00180C00180E00300D00200CC0C0083F80012 1E7E9C17>I<7FFFFFC0700F01C0600F00C0400F0040400F0040C00F0020800F0020800F 0020800F0020000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F 0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F 0000001F800003FFFC001B1C7F9B1E>IIII89 D<08081010202040404040808080808080B0B0F8F8787830300D0C7A9C15>92 D<0810204040808080B0F87830050C7D9C0C>96 D<1FC000307000783800781C00301C00 001C00001C0001FC000F1C00381C00701C00601C00E01C40E01C40E01C40603C40304E80 1F870012127E9115>II<07E00C30 1878307870306000E000E000E000E000E000E00060007004300418080C3007C00E127E91 12>I<003F00000700000700000700000700000700000700000700000700000700000700 03E7000C1700180F00300700700700600700E00700E00700E00700E00700E00700E00700 600700700700300700180F000C370007C7E0131D7E9C17>I<03E00C301818300C700E60 06E006FFFEE000E000E000E00060007002300218040C1803E00F127F9112>I<00F8018C 071E061E0E0C0E000E000E000E000E000E00FFE00E000E000E000E000E000E000E000E00 0E000E000E000E000E000E000E000E007FE00F1D809C0D>I<00038003C4C00C38C01C38 80181800381C00381C00381C00381C001818001C38000C300013C0001000003000001800 001FF8001FFF001FFF803003806001C0C000C0C000C0C000C06001803003001C0E0007F8 00121C7F9215>II<18003C003C00 18000000000000000000000000000000FC001C001C001C001C001C001C001C001C001C00 1C001C001C001C001C001C001C00FF80091D7F9C0C>I<00C001E001E000C00000000000 0000000000000000000FE000E000E000E000E000E000E000E000E000E000E000E000E000 E000E000E000E000E000E000E000E060E0F0C0F1C061803E000B25839C0D>IIII< FC7C001C87001D03001E03801C03801C03801C03801C03801C03801C03801C03801C0380 1C03801C03801C03801C03801C0380FF9FF014127F9117>I<03F0000E1C001806003003 00700380600180E001C0E001C0E001C0E001C0E001C0E001C06001807003803003001806 000E1C0003F00012127F9115>II<03C1000C3300180B00 300F00700700700700E00700E00700E00700E00700E00700E00700600700700700300F00 180F000C370007C700000700000700000700000700000700000700000700003FE0131A7E 9116>II<1F9030704030C010C010E010F8007F803FE00FF000F8803880 18C018C018E010D0608FC00D127F9110>I<04000400040004000C000C001C003C00FFE0 1C001C001C001C001C001C001C001C001C001C101C101C101C101C100C100E2003C00C1A 7F9910>IIII<7F8FF00F03800F030007 020003840001C80001D80000F00000700000780000F800009C00010E00020E0006070004 03801E07C0FF0FF81512809116>II<7FFC703860384070 40F040E041C003C0038007000F040E041C043C0C380870087038FFF80E127F9112>II E /Fe 8 118 df<78FCFCFCFC7800000000000078FCFCFCFC780612 7D910D>58 D68 D<03FC000E0E001C1F003C1F00781F00780E00 F80000F80000F80000F80000F80000F800007800007801803C01801C03000E0E0003F800 11127E9115>99 D<1E003F003F003F003F001E00000000000000000000000000FF00FF00 1F001F001F001F001F001F001F001F001F001F001F001F001F001F00FFE0FFE00B1E7F9D 0E>105 D110 D<01FC000F07801C01C03C01E07800F07800F0F800F8F800F8F800F8F800F8F800F8F800 F87800F07800F03C01E01E03C00F078001FC0015127F9118>I<1FD830786018E018E018 F000FF807FE07FF01FF807FC007CC01CC01CE01CE018F830CFC00E127E9113>115 D117 D E /Ff 29 122 df76 DII80 D<03FC080FFF381E03F83800F870007870 0038F00038F00018F00018F80000FC00007FC0007FFE003FFF801FFFE00FFFF007FFF000 FFF80007F80000FC00007C00003CC0003CC0003CC0003CE00038E00078F80070FE01E0E7 FFC081FF00161F7D9E1D>83 D85 DI<07FC001FFF003F0F803F07C03F03E03F03 E00C03E00003E0007FE007FBE01F03E03C03E07C03E0F803E0F803E0F803E0FC05E07E0D E03FF8FE0FE07E17147F9319>97 DI<01FE0007FF801F0FC03E0FC03E0FC07C0FC07C0300FC0000FC00 00FC0000FC0000FC0000FC00007C00007E00003E00603F00C01F81C007FF0001FC001314 7E9317>I<0007F80007F80000F80000F80000F80000F80000F80000F80000F80000F800 00F80000F801F8F80FFEF81F83F83E01F87E00F87C00F87C00F8FC00F8FC00F8FC00F8FC 00F8FC00F8FC00F87C00F87C00F87E00F83E01F81F07F80FFEFF03F8FF18207E9F1D>I< 01FE0007FF800F83C01E01E03E00F07C00F07C00F8FC00F8FFFFF8FFFFF8FC0000FC0000 FC00007C00007C00003E00181E00180F807007FFE000FF8015147F9318>I<001F8000FF C001F3E003E7E003C7E007C7E007C3C007C00007C00007C00007C00007C000FFFC00FFFC 0007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C0 0007C00007C00007C00007C0003FFC003FFC0013207F9F10>I<01FC3C07FFFE0F079E1E 03DE3E03E03E03E03E03E03E03E03E03E01E03C00F07800FFF0009FC001800001800001C 00001FFF800FFFF007FFF81FFFFC3C007C70003EF0001EF0001EF0001E78003C78003C3F 01F80FFFE001FF00171E7F931A>II<1C003E007F007F007F003E001C00000000000000000000000000 FF00FF001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F00 FFE0FFE00B217EA00E>I107 DII I<01FF0007FFC01F83F03E00F83E00F87C007C7C007CFC007EFC007EFC007EFC007EFC00 7EFC007E7C007C7C007C3E00F83E00F81F83F007FFC001FF0017147F931A>I<01F81807 FE381F87783F01F83E01F87E00F87C00F8FC00F8FC00F8FC00F8FC00F8FC00F8FC00F87C 00F87E00F87E00F83F01F81F87F80FFEF803F8F80000F80000F80000F80000F80000F800 00F80000F80007FF0007FF181D7E931C>113 DI<0FE63FFE701E600EE006E006F800FFC07FF83FFC1F FE03FE001FC007C007E007F006F81EFFFCC7F010147E9315>I<01800180018003800380 038007800F803F80FFFCFFFC0F800F800F800F800F800F800F800F800F800F800F860F86 0F860F860F8607CC03F801F00F1D7F9C14>II119 DII E /Fg 35 125 df<0E1F3F3F1D0102020404081020C0080E779F0E> 39 D<70F8F8F0E005057B840E>46 D<070F1F1F0E0000000000000000000070F8F8F0E0 08147B930E>58 D<00000200000006000000060000000E0000001E0000001E0000003F00 00002F0000004F0000004F0000008F0000010F0000010F0000020F0000020F0000040F00 000C0F0000080F0000100F0000100F0000200F80003FFF800040078000C0078000800780 01000780010007800200078002000780060007801E000F80FF807FF81D207E9F22>65 D<0000FE0200078186001C004C0038003C0060003C00C0001C01C0001803800018070000 180F0000181E0000101E0000103C0000003C000000780000007800000078000000780000 00F0000000F0000000F0000000F0000000F0000080700000807000008070000100380001 0038000200180004000C001800060020000381C00000FE00001F217A9F21>67 D<01FFFFFE001E001C001E000C001E0004001E0004003C0004003C0004003C0004003C00 040078080800780800007808000078180000F0300000FFF00000F0300000F0300001E020 0001E0200001E0200001E0001003C0002003C0002003C0004003C0004007800080078001 8007800100078007000F001F00FFFFFE001F1F7D9E1F>69 D<0000FC040007030C001C00 980030007800E0007801C000380380003003800030070000300E0000301E0000201E0000 203C0000003C00000078000000780000007800000078000000F0000000F000FFF0F00007 80F0000780F0000F0070000F0070000F0070000F0070001E0038001E0018003E001C002E 000E00CC000383040000FC00001E217A9F23>71 D<01FE00007FC0001E0000FC00001E00 00F80000170001780000170001780000270002F00000270004F00000270004F000002700 08F00000470009E00000470011E00000470021E00000470021E00000870043C000008380 43C00000838083C00000838083C000010381078000010382078000010382078000010384 0780000203840F00000203880F00000203900F00000203900F00000401E01E00000401E0 1E00000401C01E00000C01801E00001C01803E0000FF8103FFC0002A1F7D9E29>77 D<01FF007FE0001F000F00001F0004000017800400001780040000278008000023C00800 0023C008000023C008000041E010000041E010000041F010000040F010000080F0200000 807820000080782000008078200001003C400001003C400001003C400001001E40000200 1E800002001E800002000F800002000F800004000F0000040007000004000700000C0007 00001C00020000FF80020000231F7D9E22>I<01FFFF80001E00E0001E0070001E003800 1E003C003C003C003C003C003C003C003C003C0078007800780078007800F0007800E000 F003C000F00F0000FFFC0000F0000001E0000001E0000001E0000001E0000003C0000003 C0000003C0000003C00000078000000780000007800000078000000F800000FFF000001E 1F7D9E1F>80 D<0FFFFFF01E0780E0180780201007802020078020200F0020600F002040 0F0020400F0020801E0040001E0000001E0000001E0000003C0000003C0000003C000000 3C00000078000000780000007800000078000000F0000000F0000000F0000000F0000001 E0000001E0000001E0000001E0000003E00000FFFF00001C1F789E21>84 D<7FFC1FF807C003C00780010007800100078001000F0002000F0002000F0002000F0002 001E0004001E0004001E0004001E0004003C0008003C0008003C0008003C000800780010 00780010007800100078001000F0002000F0002000F0002000F0004000F0004000700080 007001000030020000380400000C18000007E000001D20779E22>II<00F1800389C00707800E03801C03803C03 80380700780700780700780700F00E00F00E00F00E00F00E20F01C40F01C40703C40705C 40308C800F070013147C9317>97 D<007E0001C1000300800E07801E07801C07003C0200 780000780000780000F00000F00000F00000F00000F00000700100700200300400183800 07C00011147C9315>99 D<0000780003F80000700000700000700000700000E00000E000 00E00000E00001C00001C000F1C00389C00707800E03801C03803C038038070078070078 0700780700F00E00F00E00F00E00F00E20F01C40F01C40703C40705C40308C800F070015 207C9F17>I<007C01C207010E011C013C013802780C7BF07C00F000F000F000F0007000 700170023804183807C010147C9315>I<00007800019C00033C00033C00071800070000 0700000E00000E00000E00000E00000E0001FFE0001C00001C00001C00001C0000380000 380000380000380000380000700000700000700000700000700000700000E00000E00000 E00000E00000C00001C00001C0000180003180007B0000F300006600003C00001629829F 0E>I<003C6000E27001C1E00380E00700E00F00E00E01C01E01C01E01C01E01C03C0380 3C03803C03803C03803C07003C07001C0F001C17000C2E0003CE00000E00000E00001C00 001C00301C00783800F0700060E0003F8000141D7E9315>I<01E0000FE00001C00001C0 0001C00001C000038000038000038000038000070000070000071E000763000E81800F01 C00E01C00E01C01C03801C03801C03801C0380380700380700380700380E10700E20700C 20701C20700C40E00CC060070014207D9F17>I<00C001E001E001C00000000000000000 0000000000000E003300230043804300470087000E000E000E001C001C001C0038403880 30807080310033001C000B1F7C9E0E>I<01E0000FE00001C00001C00001C00001C00003 80000380000380000380000700000700000703C00704200E08E00E11E00E21E00E40C01C 80001D00001E00001FC00038E000387000387000383840707080707080707080703100E0 3100601E0013207D9F15>107 D<03C01FC0038003800380038007000700070007000E00 0E000E000E001C001C001C001C0038003800380038007000700070007100E200E200E200 E200640038000A207C9F0C>I<1C0F80F0002630C318004740640C004780680E00470070 0E004700700E008E00E01C000E00E01C000E00E01C000E00E01C001C01C038001C01C038 001C01C038001C01C0708038038071003803806100380380E10038038062007007006600 300300380021147C9325>I<1C0F802630C04740604780604700704700708E00E00E00E0 0E00E00E00E01C01C01C01C01C01C01C03843803883803083807083803107003303001C0 16147C931A>I<007C0001C3000301800E01C01E01C01C01E03C01E07801E07801E07801 E0F003C0F003C0F003C0F00780F00700700F00700E0030180018700007C00013147C9317 >I<01C1E002621804741C04781C04701E04701E08E01E00E01E00E01E00E01E01C03C01 C03C01C03C01C0380380780380700380E003C1C0072380071E000700000700000E00000E 00000E00000E00001C00001C0000FFC000171D809317>I<00F0400388C00705800E0380 1C03803C0380380700780700780700780700F00E00F00E00F00E00F00E00F01C00F01C00 703C00705C0030B8000F380000380000380000700000700000700000700000E00000E000 0FFE00121D7C9315>I<1C1E002661004783804787804707804703008E00000E00000E00 000E00001C00001C00001C00001C00003800003800003800003800007000003000001114 7C9313>I<00FC030206010C030C070C060C000F800FF007F803FC003E000E700EF00CF0 0CE008401020601F8010147D9313>I<018001C0038003800380038007000700FFF00700 0E000E000E000E001C001C001C001C003800380038003820704070407080708031001E00 0C1C7C9B0F>I<0E00C03300E02301C04381C04301C04701C08703800E03800E03800E03 801C07001C07001C07001C07101C0E20180E20180E201C1E200C264007C38014147C9318 >I<0E03803307802307C04383C04301C04700C08700800E00800E00800E00801C01001C 01001C01001C02001C02001C04001C04001C08000E300003C00012147C9315>I<0E00C1 C03300E3C02301C3E04381C1E04301C0E04701C060870380400E0380400E0380400E0380 401C0700801C0700801C0700801C0701001C0701001C0602001C0F02000C0F04000E1308 0003E1F0001B147C931E>I124 D E /Fh 78 126 df<70F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F870000000000070F8F8F87005 1C779B18>33 D<4010E038F078E038E038E038E038E038E038E038E038E038E03860300D 0E7B9C18>I<030600078F00078F00078F00078F00078F00078F007FFFC0FFFFE0FFFFE0 7FFFC00F1E000F1E000F1E000F1E000F1E000F1E007FFFC0FFFFE0FFFFE07FFFC01E3C00 1E3C001E3C001E3C001E3C001E3C000C1800131C7E9B18>I<01C00007E0000FF0000E70 001C38001C38001C38001C38001C73F01C73F01CE3F00FE3800FC7000F87000F07001F0E 003F0E007B8E0073DC00E1DC00E0F800E0F800E07070E0787070FC707FFFE03FCFE00F03 C0141C7F9B18>38 D<387C7C7E3E0E0E0E1C1C38F8F0C0070E789B18>I<007000F001E0 03C007800F001E001C00380038007000700070007000E000E000E000E000E000E000E000 E0007000700070007000380038001C001E000F00078003C001F000F000700C24799F18> I<6000F00078003C001E000F000780038001C001C000E000E000E000E000700070007000 70007000700070007000E000E000E000E001C001C0038007800F001E003C007800F00060 000C247C9F18>I<01C00001C00001C00001C000C1C180F1C780F9CF807FFF001FFC0007 F00007F0001FFC007FFF00F9CF80F1C780C1C18001C00001C00001C00001C00011147D97 18>I<00600000F00000F00000F00000F00000F00000F00000F0007FFFC0FFFFE0FFFFE0 7FFFC000F00000F00000F00000F00000F00000F00000F00000600013147E9718>I<1C3E 7E7F3F1F070E1E7CF860080C788518>I<7FFF00FFFF80FFFF807FFF0011047D8F18>I<30 78FCFC78300606778518>I<000300000780000780000F80000F00001F00001E00001E00 003E00003C00007C0000780000780000F80000F00001F00001E00003E00003C00003C000 07C0000780000F80000F00000F00001F00001E00003E00003C00003C00007C0000780000 F80000F00000F0000060000011247D9F18>I<01F00007FC000FFE001F1F001C07003803 807803C07001C07001C0E000E0E000E0E000E0E000E0E000E0E000E0E000E0E000E0E000 E0F001E07001C07001C07803C03803801C07001F1F000FFE0007FC0001F000131C7E9B18 >I<01800380038007800F803F80FF80FB80438003800380038003800380038003800380 038003800380038003800380038003807FFCFFFE7FFC0F1C7B9B18>I<03F0000FFE003F FF007C0F807003C0E001C0F000E0F000E06000E00000E00000E00001C00001C00003C000 0780000F00001E00003C0000780000F00001E00007C0000F80001E00E03C00E07FFFE0FF FFE07FFFE0131C7E9B18>I<07F8001FFE003FFF007807807803C07801C03001C00001C0 0003C0000380000F0003FF0003FE0003FF000007800003C00001C00000E00000E00000E0 F000E0F000E0F001C0F003C07C07803FFF001FFE0003F800131C7E9B18>I<001F00003F 0000770000770000E70001E70001C7000387000787000707000E07001E07003C07003807 00780700F00700FFFFF8FFFFF8FFFFF8000700000700000700000700000700000700007F F000FFF8007FF0151C7F9B18>I<1FFF803FFF803FFF8038000038000038000038000038 00003800003800003800003BF8003FFE003FFF003C07801803C00001C00000E00000E060 00E0F000E0F000E0E001C07003C07C0F803FFF001FFC0003F000131C7E9B18>I<3078FC FC783000000000000000003078FCFC78300614779318>58 D<183C7E7E3C180000000000 000000183C7E7E3E1E0E1C3C78F060071A789318>I<000300000780001F80003F00007E 0001FC0003F00007E0001FC0003F00007E0000FC0000FC00007E00003F00001FC00007E0 0003F00001FC00007E00003F00001F8000078000030011187D9918>I<7FFFC0FFFFE0FF FFE0FFFFE0000000000000000000000000FFFFE0FFFFE0FFFFE07FFFC0130C7E9318>I< 600000F00000FC00007E00003F00001FC00007E00003F00001FC00007E00003F00001F80 001F80003F00007E0001FC0003F00007E0001FC0003F00007E0000FC0000F00000600000 11187D9918>I<00700000F80000F80000D80000D80001DC0001DC0001DC00018C00038E 00038E00038E00038E000306000707000707000707000707000FFF800FFF800FFF800E03 800E03801C01C01C01C07F07F0FF8FF87F07F0151C7F9B18>65 DI<00F8E003FEE007FFE00F07E01E03E03C01E03800E07000E07000 E0700000E00000E00000E00000E00000E00000E00000E00000E000007000007000E07000 E03800E03C00E01E01C00F07C007FF8003FE0000F800131C7E9B18>I<7FF800FFFE007F FF001C0F801C03C01C03C01C01E01C00E01C00E01C00F01C00701C00701C00701C00701C 00701C00701C00701C00701C00F01C00E01C00E01C01E01C01C01C03C01C0F807FFF00FF FE007FF800141C7F9B18>III<01F1C003FDC00FFFC01F0FC01C03C03803C03801C070 01C07001C0700000E00000E00000E00000E00000E00000E00FF0E01FF0E00FF07001C070 01C07003C03803C03803C01C07C01F0FC00FFFC003FDC001F1C0141C7E9B18>I<7F07F0 FF8FF87F07F01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01FFFC0 1FFFC01FFFC01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C0 7F07F0FF8FF87F07F0151C7F9B18>I<7FFF00FFFF807FFF0001C00001C00001C00001C0 0001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C0 0001C00001C00001C00001C00001C00001C0007FFF00FFFF807FFF00111C7D9B18>I<7F E000FFE0007FE0000E00000E00000E00000E00000E00000E00000E00000E00000E00000E 00000E00000E00000E00000E00000E00000E00000E00000E00700E00700E00700E00700E 00707FFFF0FFFFF07FFFF0141C7F9B18>76 DI<7E07F0FF0FF87F07F01D81C01D81C01D81C01DC1C01CC1C01CC1C01CE1C01CE1C01C E1C01C61C01C71C01C71C01C31C01C39C01C39C01C39C01C19C01C19C01C1DC01C0DC01C 0DC01C0DC07F07C0FF87C07F03C0151C7F9B18>I<0FF8003FFE007FFF00780F00700700 F00780E00380E00380E00380E00380E00380E00380E00380E00380E00380E00380E00380 E00380E00380E00380E00380E00380F00780700700780F007FFF003FFE000FF800111C7D 9B18>II<0FF8003FFE007FFF00780F0070 0700F00780E00380E00380E00380E00380E00380E00380E00380E00380E00380E00380E0 0380E00380E00380E00380E1E380E1E380F0E78070F700787F007FFF003FFE000FFC0000 1C00001E00000E00000F0000070000070011227D9B18>I<7FF800FFFE007FFF001C0F80 1C03801C03C01C01C01C01C01C01C01C03C01C03801C0F801FFF001FFE001FFE001C0F00 1C07001C03801C03801C03801C03801C03801C039C1C039C1C039C7F01F8FF81F87F00F0 161C7F9B18>I<03F3801FFF803FFF807C0F80700780E00380E00380E00380E000007000 007800003F00001FF00007FE0000FF00000F800003C00001C00000E00000E06000E0E000 E0E001E0F001C0F80780FFFF80FFFE00E7F800131C7E9B18>I<7FFFF8FFFFF8FFFFF8E0 7038E07038E07038E0703800700000700000700000700000700000700000700000700000 700000700000700000700000700000700000700000700000700000700007FF0007FF0007 FF00151C7F9B18>IIII<7F8FE07F9FE0 7F8FE00E07000F0700070E00078E00039C0003DC0001F80001F80000F00000F000007000 00F00000F80001F80001DC00039E00038E00070F000707000E07800E03801E03C07F07F0 FF8FF87F07F0151C7F9B18>II<3FFFE07F FFE07FFFE07001C07003C0700780700700000F00001E00001C00003C0000780000700000 F00001E00001C00003C0000780000700000F00001E00E01C00E03C00E07800E07000E0FF FFE0FFFFE0FFFFE0131C7E9B18>II93 D<7FFF00FFFF80FFFF807FFF0011047D7F18>95 D<1FE0003FF8007FFC00781E00300E00 00070000070000FF0007FF001FFF007F0700780700E00700E00700E00700F00F00781F00 3FFFF01FFBF007E1F014147D9318>97 D<7E0000FE00007E00000E00000E00000E00000E 00000E00000E3E000EFF800FFFC00FC1E00F80E00F00700E00700E00380E00380E00380E 00380E00380E00380F00700F00700F80E00FC1E00FFFC00EFF80063E00151C809B18>I< 01FE0007FF001FFF803E0780380300700000700000E00000E00000E00000E00000E00000 E000007000007001C03801C03E03C01FFF8007FF0001FC0012147D9318>I<001F80003F 80001F8000038000038000038000038000038003E3800FFB801FFF803C1F80380F807007 80700380E00380E00380E00380E00380E00380E00380700780700780380F803C1F801FFF F00FFBF803E3F0151C7E9B18>I<01F00007FC001FFE003E0F00380780700380700380E0 01C0E001C0FFFFC0FFFFC0FFFFC0E000007000007001C03801C03E03C01FFF8007FF0001 FC0012147D9318>I<001F80007FC000FFE000E1E001C0C001C00001C00001C0007FFFC0 FFFFC0FFFFC001C00001C00001C00001C00001C00001C00001C00001C00001C00001C000 01C00001C00001C00001C0007FFF007FFF007FFF00131C7F9B18>I<01E1F007FFF80FFF F81E1E301C0E003807003807003807003807003807001C0E001E1E001FFC001FF80039E0 003800001C00001FFE001FFFC03FFFE07801F0700070E00038E00038E00038E000387800 F07E03F01FFFC00FFF8001FC00151F7F9318>I<7E0000FE00007E00000E00000E00000E 00000E00000E00000E3E000EFF800FFFC00FC1C00F80E00F00E00E00E00E00E00E00E00E 00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E07FC3FCFFE7FE7FC3FC171C809B 18>I<03800007C00007C00007C0000380000000000000000000000000007FC000FFC000 7FC00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C000 01C00001C00001C000FFFF00FFFF80FFFF00111D7C9C18>I107 D<7FE000FFE0007FE00000E00000E00000E00000E00000E00000E0 0000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E0 0000E00000E00000E00000E0007FFFC0FFFFE07FFFC0131C7E9B18>I<7CE0E000FFFBF8 007FFFF8001F1F1C001E1E1C001E1E1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C 001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C007F1F1F00FFBFBF807F1F1F 001914819318>I<7E3E00FEFF807FFFC00FC1C00F80E00F00E00E00E00E00E00E00E00E 00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E07FC3FCFFE7FE7FC3FC17148093 18>I<01F0000FFE001FFF003E0F803803807001C07001C0E000E0E000E0E000E0E000E0 E000E0F001E07001C07803C03C07803E0F801FFF000FFE0001F00013147E9318>I<7E3E 00FEFF807FFFC00FC1E00F80E00F00700E00700E00380E00380E00380E00380E00380E00 380F00700F00700F80E00FC1E00FFFC00EFF800E3E000E00000E00000E00000E00000E00 000E00000E00007FC000FFE0007FC000151E809318>I<01E38007FB801FFF803E1F8038 0F80700780700780E00380E00380E00380E00380E00380E00380700780700780380F803C 1F801FFF800FFB8003E380000380000380000380000380000380000380000380003FF800 3FF8003FF8151E7E9318>I<7F87E0FF9FF07FBFF803F87803F03003E00003C00003C000 0380000380000380000380000380000380000380000380000380007FFE00FFFF007FFE00 15147F9318>I<07F7003FFF007FFF00780F00E00700E00700E007007C00007FE0001FFC 0003FE00001F00600780E00380E00380F00380F80F00FFFF00FFFC00E7F00011147D9318 >I<0180000380000380000380000380007FFFC0FFFFC0FFFFC003800003800003800003 80000380000380000380000380000380000380400380E00380E00380E001C1C001FFC000 FF80003E0013197F9818>I<7E07E0FE0FE07E07E00E00E00E00E00E00E00E00E00E00E0 0E00E00E00E00E00E00E00E00E00E00E00E00E00E00E01E00F03E007FFFC03FFFE01FCFC 1714809318>I<7F8FF0FF8FF87F8FF01E03C00E03800E03800E03800707000707000707 00038E00038E00038E00038E0001DC0001DC0001DC0000F80000F80000700015147F9318 >II<7F8FF0 7F9FF07F8FF0070700078E00039E0001DC0001F80000F80000700000F00000F80001DC00 039E00038E000707000F07807F8FF0FF8FF87F8FF015147F9318>I<7F8FF0FF8FF87F8F F00E01C00E03800E0380070380070700070700038700038600038E0001CE0001CE0000CC 0000CC0000DC0000780000780000780000700000700000700000F00000E00079E0007BC0 007F80003F00001E0000151E7F9318>I<3FFFF07FFFF07FFFF07001E07003C070078000 0F00001E00003C0000F80001F00003C0000780000F00701E00703C0070780070FFFFF0FF FFF0FFFFF014147F9318>I<0007E0001FE0007FE000780000E00000E00000E00000E000 00E00000E00000E00000E00000E00000E00000E00001E0007FC000FF8000FF80007FC000 01E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E000 007800007FE0001FE00007E013247E9F18>I<7C0000FF0000FFC00003C00000E00000E0 0000E00000E00000E00000E00000E00000E00000E00000E00000E00000F000007FC0003F E0003FE0007FC000F00000E00000E00000E00000E00000E00000E00000E00000E00000E0 0000E00000E00003C000FFC000FF00007C000013247E9F18>125 D E /Fi 45 122 df<000078007C7800FC7801FC7803C000038000078000078000078000 078000078000078000078000078000FFFC78FFFC78FFFC78078078078078078078078078 078078078078078078078078078078078078078078078078078078078078078078078078 078078078078078078152480A31A>12 D<007000E001C003C0078007000F000E001E001C 003C003C00380078007800780078007000F000F000F000F000F000F000F000F000F000F0 00F000F000F000F0007000780078007800780038003C003C001C001E000E000F00070007 8003C001C000E000700C327DA413>40 DI45 DI<00C001C007C0FFC0FFC0FBC003C003C003C003C003C003C0 03C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C0 03C0FFFFFFFFFFFF10227CA118>49 D<03F0000FFC001FFE003C1F003007807007C06003 C0E003E0C001E04001E04001E00001E00001E00001E00003C00003C0000780000780000F 00001E00003C0000780000F00001E00001C0000380000700000E00001C00003800007000 00FFFFE0FFFFE0FFFFE013227EA118>I<01F00007FC001FFF003E0F003807807003C020 03C02003C00003C00003C00003C0000780000780000F00001E0003FC0003F80003FE0000 0F000007800003C00003C00001E00001E00001E00001E00001E08001E0C003C0E003C070 07803C0F801FFF000FFC0003F00013237EA118>I<001F00001F00002F00002F00006F00 00EF0000CF0001CF0001CF00038F00038F00078F00070F000F0F000E0F001E0F003C0F00 3C0F00780F00780F00F00F00FFFFF8FFFFF8FFFFF8000F00000F00000F00000F00000F00 000F00000F00000F00000F0015217FA018>I<3FFF803FFF803FFF803C00003C00003C00 003C00003C00003C00003C00003C00003C00003CF8003FFE003FFF003F0F803E07803C03 C03803C00001E00001E00001E00001E00001E00001E00001E04003C04003C0E003C07007 807C1F003FFE000FFC0003F00013227EA018>I58 D<001F0000001F0000003F8000003F8000003B8000007B C0000073C0000071C00000F1E00000F1E00000E0E00001E0F00001E0F00001C0F00003C0 780003C078000380780007803C0007803C0007003C000F001E000F001E000FFFFE001FFF FF001FFFFF001C000F003C0007803C00078038000780780003C0780003C0700003C0F000 01E0F00001E0E00001E01B237EA220>65 D<000FF000007FFC0000FFFF0001F01F0003C0 0700078000000F0000001E0000003E0000003C0000003C00000078000000780000007800 0000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F000 00007800000078000000780000003C0000003C0000003E0000001E0000000F0000000780 008003C0038001F00F8000FFFF00007FFC00000FF00019257DA31F>67 DIII73 D77 DI80 D82 D<00FE0003FFC007FFE00F81E01E00603C00003C0000780000780000780000 7800007800007C00003C00003F00001FC0000FFC0007FF0001FF80003FC00007E00001F0 0000F00000F8000078000078000078000078000078000078C000F0E000F0F801E07E07C0 3FFF800FFF0001FC0015257EA31B>III<07E01FF83FFC381E201E 000F000F000F000F00FF07FF1FFF3E0F780FF00FF00FF00FF00FF83F7FFF3FEF1F8F1016 7E9517>97 D<01FC0007FF000FFF801F03803C0180780000780000700000F00000F00000 F00000F00000F00000F000007800007800007800003C00401F03C00FFFC007FF8001FC00 12167E9516>99 D<0003C00003C00003C00003C00003C00003C00003C00003C00003C000 03C00003C00003C00003C003E3C00FFBC01FFFC03F0FC03C07C07803C07803C0F003C0F0 03C0F003C0F003C0F003C0F003C0F003C0F003C07803C07803C03C07C03E0FC01FFFC00F FBC003E3C012237EA219>I<03F00007FC001FFE003E0F003C0780780380780380F001C0 FFFFC0FFFFC0FFFFC0F00000F00000F000007000007800007800003C00801F07800FFF80 07FF0001F80012167E9516>I<003F00FF01FF03C0038007800780078007800780078007 800780FFF8FFF8FFF8078007800780078007800780078007800780078007800780078007 8007800780078007800780102380A20F>I<01F07807FFF80FFFF81F1F001E0F003C0780 3C07803C07803C07803C07801E0F001F1F000FFE001FFC0019F0003800003800003C0000 1FFE001FFFC01FFFE03FFFF07801F07800F8F00078F00078F00078F000787800F03E03E0 1FFFC00FFF8001FC0015217F9518>III108 DII<01FC0007FF000FFF801F07C03C01E07800F07800F0700070F00078F00078F00078 F00078F00078F000787800F07800F07C01F03E03E01F07C00FFF8007FF0001FC0015167F 9518>II<03 E3C007FBC00FFFC01F0FC03C07C07C03C07803C0F803C0F003C0F003C0F003C0F003C0F0 03C0F003C0F803C07803C07803C03C07C03F0FC01FFFC00FFBC003E3C00003C00003C000 03C00003C00003C00003C00003C00003C00003C00003C012207E9519>II<07F01FFC3FFE3C0E7806780078007C003F003FF01FF80FFC01FE00 1F000F000F000FC00FF81EFFFE3FFC0FF010167F9513>I<0F000F000F000F000F000F00 FFF8FFF8FFF80F000F000F000F000F000F000F000F000F000F000F000F000F000F000F08 0F1C07FC07F803E00E1C7F9B12>IIII121 D E /Fj 10 58 df<1F00318060C04040C060C060C060 C060C060C060C060C060404060C031801F000B107F8F0F>48 D<0C003C00CC000C000C00 0C000C000C000C000C000C000C000C000C000C00FF8009107E8F0F>I<1F00618040C080 60C0600060006000C00180030006000C00102020207FC0FFC00B107F8F0F>I<1F002180 60C060C000C0008001800F00008000400060C060C060804060801F000B107F8F0F>I<03 00030007000F000B001300330023004300C300FFE003000300030003001FE00B107F8F0F >I<20803F002C002000200020002F0030802040006000600060C06080C061801F000B10 7F8F0F>I<0780184030C060C06000C000CF00F080E040C060C060C060406060C030801F 000B107F8F0F>I<40007FE07FC08080808001000200040004000C000800080018001800 1800180018000B117E900F>I<1F00318060C060C060C071803F000F00338061C0C060C0 60C060404060801F000B107F8F0F>I<1F00318060C0C040C060C060C06040E021E01E60 0060004060C0608043003E000B107F8F0F>I E /Fk 25 90 df 45 D<3078F8787005057C840D>I<000C001C00FC0F380038003800380038003800700070 007000700070007000E000E000E000E000E000E001C001C001C001C001C001C0038003C0 FFFE0F1E7C9D17>49 D<003F8000C1E00100F00200780400780400780F007C0F807C0F80 7C0F00780600780000F80000F00001E00001C0000380000700000E00001C000038000060 0000C0000180000300200600200800401000403FFFC07FFF80FFFF80161E7E9D17>I<00 7F000183C00201E00400F00700F00F00F00F01F00F01F00001E00001E00003C000038000 0700000E0000F800000E000007000007800007C00003C00007C03007C07807C0F807C0F8 07C0F00780800F00400E00201C0018780007E000141F7D9D17>I<0000600000600000E0 0001C00003C00005C0000DC00009C00011C000238000438000C380008380010380020380 040700080700180700100700200700400700FFFFF0000E00000E00000E00000E00000E00 001C00001E0001FFE0141E7E9D17>I<01803001FFE003FFC003FF0003FC000200000200 00020000040000040000040000047C000587000603800C01800801C00001C00001E00001 E00001E00001E07003C0F803C0F003C0E00380800780400700400E00201C0018700007C0 00141F7D9D17>I<0000100000001800000038000000380000007800000078000000FC00 0001BC0000013C0000033C0000023C0000063C0000043E0000081E0000081E0000101E00 00101E0000201E0000200F0000400F0000400F0000FFFF0000800F0001000F8001000780 020007800200078004000780040007800C0007C03E0007C0FF807FFC1E207E9F22>65 D<0001F808000E061800380138007000F801E0007803C0007007800030078000300F0000 301F0000301E0000303E0000203C0000007C0000007C0000007C0000007C000000F80000 00F8000000F8000000F8000000F80000007800004078000080780000803C0000803C0001 001C0002000E00020006000C000300100001C0E000003F00001D217B9F21>67 D<07FFFF00007C01E0003C00F0003C00780078003C0078003C0078001E0078001E007800 1E0078001F00F0001F00F0001F00F0001F00F0001F00F0001F00F0001F01E0001E01E000 3E01E0003E01E0003E01E0003C01E0007C03C0007803C000F003C000F003C001E003C003 C003C0078007800F0007803C00FFFFE000201F7E9E23>I<07FFFFF8007C0078003C0038 003C001800780018007800080078000800780008007800080078080800F0100000F01000 00F0100000F0300000FFF00000F0700001E0200001E0200001E0200001E0200001E00008 01E0001003C0001003C0001003C0002003C0002003C0006003C000C0078001C0078007C0 FFFFFF801D1F7E9E1F>I<07FFFFF8007C0078003C0038003C0018007800180078000800 78000800780008007800080078000800F0100000F0100000F0100000F0300000F0700000 FFF00001E0600001E0200001E0200001E0200001E0200001E0000003C0000003C0000003 C0000003C0000003C0000003C000000780000007C00000FFFE00001D1F7E9E1E>I<0001 FC04000F030C003C009C0070007C00E0003C01C0003803800018078000180F0000181F00 00181E0000183E0000103C0000007C0000007C0000007C0000007C000000F8000000F800 0000F8007FFCF80003E0780001E0780001E0780003C0780003C03C0003C03C0003C01C00 03C00E0007C007000B800380118001E06080003F80001E217B9F24>I<07FFC7FFC0007C 00F800003C007800003C007800007800F000007800F000007800F000007800F000007800 F000007800F00000F001E00000F001E00000F001E00000F001E00000FFFFE00000F001E0 0001E003C00001E003C00001E003C00001E003C00001E003C00001E003C00003C0078000 03C007800003C007800003C007800003C007800003C007800007800F000007C00F8000FF F8FFF800221F7E9E22>I<07FFE0007C00003C00003C0000780000780000780000780000 780000780000F00000F00000F00000F00000F00000F00001E00001E00001E00001E00001 E00001E00003C00003C00003C00003C00003C00003C00007800007C000FFFC00131F7F9E 10>I<07FFF000007E0000003C0000003C00000078000000780000007800000078000000 7800000078000000F0000000F0000000F0000000F0000000F0000000F0000001E0000001 E0000001E0000001E0000001E0008001E0010003C0010003C0010003C0030003C0020003 C0060003C0060007801E0007807C00FFFFFC00191F7E9E1C>76 D<07FC0000FFC0007C00 00F800003C00017800003C00017800004E0002F000004E0002F000004E0004F000004E00 04F000004E0008F000004E0008F00000870011E00000870011E00000870021E000008700 21E00000870041E00000838041E00001038083C00001038083C00001038103C000010382 03C0000101C203C0000101C403C0000201C40780000201C80780000201C80780000201D0 0780000200F00780000600E00780000600E00F00000F00C00F8000FFE0C1FFF8002A1F7E 9E2A>I<07FC01FFC0003E003E00003E001800003E001800004F001000004F0010000047 80100000478010000043C010000043C010000083C020000081E020000081E020000080F0 20000080F020000080782000010078400001007C400001003C400001003C400001001E40 0001001E400002000F800002000F800002000F8000020007800002000780000600038000 06000300000F00010000FFE0010000221F7E9E22>I<0003F800001E0E000038070000E0 038001C001C003C001E0078000E00F0000F00F0000F01E0000F01E0000F83E0000F83C00 00F87C0000F87C0000F87C0000F87C0000F8F80001F0F80001F0F80001F0F80001F0F800 03E0780003E0780003C0780007C07C0007803C000F003C001E001E001C000E0038000700 F00003C3C00000FE00001D217B9F23>I<07FFFF00007C03C0003C01E0003C00F0007800 F0007800F8007800F8007800F8007800F8007800F000F001F000F001E000F003C000F007 8000F00F0000FFF80001E0000001E0000001E0000001E0000001E0000001E0000003C000 0003C0000003C0000003C0000003C0000003C000000780000007C00000FFFC00001D1F7E 9E1F>I<07FFFC00007C0700003C03C0003C01E0007801E0007801F0007801F0007801F0 007801F0007801E000F003E000F003C000F0078000F00F0000F03C0000FFF00001E03000 01E0380001E01C0001E01C0001E01C0001E01E0003C03E0003C03E0003C03E0003C03E00 03C03E0003C03E0207803E0407C01F04FFFC0F18000003E01F207E9E21>82 D<003F040060CC01803C03801C03001C0700180600080E00080E00080E00080E00000F00 000F80000FE00007FE0003FF8001FFC0007FE00007E00001E00000E00000F00000F04000 E04000E04000E04000E06000C0600180E00380F80300C60C0081F80016217D9F19>I<3F FFFFF03C0780F03007803060078030400F0010400F0010C00F0010800F0010800F001080 0F0010001E0000001E0000001E0000001E0000001E0000001E0000003C0000003C000000 3C0000003C0000003C0000003C0000007800000078000000780000007800000078000000 78000000F0000001F800007FFFE0001C1F7A9E21>I<03FFC0FFC0007F007E00003E0038 00001E003000001E002000000F004000000F008000000F81000000078200000007C60000 0003C400000003E800000001F000000001F000000000F000000000F800000000F8000000 017C000000023C000000043C0000000C1E000000081E000000101F000000200F00000040 0F800000C0078000008007C000010003C000070003E0001F8007E000FFE01FFE00221F7F 9E22>88 DI E /Fl 33 122 df45 DI<0018 0000380000F80007F800FFF800FFF800F8F80000F80000F80000F80000F80000F80000F8 0000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F8 0000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F8 007FFFF07FFFF07FFFF014287BA71E>49 D<00FE0003FFC007FFE00FFFF01F03F83C00FC 38007E78003E70003EF0001FF0001F60001F20001F00001F00001F00001F00003E00003E 00007C00007C0000F80001F00001E00003C0000780000F00001E00003C0000780000F000 01E00003C0000780000F00001E00003C00007FFFFF7FFFFF7FFFFF7FFFFF18287EA71E> I<007F000001FFC00007FFF0000FFFF8001FC1F8003E007C003C003E0078003E0038003E 0010003E0000003E0000003E0000003C0000007C000000FC000001F8000007F00000FFE0 0000FFC00000FFE00000FFF0000001FC0000007C0000003E0000001F0000001F0000000F 8000000F8000000F8000000F8000000F8040000F8060001F00F0001F00F8003F007E007E 003F81FC001FFFF8000FFFF00003FFE000007F000019297EA71E>I<0003F0000007F000 0005F000000DF000000DF000001DF0000039F0000039F0000079F0000079F00000F1F000 00F1F00001E1F00003E1F00003E1F00007C1F00007C1F0000F81F0000F81F0001F01F000 1F01F0003E01F0007C01F0007C01F000F801F000FFFFFF80FFFFFF80FFFFFF80FFFFFF80 0001F0000001F0000001F0000001F0000001F0000001F0000001F0000001F0000001F000 0001F00019277EA61E>I<3FFFFC3FFFFC3FFFFC3FFFFC3E00003E00003E00003E00003E 00003E00003E00003E00003E00003E00003E3F003EFFC03FFFE03FFFF03FE1F83F807C3F 003E3E003E00003E00001F00001F00001F00001F00001F00001F00001F20001F60003E70 003EF8007C7C00FC3F03F81FFFF00FFFE007FF8000FE0018287EA61E>I<0001F0000000 03F800000003F800000007FC00000007BC00000007BC0000000F3E0000000F1E0000000F 1E0000001F1F0000001E1F0000001E0F0000003E0F8000003C0F8000003C078000007C07 C000007807C00000F803E00000F803E00000F003E00001F001F00001F001F00001E001F0 0003E000F80003E000F80003C000F80007FFFFFC0007FFFFFC000FFFFFFE000F80003E00 0F80003E001F00003F001F00001F001E00001F003E00000F803E00000F803C00000F807C 000007C07C000007C078000007C0F8000003E0F8000003E0232A7EA928>65 D68 DI73 D77 D80 D84 D<01FE000FFF803FFFC03FFFE03C03F03001F0 0001F80000F80000F80000F80000F80000F8007FF807FFF81FFFF83FE0F87F00F8FC00F8 F800F8F800F8F800F8FC01F87E07F87FFFF83FFFF81FFCF80FE0F8151B7E9A1D>97 D<007FC001FFF007FFFC0FFFFC1FC07C1F00083E00007C00007C00007C0000F80000F800 00F80000F80000F80000F80000F800007C00007C00007E00003E00001F000C1FC07C0FFF FC07FFFC01FFF0007F80161B7E9A1B>99 D<00003E00003E00003E00003E00003E00003E 00003E00003E00003E00003E00003E00003E00003E00003E00003E00FC3E03FF3E07FFFE 0FFFFE1FC1FE3F007E3E003E7C003E7C003EFC003EF8003EF8003EF8003EF8003EF8003E F8003EF8003EFC003E7C003E7C003E3E007E3F00FE1FC1FE0FFFFE07FFBE03FF3E00FC3E 172A7EA91F>I<007E0003FF8007FFC00FFFE01F83F03F00F03E00787C00787C00387800 3CFFFFFCFFFFFCFFFFFCFFFFFCF80000F80000F800007800007C00007C00003E00003F00 0C1FC07C0FFFFC07FFFC01FFF0007F80161B7E9A1B>I<001FC0007FC000FFC001FFC003 F00003E00007C00007C00007C00007C00007C00007C00007C00007C00007C000FFFE00FF FE00FFFE0007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007 C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007 C00007C000122A7FA912>I<00F8078003FE7FC00FFFFFC01FFFFFC01F07C0003E03E000 3E03E0007C01F0007C01F0007C01F0007C01F0007C01F0007C01F0003E03E0003E03E000 1F07C0001FFFC0003FFF80003BFE000038F8000078000000780000003C0000003FFFC000 3FFFF8001FFFFC001FFFFE003FFFFF007C007F00F8001F80F8000F80F8000F80F8000F80 FC001F807E003F003F80FE003FFFFE000FFFF80007FFF00000FF80001A287E9A1E>II I108 DII<007F000001FF C00007FFF0000FFFF8001FC1FC003F007E003E003E007C001F007C001F0078000F00F800 0F80F8000F80F8000F80F8000F80F8000F80F8000F80F8000F807C001F007C001F007E00 3F003E003E003F007E001FC1FC000FFFF80007FFF00001FFC000007F0000191B7E9A1E> II114 D<03FC001FFF803FFFC07FFFC07C07C0F80080F80000F80000F80000FC 00007F80007FF8003FFE001FFF0007FF8000FFC0000FE00007E00003E00003E04003E0E0 07E0FC0FC0FFFFC07FFF801FFE0003F800131B7E9A17>I<07C00007C00007C00007C000 07C00007C00007C000FFFFC0FFFFC0FFFFC007C00007C00007C00007C00007C00007C000 07C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C000 07C04007E1C003FFE003FFE001FF8000FC0013227FA116>II<7C000FC03E001F803F001F001F803E000F807C0007C0FC0003E0F80001F1F0 0001FBE00000FFC000007FC000003F8000001F0000001F0000003F8000007FC00000FBC0 0000F3E00001F1F00003E0F80007C07C000F807C000F803E001F001F003E000F807E000F C0FC0007E01B1B809A1C>120 DI E /Fm 53 122 df<007000E001C00380078007000E001E001E003C003C003C0078007800 780078007000F000F000F000F000F000F000F000F000F000F000F000F000700078007800 780078003C003C003C001E001E000E0007000780038001C000E000700C2E7EA112>40 DI<7878787818303060 60E0050A7D830D>44 DII<000100 030003000600060006000C000C000C00180018001800300030003000600060006000C000 C000C00180018001800300030003000600060006000C000C000C00180018001800300030 003000600060006000C000C000C000102D7DA117>I<001F0000001F0000003F8000003B 8000003B8000007BC0000073C0000071C00000F1E00000E1E00000E0E00001E0F00001E0 F00001C0F00003C0780003C078000380780007803C0007803C0007003C000FFFFE000FFF FE000FFFFE001E000F001E000F003C000F803C0007803C000780780007C0780003C07800 03C0F00003E01B207F9F1E>65 D<001FC000FFF801FFFC03E03C07800C0F00001E00003E 00003C00007C0000780000780000780000F00000F00000F00000F00000F00000F00000F0 0000F000007800007800007800007C00003C00003E00001E00000F000207800E03E03E01 FFFC00FFF0001FC017227DA01D>67 DII I<001FE000FFF801FFFE03E03E07800E0F00001E00003E00003C00007C00007800007800 00780000F00000F00000F00000F00000F00000F00000F003FEF003FE7803FE78001E7800 1E7C001E3C001E3E001E1E001E0F001E07801E03E03E01FFFE00FFF8001FC017227DA01E >III< F0003EF0007CF000F8F001F0F003E0F007C0F00780F00F00F01F00F03E00F07C00F0F800 F1F000F3F800F3F800F7FC00FFBC00FF1E00FE1F00FC0F00F80780F00780F003C0F003E0 F001E0F000F0F000F8F00078F0003CF0003CF0001EF0001F18207B9F20>75 DIII<003F000000FFC00003FFF00007E1F8000F80 7C001F003E001E001E003C000F003C000F00780007807800078078000780F00003C0F000 03C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F80007C07800 078078000780780007803C000F003C000F001E001E001F003E000F807C0007E1F80003FF F00000FFC000003F00001A227DA021>II82 D<01FC0007FF800FFFC01F03C03C00C03C000078000078000078000078 00007800007C00003C00003F00001FE0000FFC0007FE0001FF00003F800007C00003C000 03E00001E00001E00001E00001E00001E00001C0C003C0F007C0FC0F807FFF001FFE0003 F80013227EA019>II< F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0 F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0 F000F07801E07801E03C03C03C03C01F0F800FFF0007FE0001F80014217B9F1F>III89 DI<07 E03FF87FFC701E401F000F000F000F003F07FF1FFF7E0FF80FF00FF00FF00FF83F7FFF3F EF1F8F10147E9316>97 DI<03F00FFC1FFE3E0E3C0278007800F000F000F000F000F000F00078007800 3C013E0F1FFF0FFE03F010147E9314>I<00078000078000078000078000078000078000 078000078000078000078000078000078007C7800FF7801FFF803E1F807C0780780780F8 0780F00780F00780F00780F00780F00780F00780F00780780780780F803E1F801FFF800F F78007C78011207E9F17>I<03F0000FFC001FFE003E1F003C0700780700700380FFFF80 FFFF80FFFF80F00000F00000F000007000007800003C01003E07001FFF0007FE0001F800 11147F9314>I<007E01FE03FE078007000F000F000F000F000F000F000F00FFF0FFF0FF F00F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F 20809F0E>I<03E0F00FFFF01FFFF03E3E003C1E00780F00780F00780F00780F00780F00 3C1E003E3E001FFC003FF80033E0003000003800003FFE003FFF801FFFC03FFFE07803F0 F000F0F000F0F000F0F801F07E07E03FFFC00FFF0003FC00141E7F9317>III<01E001 E001E001E00000000000000000000000000000000001E001E001E001E001E001E001E001 E001E001E001E001E001E001E001E001E001E001E001E001E001E001E001E001E001E0C3 C0FFC0FF803F000B29839F0C>IIIII<01F80007FE001FFF803F0FC03C03C07801E07801E0F000F0F000F0F000F0F000F0F0 00F0F000F07801E07801E03C03C03F0FC01FFF8007FE0001F80014147F9317>II<03C7800FF7801FFF803E1F807C0F807807 80780780F00780F00780F00780F00780F00780F00780F807807807807C0F803E1F801FFF 800FF78007C780000780000780000780000780000780000780000780000780000780111D 7E9317>II<07F01FFC3FFC780C7800780078007C003FC01FF0 0FF803F8007C003C003CC03CF07CFFF87FF00FC00E147F9311>I<1E001E001E001E001E 001E00FFF0FFF0FFF01E001E001E001E001E001E001E001E001E001E001E001E001E001E 201FF00FF007C00C1A7F9910>IIII<78 01E07C03C03E07801E0F000F0F00079E0003FC0003F80001F80000F00001F00001F80003 FC00079E000F0F000E0F001E07803C03C07801E0F801F01414809315>II E /Fn 1 16 df<03C00FF01FF83FFC7FFE7FFEFF FFFFFFFFFFFFFFFFFFFFFF7FFE7FFE3FFC1FF80FF003C010127D9317>15 D E /Fo 18 122 df66 D68 D80 D<0007FFFC000000007FFFFFC0000001FFFFFFF8000003FFFFFFFE000007FE00 1FFF000007FF0003FFC0000FFF8001FFE0000FFF8000FFF0000FFF80007FF0000FFF8000 7FF8000FFF80007FF80007FF00003FFC0007FF00003FFC0003FE00003FFC0000F800003F FC00000000003FFC00000000003FFC00000000003FFC00000000003FFC00000007FFFFFC 000000FFFFFFFC000007FFFFFFFC00003FFFE03FFC0000FFFE003FFC0003FFF0003FFC00 07FFC0003FFC000FFF00003FFC001FFE00003FFC003FFC00003FFC007FF800003FFC007F F800003FFC00FFF000003FFC00FFF000003FFC00FFF000003FFC00FFF000003FFC00FFF0 00003FFC00FFF000007FFC007FF80000FFFC007FF80001EFFC003FFC0003EFFC003FFF00 07CFFF000FFFC03F8FFFF807FFFFFF07FFFC01FFFFFC03FFFC007FFFF001FFFC0003FF80 007FF8362E7DAD3A>97 D<007FC00000000000FFFFC00000000000FFFFC00000000000FF FFC00000000000FFFFC00000000000FFFFC0000000000003FFC0000000000001FFC00000 00000001FFC0000000000001FFC0000000000001FFC0000000000001FFC0000000000001 FFC0000000000001FFC0000000000001FFC0000000000001FFC0000000000001FFC00000 00000001FFC0000000000001FFC0000000000001FFC0000000000001FFC0000000000001 FFC0000000000001FFC0000000000001FFC0000000000001FFC0000000000001FFC00000 00000001FFC00FFC00000001FFC07FFFC0000001FFC3FFFFF0000001FFCFFFFFFC000001 FFDFF00FFF000001FFFF8003FF800001FFFE0001FFC00001FFF800007FE00001FFF00000 7FF00001FFE000003FF80001FFE000001FFC0001FFE000001FFC0001FFE000001FFE0001 FFE000001FFE0001FFE000000FFF0001FFE000000FFF0001FFE000000FFF0001FFE00000 0FFF0001FFE000000FFF8001FFE000000FFF8001FFE000000FFF8001FFE000000FFF8001 FFE000000FFF8001FFE000000FFF8001FFE000000FFF8001FFE000000FFF8001FFE00000 0FFF8001FFE000000FFF8001FFE000000FFF0001FFE000000FFF0001FFE000000FFF0001 FFE000001FFE0001FFE000001FFE0001FFE000001FFC0001FFE000001FFC0001FFE00000 3FF80001FFF000003FF80001FFF800007FF00001FFFC0000FFE00001FFFE0001FFC00001 FFBF0007FF800001FF1FE01FFE000001FE0FFFFFFC000001FC03FFFFF0000001F800FFFF 80000001F0001FF800000039487CC742>I<00001FFFC0000000FFFFF8000007FFFFFE00 001FFFFFFF80007FFC00FFC000FFE001FFC001FFC003FFE003FF8003FFE007FF0003FFE0 0FFE0003FFE00FFE0003FFE01FFC0001FFC01FFC0001FFC03FFC0000FF803FFC00003E00 7FF8000000007FF8000000007FF800000000FFF800000000FFF800000000FFF800000000 FFF800000000FFF800000000FFF800000000FFF800000000FFF800000000FFF800000000 FFF8000000007FF8000000007FF8000000007FFC000000003FFC000000003FFC00000000 1FFC000000F81FFE000000F80FFE000000F80FFF000001F007FF800003F003FFC00007E0 01FFE0000FC000FFF0001F80007FFE00FF00001FFFFFFE000007FFFFF8000000FFFFE000 00001FFE00002D2E7CAD35>I<00001FFE00000001FFFFE0000007FFFFF800001FFFFFFE 00007FFC07FF0000FFE001FF8001FFC0007FC003FF80003FE007FF00003FF00FFE00001F F01FFE00000FF81FFC00000FF83FFC00000FFC3FFC000007FC7FFC000007FC7FF8000007 FC7FF8000007FE7FF8000007FEFFF8000007FEFFF8000007FEFFFFFFFFFFFEFFFFFFFFFF FEFFFFFFFFFFFEFFFFFFFFFFFCFFF800000000FFF800000000FFF800000000FFF8000000 007FF8000000007FF8000000007FFC000000003FFC000000003FFC000000003FFC000000 1C1FFE0000003E0FFE0000003E07FF0000007E07FF000000FC03FF800001F801FFC00003 F0007FF0001FE0003FFE00FFC0001FFFFFFF800007FFFFFE000000FFFFF80000000FFF80 002F2E7DAD36>101 D<00000000001F8000007FF000FFE00007FFFF03FFF0001FFFFFC7 FFF0007FFFFFFFC7F800FFE03FFE0FF801FF800FFC0FF803FF0007FE0FF807FE0003FF07 F007FE0003FF07F00FFC0001FF81C00FFC0001FF80000FFC0001FF80001FFC0001FFC000 1FFC0001FFC0001FFC0001FFC0001FFC0001FFC0001FFC0001FFC0001FFC0001FFC0000F FC0001FF80000FFC0001FF80000FFC0001FF800007FE0003FF000007FE0003FF000003FF 0007FE000001FF800FFC000000FFE03FF8000001FFFFFFF0000001DFFFFFC0000003C7FF FF00000003C07FF000000007C0000000000007C0000000000007C0000000000007C00000 00000007E0000000000007F0000000000007F8000000000007FFFFFFF0000007FFFFFFFF 000003FFFFFFFFE00003FFFFFFFFF80001FFFFFFFFFE0001FFFFFFFFFF0000FFFFFFFFFF 80007FFFFFFFFF8003FFFFFFFFFFC00FFFFFFFFFFFC01FF800001FFFE03FE0000001FFE0 7FC00000007FF07FC00000003FF0FF800000001FF0FF800000001FF0FF800000001FF0FF 800000001FF0FF800000001FF07FC00000003FE07FC00000003FE03FE00000007FC03FF0 000000FFC01FFC000003FF800FFF00000FFF0003FFF000FFFC0000FFFFFFFFF000003FFF FFFFC0000007FFFFFE000000003FFFC0000035447DAE3B>103 D<007FC00000000000FF FFC00000000000FFFFC00000000000FFFFC00000000000FFFFC00000000000FFFFC00000 00000003FFC0000000000001FFC0000000000001FFC0000000000001FFC0000000000001 FFC0000000000001FFC0000000000001FFC0000000000001FFC0000000000001FFC00000 00000001FFC0000000000001FFC0000000000001FFC0000000000001FFC0000000000001 FFC0000000000001FFC0000000000001FFC0000000000001FFC0000000000001FFC00000 00000001FFC0000000000001FFC0000000000001FFC001FFC0000001FFC00FFFF8000001 FFC03FFFFE000001FFC0FFFFFF000001FFC1FC07FF800001FFC3E003FFC00001FFC7C001 FFC00001FFCF0001FFE00001FFDE0000FFE00001FFDC0000FFE00001FFFC0000FFF00001 FFF80000FFF00001FFF00000FFF00001FFF00000FFF00001FFF00000FFF00001FFE00000 FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001 FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000 FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001 FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000 FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001 FFE00000FFF00001FFE00000FFF00001FFE00000FFF000FFFFFFC07FFFFFE0FFFFFFC07F FFFFE0FFFFFFC07FFFFFE0FFFFFFC07FFFFFE0FFFFFFC07FFFFFE03B487CC742>I<00FC 0001FE0003FF0007FF800FFFC01FFFE01FFFE01FFFE01FFFE01FFFE01FFFE00FFFC007FF 8003FF0001FE0000FC000000000000000000000000000000000000000000000000000000 00000000000000007FC0FFFFC0FFFFC0FFFFC0FFFFC0FFFFC003FFC001FFC001FFC001FF C001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FF C001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FF C001FFC001FFC001FFC001FFC001FFC001FFC001FFC0FFFFFFFFFFFFFFFFFFFFFFFFFFFF FF18497CC820>I<007FC000FFFFC000FFFFC000FFFFC000FFFFC000FFFFC00003FFC000 01FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC000 01FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC000 01FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC000 01FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC000 01FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC000 01FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC000 01FFC00001FFC00001FFC00001FFC00001FFC00001FFC000FFFFFF80FFFFFF80FFFFFF80 FFFFFF80FFFFFF8019487CC720>108 D<007FC001FFC00000FFE00000FFFFC00FFFF800 07FFFC0000FFFFC03FFFFE001FFFFF0000FFFFC0FFFFFF007FFFFF8000FFFFC1FC07FF80 FE03FFC000FFFFC3E003FFC1F001FFE00003FFC7C001FFC3E000FFE00001FFCF0001FFE7 8000FFF00001FFDE0000FFEF00007FF00001FFDC0000FFEE00007FF00001FFFC0000FFFE 00007FF80001FFF80000FFFC00007FF80001FFF00000FFF800007FF80001FFF00000FFF8 00007FF80001FFF00000FFF800007FF80001FFE00000FFF000007FF80001FFE00000FFF0 00007FF80001FFE00000FFF000007FF80001FFE00000FFF000007FF80001FFE00000FFF0 00007FF80001FFE00000FFF000007FF80001FFE00000FFF000007FF80001FFE00000FFF0 00007FF80001FFE00000FFF000007FF80001FFE00000FFF000007FF80001FFE00000FFF0 00007FF80001FFE00000FFF000007FF80001FFE00000FFF000007FF80001FFE00000FFF0 00007FF80001FFE00000FFF000007FF80001FFE00000FFF000007FF80001FFE00000FFF0 00007FF80001FFE00000FFF000007FF80001FFE00000FFF000007FF80001FFE00000FFF0 00007FF80001FFE00000FFF000007FF80001FFE00000FFF000007FF80001FFE00000FFF0 00007FF80001FFE00000FFF000007FF80001FFE00000FFF000007FF80001FFE00000FFF0 00007FF800FFFFFFC07FFFFFE03FFFFFF0FFFFFFC07FFFFFE03FFFFFF0FFFFFFC07FFFFF E03FFFFFF0FFFFFFC07FFFFFE03FFFFFF0FFFFFFC07FFFFFE03FFFFFF05C2E7CAD63>I< 007FC001FFC00000FFFFC00FFFF80000FFFFC03FFFFE0000FFFFC0FFFFFF0000FFFFC1FC 07FF8000FFFFC3E003FFC00003FFC7C001FFC00001FFCF0001FFE00001FFDE0000FFE000 01FFDC0000FFE00001FFFC0000FFF00001FFF80000FFF00001FFF00000FFF00001FFF000 00FFF00001FFF00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF000 01FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE000 00FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF000 01FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE000 00FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF000 01FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE000 00FFF000FFFFFFC07FFFFFE0FFFFFFC07FFFFFE0FFFFFFC07FFFFFE0FFFFFFC07FFFFFE0 FFFFFFC07FFFFFE03B2E7CAD42>I<00000FFF0000000000FFFFF000000007FFFFFE0000 001FFFFFFF8000003FFC03FFC00000FFE0007FF00001FF80001FF80003FF00000FFC0007 FE000007FE000FFE000007FF000FFC000003FF001FFC000003FF803FFC000003FFC03FF8 000001FFC03FF8000001FFC07FF8000001FFE07FF8000001FFE07FF8000001FFE0FFF800 0001FFF0FFF8000001FFF0FFF8000001FFF0FFF8000001FFF0FFF8000001FFF0FFF80000 01FFF0FFF8000001FFF0FFF8000001FFF0FFF8000001FFF0FFF8000001FFF07FF8000001 FFE07FF8000001FFE07FF8000001FFE07FF8000001FFE03FFC000003FFC03FFC000003FF C01FFC000003FF801FFE000007FF800FFE000007FF0007FF00000FFE0003FF80001FFC00 01FFC0003FF80000FFE0007FF000007FFC03FFE000001FFFFFFF80000007FFFFFE000000 00FFFFF0000000000FFF000000342E7DAD3B>I<007FC00FFC000000FFFFC07FFFC00000 FFFFC3FFFFF00000FFFFCFFFFFFC0000FFFFDFF01FFF0000FFFFFF8007FF800003FFFE00 01FFC00001FFF80000FFE00001FFF00000FFF00001FFE000007FF80001FFE000003FFC00 01FFE000003FFC0001FFE000003FFE0001FFE000001FFE0001FFE000001FFF0001FFE000 001FFF0001FFE000001FFF0001FFE000000FFF0001FFE000000FFF8001FFE000000FFF80 01FFE000000FFF8001FFE000000FFF8001FFE000000FFF8001FFE000000FFF8001FFE000 000FFF8001FFE000000FFF8001FFE000000FFF8001FFE000000FFF8001FFE000000FFF00 01FFE000001FFF0001FFE000001FFF0001FFE000001FFE0001FFE000001FFE0001FFE000 003FFC0001FFE000003FFC0001FFE000007FF80001FFF000007FF80001FFF80000FFF000 01FFFC0001FFE00001FFFE0003FFC00001FFFF0007FF800001FFFFE03FFE000001FFEFFF FFFC000001FFE3FFFFF0000001FFE0FFFF80000001FFE01FF800000001FFE00000000000 01FFE0000000000001FFE0000000000001FFE0000000000001FFE0000000000001FFE000 0000000001FFE0000000000001FFE0000000000001FFE0000000000001FFE00000000000 01FFE0000000000001FFE0000000000001FFE0000000000001FFE0000000000001FFE000 00000000FFFFFFC000000000FFFFFFC000000000FFFFFFC000000000FFFFFFC000000000 FFFFFFC00000000039427CAD42>I<00FF803F8000FFFF80FFF000FFFF83FFFC00FFFF87 FFFE00FFFF8FC3FF00FFFF8F07FF0003FF9E0FFF8001FFBC0FFF8001FFB80FFF8001FFF8 0FFF8001FFF00FFF8001FFF007FF0001FFF007FF0001FFE003FE0001FFE000F80001FFE0 00000001FFE000000001FFC000000001FFC000000001FFC000000001FFC000000001FFC0 00000001FFC000000001FFC000000001FFC000000001FFC000000001FFC000000001FFC0 00000001FFC000000001FFC000000001FFC000000001FFC000000001FFC000000001FFC0 00000001FFC000000001FFC000000001FFC000000001FFC000000001FFC000000001FFC0 00000001FFC0000000FFFFFFE00000FFFFFFE00000FFFFFFE00000FFFFFFE00000FFFFFF E00000292E7CAD31>114 D<000FFF00E0007FFFF3E001FFFFFFE007FFFFFFE00FF800FF E01FC0001FE03F80000FE03F000007E07F000003E07F000003E0FF000003E0FF000003E0 FF800003E0FFC0000000FFF0000000FFFE000000FFFFF800007FFFFFC0007FFFFFF0003F FFFFFC001FFFFFFF000FFFFFFF8007FFFFFFC003FFFFFFE000FFFFFFF0003FFFFFF00003 FFFFF800001FFFF8000000FFFC0000001FFC7800000FFCF8000007FCF8000003FCFC0000 03FCFC000003FCFE000003F8FE000003F8FF000003F8FF800007F0FFC0000FF0FFF0001F E0FFFC00FFC0FFFFFFFF80FC7FFFFE00F81FFFF800E003FF8000262E7CAD2F>I<7FFFFF C000FFFF807FFFFFC000FFFF807FFFFFC000FFFF807FFFFFC000FFFF807FFFFFC000FFFF 8000FFF000000FE00000FFF800000FC00000FFF800000FC000007FFC00000F8000007FFC 00001F8000003FFC00001F0000003FFE00003F0000001FFE00003E0000001FFF00007E00 00000FFF00007C0000000FFF8000FC00000007FF8000F800000007FFC001F800000003FF C001F000000003FFE003F000000003FFE003F000000001FFF003E000000001FFF007E000 000000FFF007C000000000FFF80FC0000000007FF80F80000000007FFC1F80000000003F FC1F00000000003FFE3F00000000001FFE3E00000000001FFF7E00000000000FFF7C0000 0000000FFFFC00000000000FFFFC000000000007FFF8000000000007FFF8000000000003 FFF0000000000003FFF0000000000001FFE0000000000001FFE0000000000000FFC00000 00000000FFC00000000000007F800000000000007F800000000000003F00000000000000 3F000000000000003F000000000000003E000000000000007E000000000000007C000000 00000000FC000000001F8000F8000000003FC001F8000000007FE001F000000000FFF003 F000000000FFF003E000000000FFF007E000000000FFF00FC000000000FFF01F80000000 00FFF03F80000000007FE07F00000000007F43FE00000000003FFFF800000000001FFFF0 000000000007FFC0000000000001FE00000000000039427EAD3F>121 D E /Fp 8 117 df<00001E000000003E00000000FE00000003FE0000003FFE0000FFFF FE0000FFFFFE0000FFFFFE0000FFCFFE0000000FFE0000000FFE0000000FFE0000000FFE 0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE00 00000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000 000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE000000 0FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000F FE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE 0000000FFE0000000FFE0000000FFE00007FFFFFFFC07FFFFFFFC07FFFFFFFC07FFFFFFF C0223879B731>49 D<0000001FFF000030000001FFFFE000F000000FFFFFFC01F000007F FFFFFE03F00001FFFE007F87F00003FFE0000FCFF0000FFF000003FFF0001FFC000001FF F0003FF80000007FF0007FF00000003FF000FFC00000003FF001FFC00000001FF003FF80 0000000FF007FF000000000FF00FFF0000000007F00FFE0000000007F01FFE0000000003 F01FFE0000000003F03FFC0000000003F03FFC0000000001F03FFC0000000001F07FFC00 00000001F07FF80000000001F07FF80000000000007FF8000000000000FFF80000000000 00FFF8000000000000FFF8000000000000FFF8000000000000FFF8000000000000FFF800 0000000000FFF8000000000000FFF8000000000000FFF8000000000000FFF80000000000 00FFF80000000000007FF80000000000007FF80000000000007FF80000000000007FFC00 00000000F03FFC0000000000F03FFC0000000000F03FFC0000000000F01FFE0000000000 F01FFE0000000001E00FFE0000000001E00FFF0000000001E007FF0000000003C003FF80 00000003C001FFC0000000078000FFE00000000F00007FF00000001F00003FF80000003E 00001FFC0000007C00000FFF000001F8000003FFE00007F0000001FFFE003FC00000007F FFFFFF000000000FFFFFFC0000000001FFFFF000000000001FFF0000003C3D7BBB47>67 D<001FFF00000001FFFFF0000003FFFFFC000007F007FE00000FF801FF00001FFC00FF80 001FFC007FC0001FFC007FE0001FFC003FE0000FF8003FF0000FF8003FF00007F0003FF0 0001C0003FF0000000003FF0000000003FF0000000003FF0000000FFFFF000000FFFFFF0 00007FF83FF00001FF803FF00007FE003FF0000FF8003FF0001FF0003FF0003FE0003FF0 007FE0003FF0007FE0003FF000FFC0003FF000FFC0003FF000FFC0003FF000FFC0003FF0 00FFC0007FF0007FE0007FF0007FE000DFF0003FF0039FF8001FFC0F0FFFF007FFFE0FFF F001FFFC07FFF0003FE000FFF02C267DA530>97 D<0001FFC000000FFFF800003FFFFE00 00FF80FF0001FE003F8007FC001FC00FF8000FE00FF8000FF01FF00007F03FF00007F83F F00007F87FE00007F87FE00003FC7FE00003FC7FE00003FCFFE00003FCFFFFFFFFFCFFFF FFFFFCFFFFFFFFFCFFE0000000FFE0000000FFE0000000FFE00000007FE00000007FE000 00007FE00000003FE00000003FF000003C1FF000003C1FF000003C0FF800007807FC0000 F803FE0001F001FF0007E000FFC03FC0003FFFFF000007FFFC000000FFE00026267DA52D >101 D<00FF00000000FFFF00000000FFFF00000000FFFF00000000FFFF0000000007FF 0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF 0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF 0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF007FC00003FF 01FFF80003FF07FFFC0003FF0F03FE0003FF1C01FF0003FF3001FF8003FF6000FF8003FF E000FFC003FFC000FFC003FF8000FFC003FF8000FFC003FF8000FFC003FF0000FFC003FF 0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF 0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF 0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF 0000FFC003FF0000FFC003FF0000FFC0FFFFFC3FFFFFFFFFFC3FFFFFFFFFFC3FFFFFFFFF FC3FFFFF303C7CBB37>104 D<00FF01FF8000FFFF0FFFF000FFFF3FFFFC00FFFFFE03FF 00FFFFF000FF8003FFC0007FC003FF80003FE003FF00003FF003FF00001FF803FF00001F FC03FF00000FFC03FF00000FFE03FF00000FFE03FF000007FE03FF000007FF03FF000007 FF03FF000007FF03FF000007FF03FF000007FF03FF000007FF03FF000007FF03FF000007 FF03FF000007FF03FF000007FE03FF000007FE03FF00000FFE03FF00000FFC03FF00000F FC03FF00001FF803FF00001FF803FF00003FF003FF80003FE003FFC0007FC003FFF001FF 8003FFFC07FF0003FF3FFFFC0003FF0FFFF00003FF01FF000003FF0000000003FF000000 0003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF000000 0003FF0000000003FF0000000003FF0000000003FF0000000003FF00000000FFFFFC0000 00FFFFFC000000FFFFFC000000FFFFFC00000030377DA537>112 D<00FE03F000FFFE0FFE00FFFE1FFF00FFFE3C3F80FFFE707FC007FE60FFE003FEE0FFE0 03FEC0FFE003FFC0FFE003FF807FC003FF807FC003FF803F8003FF800E0003FF00000003 FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF 00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00 000003FF00000003FF00000003FF00000003FF00000003FF000000FFFFFE0000FFFFFE00 00FFFFFE0000FFFFFE000023267DA529>114 D<00078000000780000007800000078000 00078000000F8000000F8000000F8000000F8000001F8000001F8000003F8000003F8000 007F800000FF800001FF800007FF80001FFFFFF0FFFFFFF0FFFFFFF0FFFFFFF001FF8000 01FF800001FF800001FF800001FF800001FF800001FF800001FF800001FF800001FF8000 01FF800001FF800001FF800001FF800001FF800001FF800001FF800001FF800001FF8000 01FF803C01FF803C01FF803C01FF803C01FF803C01FF803C01FF803C01FF803C00FF8078 00FFC078007FC070003FE0E0001FFFC00007FF800001FF001E377EB626>116 D E /Fq 77 123 df<001F83E000F06E3001C078780380F8780300F03007007000070070 000700700007007000070070000700700007007000FFFFFF800700700007007000070070 000700700007007000070070000700700007007000070070000700700007007000070070 000700700007007000070070000700700007007000070070007FE3FF001D20809F1B>11 D<003F0000E0C001C0C00381E00701E00701E00700000700000700000700000700000700 00FFFFE00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700 E00700E00700E00700E00700E00700E00700E00700E07FC3FE1720809F19>I<003FE000 E0E001C1E00381E00700E00700E00700E00700E00700E00700E00700E00700E0FFFFE007 00E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E007 00E00700E00700E00700E00700E00700E07FE7FE1720809F19>I<001F81F80000F04F04 0001C07C06000380F80F000300F00F000700F00F00070070000007007000000700700000 070070000007007000000700700000FFFFFFFF0007007007000700700700070070070007 007007000700700700070070070007007007000700700700070070070007007007000700 700700070070070007007007000700700700070070070007007007000700700700070070 07007FE3FE3FF02420809F26>I<70F8F8F8F8F8F8F87070707070707070707020202020 20000000000070F8F8F87005217CA00D>33 D<7038F87CFC7EFC7E743A04020402040208 04080410081008201040200F0E7E9F17>I<70F8FCFC74040404080810102040060E7C9F 0D>39 D<0020004000800100020006000C000C0018001800300030003000700060006000 6000E000E000E000E000E000E000E000E000E000E000E000E00060006000600070003000 30003000180018000C000C000600020001000080004000200B2E7DA112>I<8000400020 00100008000C00060006000300030001800180018001C000C000C000C000E000E000E000 E000E000E000E000E000E000E000E000E000C000C000C001C00180018001800300030006 0006000C00080010002000400080000B2E7DA112>I<70F8FCFC74040404080810102040 060E7C840D>44 DI<70F8F8F87005057C840D>I<000100030003 000600060006000C000C000C00180018001800300030003000600060006000C000C000C0 0180018001800300030003000600060006000C000C000C00180018001800300030003000 600060006000C000C000C000102D7DA117>I<03F0000E1C001C0E001806003807007003 80700380700380700380F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003 C0F003C0F003C0F003C0F003C07003807003807003807807803807001806001C0E000E1C 0003F000121F7E9D17>I<018003800F80F3800380038003800380038003800380038003 8003800380038003800380038003800380038003800380038003800380038007C0FFFE0F 1E7C9D17>I<03F0000C1C00100E00200700400780800780F007C0F803C0F803C0F803C0 2007C00007C0000780000780000F00000E00001C0000380000700000600000C000018000 0300000600400C00401800401000803FFF807FFF80FFFF80121E7E9D17>I<03F0000C1C 00100E00200F00780F80780780780780380F80000F80000F00000F00000E00001C000038 0003F000003C00000E00000F000007800007800007C02007C0F807C0F807C0F807C0F007 80400780400F00200E001C3C0003F000121F7E9D17>I<000600000600000E00000E0000 1E00002E00002E00004E00008E00008E00010E00020E00020E00040E00080E00080E0010 0E00200E00200E00400E00C00E00FFFFF0000E00000E00000E00000E00000E00000E0000 0E0000FFE0141E7F9D17>I<1803001FFE001FFC001FF8001FE000100000100000100000 10000010000010000011F000161C00180E001007001007800003800003800003C00003C0 0003C07003C0F003C0F003C0E00380400380400700200600100E000C380003E000121F7E 9D17>I<007C000182000701000E03800C07801C07803803003800007800007000007000 00F1F000F21C00F40600F80700F80380F80380F003C0F003C0F003C0F003C0F003C07003 C07003C07003803803803807001807000C0E00061C0001F000121F7E9D17>I<4000007F FFC07FFF807FFF8040010080020080020080040000080000080000100000200000200000 400000400000C00000C00001C00001800003800003800003800003800007800007800007 8000078000078000078000078000030000121F7D9D17>I<03F0000C0C00100600300300 2001806001806001806001807001807803003E03003F06001FC8000FF00003F80007FC00 0C7E00103F00300F806003804001C0C001C0C000C0C000C0C000C0C00080600180200100 1002000C0C0003F000121F7E9D17>I<03F0000E18001C0C003806003807007007007003 80F00380F00380F003C0F003C0F003C0F003C0F003C07007C07007C03807C0180BC00E13 C003E3C0000380000380000380000700300700780600780E00700C002018001070000FC0 00121F7E9D17>I<70F8F8F8700000000000000000000070F8F8F87005147C930D>I<70F8 F8F8700000000000000000000070F0F8F878080808101010202040051D7C930D>I<0001 00000003800000038000000380000007C0000007C0000007C0000009E0000009E0000009 E0000010F0000010F0000010F00000207800002078000020780000403C0000403C000040 3C0000801E0000801E0000FFFE0001000F0001000F0001000F0002000780020007800200 0780040003C00E0003C01F0007E0FFC03FFE1F207F9F22>65 DI<000FC040007030C001C009C0038005C0070003C00E0001 C01E0000C01C0000C03C0000C07C0000407C00004078000040F8000000F8000000F80000 00F8000000F8000000F8000000F8000000F8000000F8000000780000007C0000407C0000 403C0000401C0000401E0000800E000080070001000380020001C0040000703800000FC0 001A217D9F21>IIII<000FE0200078186000E004E0038002E0070001E00F0000E01E0000601E0000 603C0000603C0000207C00002078000020F8000000F8000000F8000000F8000000F80000 00F8000000F8000000F8007FFCF80003E0780001E07C0001E03C0001E03C0001E01E0001 E01E0001E00F0001E0070001E0038002E000E0046000781820000FE0001E217D9F24>I< FFF8FFF80F800F8007800F0007800F0007800F0007800F0007800F0007800F0007800F00 07800F0007800F0007800F0007800F0007800F0007FFFF0007800F0007800F0007800F00 07800F0007800F0007800F0007800F0007800F0007800F0007800F0007800F0007800F00 07800F0007800F000F800F80FFF8FFF81D1F7E9E22>II<0FFFC0007C00003C00003C00003C0000 3C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C0000 3C00003C00003C00003C00003C00003C00203C00F83C00F83C00F83C00F0380040780040 700030E0000F800012207E9E17>IIIII<00 1F800000F0F00001C0380007801E000F000F000E0007001E0007803C0003C03C0003C07C 0003E0780001E0780001E0F80001F0F80001F0F80001F0F80001F0F80001F0F80001F0F8 0001F0F80001F0F80001F0780001E07C0003E07C0003E03C0003C03C0003C01E0007800E 0007000F000F0007801E0001C0380000F0F000001F80001C217D9F23>II82 D<07E0800C1980100780300380600180600180E00180E00080E00080 E00080F00000F000007800007F00003FF0001FFC000FFE0003FF00001F800007800003C0 0003C00001C08001C08001C08001C08001C0C00180C00380E00300F00600CE0C0081F800 12217D9F19>I<7FFFFFE0780F01E0600F0060400F0020400F0020C00F0030800F001080 0F0010800F0010800F0010000F0000000F0000000F0000000F0000000F0000000F000000 0F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F000000 0F0000000F0000000F0000000F0000001F800007FFFE001C1F7E9E21>IIII<7FF83FF80F E00FC007C0070003C0020001E0040001F00C0000F0080000781000007C1000003C200000 3E4000001E4000000F8000000F8000000780000003C0000007E0000005E0000009F00000 18F8000010780000207C0000603C0000401E0000801F0001800F0001000780020007C007 0003C01F8007E0FFE01FFE1F1F7F9E22>I91 D<080410082010201040204020804080408040B85CFC7EFC7E7C3E381C0F0E7B9F17>I< FEFE06060606060606060606060606060606060606060606060606060606060606060606 06060606060606FEFE072D7FA10D>I<1FE000303000781800781C00300E00000E00000E 00000E0000FE00078E001E0E00380E00780E00F00E10F00E10F00E10F01E10781E103867 200F83C014147E9317>97 D<0E0000FE00000E00000E00000E00000E00000E00000E0000 0E00000E00000E00000E00000E3E000EC3800F01C00F00E00E00E00E00700E00700E0078 0E00780E00780E00780E00780E00780E00700E00700E00E00F00E00D01C00CC300083E00 15207F9F19>I<03F80E0C1C1E381E380C70007000F000F000F000F000F000F000700070 00380138011C020E0C03F010147E9314>I<000380003F80000380000380000380000380 00038000038000038000038000038000038003E380061B801C0780380380380380700380 700380F00380F00380F00380F00380F00380F003807003807003803803803807801C0780 0E1B8003E3F815207E9F19>I<03F0000E1C001C0E00380700380700700700700380F003 80F00380FFFF80F00000F00000F000007000007000003800801800800C010007060001F8 0011147F9314>I<007C00C6018F038F07060700070007000700070007000700FFF00700 070007000700070007000700070007000700070007000700070007000700070007007FF0 1020809F0E>I<0000E003E3300E3C301C1C30380E00780F00780F00780F00780F00780F 00380E001C1C001E380033E0002000002000003000003000003FFE001FFF800FFFC03001 E0600070C00030C00030C00030C000306000603000C01C038003FC00141F7F9417>I<0E 0000FE00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E 3E000E43000E81800F01C00F01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E01C00E01C00E01C0FFE7FC16207F9F19>I<1C001E003E00 1E001C000000000000000000000000000E007E000E000E000E000E000E000E000E000E00 0E000E000E000E000E000E000E000E000E00FFC00A1F809E0C>I<00E001F001F001F000 E0000000000000000000000000007007F000F00070007000700070007000700070007000 700070007000700070007000700070007000700070007000706070F060F0C061803F000C 28829E0E>I<0E0000FE00000E00000E00000E00000E00000E00000E00000E00000E0000 0E00000E00000E0FF00E03C00E03000E02000E04000E08000E10000E30000E70000EF800 0F38000E1C000E1E000E0E000E07000E07800E03800E03C00E03E0FFCFF815207F9F18> I<0E00FE000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E 000E000E000E000E000E000E000E000E000E000E000E000E000E00FFE00B20809F0C>I< 0E1F01F000FE618618000E81C81C000F00F00E000F00F00E000E00E00E000E00E00E000E 00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00 E00E000E00E00E000E00E00E000E00E00E000E00E00E00FFE7FE7FE023147F9326>I<0E 3E00FE43000E81800F01C00F01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E01C00E01C00E01C0FFE7FC16147F9319>I<01F800070E00 1C03803801C03801C07000E07000E0F000F0F000F0F000F0F000F0F000F0F000F07000E0 7000E03801C03801C01C0380070E0001F80014147F9317>I<0E3E00FEC3800F01C00F00 E00E00E00E00F00E00700E00780E00780E00780E00780E00780E00780E00700E00F00E00 E00F01E00F01C00EC3000E3E000E00000E00000E00000E00000E00000E00000E00000E00 00FFE000151D7F9319>I<03E0800619801C05803C0780380380780380700380F00380F0 0380F00380F00380F00380F003807003807803803803803807801C0B800E138003E38000 0380000380000380000380000380000380000380000380003FF8151D7E9318>I<0E78FE 8C0F1E0F1E0F0C0E000E000E000E000E000E000E000E000E000E000E000E000E000E00FF E00F147F9312>I<1F9030704030C010C010C010E00078007F803FE00FF0007080388018 8018C018C018E030D0608F800D147E9312>I<020002000200060006000E000E003E00FF F80E000E000E000E000E000E000E000E000E000E000E000E080E080E080E080E08061003 1001E00D1C7F9B12>I<0E01C0FE1FC00E01C00E01C00E01C00E01C00E01C00E01C00E01 C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E03C00603C0030DC001F1FC1614 7F9319>II< FF9FE1FC3C0780701C0300601C0380200E0380400E0380400E03C0400707C0800704C080 0704E080038861000388710003C8730001D0320001D03A0000F03C0000E01C0000E01C00 00601800004008001E147F9321>I<7FC3FC0F01E00701C007018003810001C20000E400 00EC00007800003800003C00007C00004E000087000107000303800201C00601E01E01E0 FF07FE1714809318>II<3FFF380E 200E201C40384078407000E001E001C00380078007010E011E011C0338027006700EFFFE 10147F9314>I E /Fr 47 122 df<70F8FCFC7404040404080810102040060F7C840E> 44 DI<70F8F8F87005057C840E>I<01F000071C000C06001803 003803803803807001C07001C07001C07001C0F001E0F001E0F001E0F001E0F001E0F001 E0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001E07001C07001C07001C07803 C03803803803801C07000C0600071C0001F00013227EA018>48 D<008003800F80F38003 800380038003800380038003800380038003800380038003800380038003800380038003 800380038003800380038003800380038007C0FFFE0F217CA018>I<03F8000C1E001007 002007804007C07807C07803C07807C03807C0000780000780000700000F00000E000038 0003F000001C00000F000007800007800003C00003C00003E02003E07003E0F803E0F803 E0F003C04003C0400780200780100F000C1C0003F00013227EA018>51 D<000200000600000E00000E00001E00001E00002E00004E00004E00008E00008E00010E 00020E00020E00040E00040E00080E00100E00100E00200E00200E00400E00800E00FFFF F8000E00000E00000E00000E00000E00000E00000E00001F0001FFF015217FA018>I<10 00801E07001FFF001FFE001FF80013E00010000010000010000010000010000010000010 F800130E001407001803801003800001C00001C00001E00001E00001E00001E07001E0F0 01E0F001E0E001C08001C04003C04003802007001006000C1C0003F00013227EA018>I< 007E0001C1000300800601C00E03C01C03C0180180380000380000780000700000700000 F0F800F30C00F40600F40300F80380F801C0F001C0F001E0F001E0F001E0F001E0F001E0 7001E07001E07001E03801C03801C01803801C03000C0600070C0001F00013227EA018> I<4000006000007FFFE07FFFC07FFFC0400080C001008001008002008002000004000008 0000080000100000300000200000600000600000600000E00000C00000C00001C00001C0 0001C00001C00003C00003C00003C00003C00003C00003C00003C00003C0000180001323 7DA118>I<01F800060E000803001001802001802000C06000C06000C06000C07000C078 01803E01003F02001FC4000FF80003F80003FC00067F00083F80100F803007C06001C060 00E0C000E0C00060C00060C00060C000606000406000C03000801803000E0E0003F00013 227EA018>I<01F000060C000C0600180700380380700380700380F001C0F001C0F001C0 F001E0F001E0F001E0F001E0F001E07001E07003E03803E01805E00C05E00619E003E1E0 0001C00001C00001C0000380000380300300780700780600700C002018001030000FC000 13227EA018>I<0001800000018000000180000003C0000003C0000003C0000005E00000 05E000000DF0000008F0000008F0000010F800001078000010780000203C0000203C0000 203C0000401E0000401E0000401E0000800F0000800F0000FFFF00010007800100078003 0007C0020003C0020003C0040003E0040001E0040001E00C0000F00C0000F03E0001F8FF 800FFF20237EA225>65 D<0007E0100038183000E0063001C00170038000F0070000F00E 0000701E0000701C0000303C0000303C0000307C0000107800001078000010F8000000F8 000000F8000000F8000000F8000000F8000000F8000000F800000078000000780000107C 0000103C0000103C0000101C0000201E0000200E000040070000400380008001C0010000 E0020000381C000007E0001C247DA223>67 D69 DI73 D77 DI<000FE00000783C0000E00E0003C0078007 8003C00F0001E00E0000E01E0000F03C0000783C0000787C00007C7C00007C7800003C78 00003CF800003EF800003EF800003EF800003EF800003EF800003EF800003EF800003EF8 00003E7800003C7C00007C7C00007C3C0000783E0000F81E0000F00F0001E00F0001E007 8003C003C0078000E00E0000783C00000FE0001F247DA226>II82 D<03F0200C0C601802603001E07000E0600060E00060E00060E00020E00020E00020F000 00F000007800007F00003FF0001FFE000FFF0003FF80003FC00007E00001E00000F00000 F0000070800070800070800070800070C00060C00060E000C0F000C0C80180C6070081FC 0014247DA21B>I<7FFFFFF87807807860078018400780084007800840078008C007800C 800780048007800480078004800780040007800000078000000780000007800000078000 000780000007800000078000000780000007800000078000000780000007800000078000 00078000000780000007800000078000000780000007800000078000000FC00003FFFF00 1E227EA123>I<0FE0001838003C0C003C0E0018070000070000070000070000FF0007C7 001E07003C0700780700700700F00708F00708F00708F00F087817083C23900FC1E01515 7E9418>97 D<0E0000FE00001E00000E00000E00000E00000E00000E00000E00000E0000 0E00000E00000E00000E00000E1F000E61C00E80600F00300E00380E003C0E001C0E001E 0E001E0E001E0E001E0E001E0E001E0E001E0E001C0E003C0E00380F00700C80600C41C0 083F0017237FA21B>I<01FE000703000C07801C0780380300780000700000F00000F000 00F00000F00000F00000F00000F000007000007800403800401C00800C010007060001F8 0012157E9416>I<0000E0000FE00001E00000E00000E00000E00000E00000E00000E000 00E00000E00000E00000E00000E001F8E00704E00C02E01C01E03800E07800E07000E0F0 00E0F000E0F000E0F000E0F000E0F000E0F000E07000E07800E03800E01801E00C02E007 0CF001F0FE17237EA21B>I<01FC000707000C03801C01C03801C07801E07000E0F000E0 FFFFE0F00000F00000F00000F00000F000007000007800203800201C00400E0080070300 00FC0013157F9416>I<003C00C6018F038F030F07000700070007000700070007000700 0700FFF80700070007000700070007000700070007000700070007000700070007000700 0700070007807FF8102380A20F>I<00007001F198071E180E0E181C07001C07003C0780 3C07803C07803C07801C07001C07000E0E000F1C0019F000100000100000180000180000 1FFE000FFFC00FFFE03800F0600030400018C00018C00018C000186000306000303800E0 0E038003FE0015217F9518>I<0E0000FE00001E00000E00000E00000E00000E00000E00 000E00000E00000E00000E00000E00000E00000E1F800E60C00E80E00F00700F00700E00 700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00 700E00700E0070FFE7FF18237FA21B>I<1C001E003E001E001C00000000000000000000 000000000000000E00FE001E000E000E000E000E000E000E000E000E000E000E000E000E 000E000E000E000E000E00FFC00A227FA10E>I<01C003E003E003E001C0000000000000 0000000000000000000001E00FE001E000E000E000E000E000E000E000E000E000E000E0 00E000E000E000E000E000E000E000E000E000E000E000E000E060E0F0C0F18061803E00 0B2C82A10F>I<0E0000FE00001E00000E00000E00000E00000E00000E00000E00000E00 000E00000E00000E00000E00000E03FC0E01F00E01C00E01800E02000E04000E08000E10 000E38000EF8000F1C000E1E000E0E000E07000E07800E03C00E01C00E01E00E00F00E00 F8FFE3FE17237FA21A>I<0E00FE001E000E000E000E000E000E000E000E000E000E000E 000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E 000E000E000E00FFE00B237FA20E>I<0E1FC07F00FE60E183801E807201C00F003C00E0 0F003C00E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E 003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E00 3800E00E003800E0FFE3FF8FFE27157F942A>I<0E1F80FE60C01E80E00F00700F00700E 00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E 00700E00700E0070FFE7FF18157F941B>I<01FC000707000C01801800C03800E0700070 700070F00078F00078F00078F00078F00078F00078F000787000707800F03800E01C01C0 0E038007070001FC0015157F9418>I<0E1F00FE61C00E80600F00700E00380E003C0E00 1C0E001E0E001E0E001E0E001E0E001E0E001E0E001E0E003C0E003C0E00380F00700E80 E00E41C00E3F000E00000E00000E00000E00000E00000E00000E00000E00000E0000FFE0 00171F7F941B>I<0E3CFE461E8F0F0F0F060F000E000E000E000E000E000E000E000E00 0E000E000E000E000E000F00FFF010157F9413>114 D<0F8830786018C018C008C008E0 08F0007F803FE00FF001F8003C801C800C800CC00CC008E018D0308FC00E157E9413>I< 02000200020002000600060006000E001E003E00FFF80E000E000E000E000E000E000E00 0E000E000E000E000E040E040E040E040E040E040708030801F00E1F7F9E13>I<0E0070 FE07F01E00F00E00700E00700E00700E00700E00700E00700E00700E00700E00700E0070 0E00700E00700E00700E00F00E00F006017003827800FC7F18157F941B>III121 D E /Fs 20 118 df45 D68 D73 D77 D80 D<007F802001FFE02007C078600F001C601E00 06E03C0003E0380001E0780000E0700000E070000060F0000060F0000060F0000020F000 0020F0000020F8000020F80000007C0000007E0000003F0000003FC000001FF800000FFF 800007FFF80003FFFC0000FFFF00000FFF800000FFC000001FE0000007E0000003F00000 01F0000000F0000000F8000000F88000007880000078800000788000007880000078C000 0078C0000070E00000F0E00000E0F00000E0F80001C0EC000380C7000700C1F01E00807F FC00800FF0001D337CB125>83 D<00FE00000303C0000C00E00010007000100038003C00 3C003E001C003E001E003E001E0008001E0000001E0000001E0000001E00000FFE0000FC 1E0003E01E000F801E001F001E003E001E003C001E007C001E00F8001E04F8001E04F800 1E04F8003E04F8003E0478003E047C005E043E008F080F0307F003FC03E01E1F7D9E21> 97 D<003F8000E0600380180700040F00041E001E1C003E3C003E7C003E7C0008780000 F80000F80000F80000F80000F80000F80000F80000F80000F800007800007C00007C0000 3C00011E00011E00020F000207000403801800E060003F80181F7D9E1D>99 D<000001E000003FE000003FE0000003E0000001E0000001E0000001E0000001E0000001 E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001 E0000001E0001F81E000F061E001C019E0078005E00F0003E00E0003E01E0001E03C0001 E03C0001E07C0001E0780001E0F80001E0F80001E0F80001E0F80001E0F80001E0F80001 E0F80001E0F80001E0F80001E0780001E0780001E03C0001E03C0001E01C0001E01E0003 E00E0005E0070009E0038011F000E061FF003F81FF20327DB125>I<003F800000E0E000 0380380007003C000E001E001E001E001C000F003C000F007C000F0078000F8078000780 F8000780F8000780FFFFFF80F8000000F8000000F8000000F8000000F8000000F8000000 780000007C0000003C0000003C0000801E0000800E0001000F0002000780020001C00C00 00F03000001FC000191F7E9E1D>I<0007E0001C1000383800707C00E07C01E07C01C038 03C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C000 FFFFC0FFFFC003C00003C00003C00003C00003C00003C00003C00003C00003C00003C000 03C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C000 03C00003C00003C00003C00007E0007FFF007FFF0016327FB114>I<000000F0007F0308 01C1C41C0380E81C070070080F0078001E003C001E003C003E003E003E003E003E003E00 3E003E003E003E003E003E001E003C001E003C000F007800070070000780E00009C1C000 087F000018000000180000001800000018000000180000001C0000000E0000000FFFF800 07FFFF0003FFFF800E000FC0180001E0300000F070000070E0000038E0000038E0000038 E0000038E00000387000007070000070380000E01C0001C00700070001C01C00003FE000 1E2F7E9F21>I<07000F801F801F800F8007000000000000000000000000000000000000 00000000000780FF80FF800F800780078007800780078007800780078007800780078007 800780078007800780078007800780078007800780078007800FC0FFF8FFF80D307EAF12 >105 D<0780FE001FC000FF83078060F000FF8C03C18078000F9001E2003C0007A001E4 003C0007A000F4001E0007C000F8001E0007C000F8001E00078000F0001E00078000F000 1E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E 00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00 078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E0007 8000F0001E00078000F0001E00078000F0001E000FC001F8003F00FFFC1FFF83FFF0FFFC 1FFF83FFF0341F7E9E38>109 D<0780FE0000FF83078000FF8C03C0000F9001E00007A0 01E00007A000F00007C000F00007C000F000078000F000078000F000078000F000078000 F000078000F000078000F000078000F000078000F000078000F000078000F000078000F0 00078000F000078000F000078000F000078000F000078000F000078000F000078000F000 078000F000078000F0000FC001F800FFFC1FFF80FFFC1FFF80211F7E9E25>I<001FC000 00F0780001C01C00070007000F0007801E0003C01C0001C03C0001E03C0001E0780000F0 780000F0780000F0F80000F8F80000F8F80000F8F80000F8F80000F8F80000F8F80000F8 F80000F8780000F07C0001F03C0001E03C0001E01E0003C01E0003C00F00078007800F00 01C01C0000F07800001FC0001D1F7E9E21>I<0783E0FF8C18FF907C0F907C07A07C07C0 3807C00007C00007C0000780000780000780000780000780000780000780000780000780 000780000780000780000780000780000780000780000780000780000780000FC000FFFE 00FFFE00161F7E9E19>114 D<01FC100E03301800F0300070600030E00030E00010E000 10E00010F00010F800007E00003FF0001FFF000FFFC003FFE0003FF00001F80000F88000 3C80003C80001CC0001CC0001CE0001CE00018F00038F00030CC0060C301C080FE00161F 7E9E1A>I<00400000400000400000400000400000C00000C00000C00001C00001C00003 C00007C0000FC0001FFFE0FFFFE003C00003C00003C00003C00003C00003C00003C00003 C00003C00003C00003C00003C00003C00003C00003C00003C00003C01003C01003C01003 C01003C01003C01003C01003C01001C02001E02000E0400078C0001F00142C7FAB19>I< 078000F000FF801FF000FF801FF0000F8001F000078000F000078000F000078000F00007 8000F000078000F000078000F000078000F000078000F000078000F000078000F0000780 00F000078000F000078000F000078000F000078000F000078000F000078000F000078000 F000078000F000078001F000078001F000078001F000038002F00003C004F00001C008F8 00007030FF80001FC0FF80211F7E9E25>I E /Ft 5 85 df<00000000C00000000000E0 0000000001E00000000003E00000000003E00000000007E00000000007E0000000000FE0 000000000FE0000000001FE0000000001FE00000000037E00000000067E00000000067E0 00000000C7E000000000C7F00000000183F00000000183F00000000303F00000000703F0 0000000603F00000000C03F00000000C03F00000001803F00000001803F00000003003F0 0000003003F00000006003F0000000C003F0000000C003F00000018003F00000018003F8 000003FFFFF8000003FFFFF80000060001F800000E0001F800000C0001F80000180001F8 0000180001F80000300001F80000300001F80000600001F80000E00001F80000C00001F8 0001C00001F80001C00001F80007C00001FC001FC00003FC00FFF8007FFFE0FFF8007FFF E02B327BB135>65 D<000FFFFFFE0000000FFFFFFF800000007F000FE00000007E0003F0 0000007E0000F80000007E0000FC0000007E00007C000000FC00003E000000FC00003E00 0000FC00003F000000FC00001F000001F800001F000001F800001F800001F800001F8000 01F800001F800003F000001F800003F000001F800003F000001F800003F000001F800007 E000003F800007E000003F800007E000003F800007E000003F80000FC000003F00000FC0 00007F00000FC000007F00000FC000007F00001F8000007E00001F800000FE00001F8000 00FE00001F800000FC00003F000001FC00003F000001F800003F000001F800003F000003 F000007E000003E000007E000007E000007E00000FC000007E00000F800000FC00001F80 0000FC00003F000000FC00007E000000FC0000FC000001F80001F0000001F80003E00000 01F8000FC0000003F8007F000000FFFFFFFC000000FFFFFFE000000031317BB036>68 D<000FFFFFFFFC000FFFFFFFFC00007F0001FC00007E00007C00007E00003C00007E0000 3C00007E0000180000FC0000180000FC0000180000FC0000180000FC0000180001F80000 180001F80000180001F80000180001F80000180003F00080100003F00180000003F00180 000003F00180000007E00300000007E00300000007E00700000007E01F0000000FFFFE00 00000FFFFE0000000FC01E0000000FC00E0000001F800C0000001F800C0000001F800C00 00001F800C0000003F00180000003F00080000003F00000000003F00000000007E000000 00007E00000000007E00000000007E0000000000FC0000000000FC0000000000FC000000 0000FC0000000001F80000000001F80000000001F80000000003F800000000FFFFF00000 00FFFFF00000002E317BB02F>70 D<000FFFFFF000000FFFFFFE0000007F003F8000007E 000FC000007E0007E000007E0003F000007E0001F80000FC0001F80000FC0001F80000FC 0001F80000FC0001F80001F80003F80001F80003F80001F80003F80001F80003F00003F0 0007F00003F00007E00003F0000FC00003F0000FC00007E0001F000007E0007E000007E0 00FC000007E007F000000FFFFFC000000FFFFF0000000FC00F8000000FC003C000001F80 03E000001F8001F000001F8001F000001F8001F800003F0001F800003F0001F800003F00 01F800003F0001F800007E0003F800007E0003F800007E0003F000007E0003F00000FC00 07F00000FC0007F00000FC0007F00800FC0007F00C01F80007F01801F80007F01801F800 03F03003F80003F030FFFFE001F0E0FFFFE000FFC0000000003F002E327BB034>82 D<07FFFFFFFFF00FFFFFFFFFF00FC00FE003F01E000FC000F01C000FC000E018000FC000 E038000FC0006030001F8000E030001F8000E060001F8000C060001F8000C060003F0000 C0C0003F0000C0C0003F0000C0C0003F0000C080007E00008000007E00000000007E0000 0000007E0000000000FC0000000000FC0000000000FC0000000000FC0000000001F80000 000001F80000000001F80000000001F80000000003F00000000003F00000000003F00000 000003F00000000007E00000000007E00000000007E00000000007E0000000000FC00000 00000FC0000000000FC0000000000FC0000000001F80000000001F80000000001F800000 00001F80000000003F00000000003F00000000003F0000000000FF00000000FFFFFF0000 00FFFFFF0000002C3173B033>84 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300dpi TeXDict begin %%EndSetup %%Page: 0 1 0 0 bop 795 908 a Ft(D)26 b(R)g(A)f(F)h(T)225 999 y Fs(Do)r(cumen)n(t) 20 b(for)i(a)f(Standard)g(Message-P)n(assing)f(In)n(terface)621 1194 y Fr(Message)c(P)o(assing)h(In)o(terface)e(F)l(orum)793 1320 y(Octob)q(er)h(17,)h(1995)77 1378 y(This)g(w)o(ork)f(w)o(as)h (supp)q(orted)g(in)f(part)h(b)o(y)e(ARP)l(A)h(and)h(NSF)e(under)h(gran) o(t)h(ASC-9310330,)i(the)192 1436 y(National)d(Science)f(F)l(oundation) i(Science)e(and)i(T)l(ec)o(hnology)f(Cen)o(ter)f(Co)q(op)q(erativ)o(e) 76 1494 y(Agreemen)o(t)e(No.)22 b(CCR-8809615,)d(and)e(b)o(y)e(the)h (Commission)e(of)j(the)f(Europ)q(ean)i(Comm)o(unit)n(y)654 1552 y(through)f(Esprit)f(pro)s(ject)g(P6643.)p eop %%Page: 1 2 1 1 bop 166 45 a Fq(This)20 b(is)h(the)f(result)g(of)f(a)h(LaT)l(eX)g (run)g(of)g(a)f(draft)g(of)h(a)f(single)j(c)o(hapter)d(of)h(the)g(MPIF) f(Final)75 102 y(Rep)q(ort)d(do)q(cumen)o(t.)969 2828 y(i)p eop %%Page: 1 3 1 2 bop 75 356 a Fp(Chapter)34 b(1)75 564 y Fo(Dynamic)40 b(Pro)s(cesses)75 786 y Fq(This)20 b(v)o(ersion)f(is)g(substan)o (tially)h(c)o(hanged)g(\(and)f(shortened!\))31 b(from)18 b(the)h(previous)h(prop)q(osal.)32 b(The)75 843 y(main)23 b(di\013erence)g(is)g(the)g(abandonmen)o(t)f(of)g(the)g(am)o(bition)h (to)f(pro)o(vide)h(a)f(p)q(ortable)h(in)o(terface)g(to)75 899 y(resource)14 b(managers.)k(This)d(simpli\014es)h(the)e(in)o (terface)g(substan)o(tially)l(,)h(at)e(some)g(cost)g(in)i(functionalit) o(y)l(.)75 956 y(The)g(main)h(c)o(hanges)f(are:)143 1062 y Fn(\017)23 b Fq(There)14 b(is)h(no)f(assumption)h(of)f(the)g (existence)i(of)e(a)g(resource)g(manager,)f(so)h(the)h(long)f (discussion)189 1119 y(at)e(the)h(b)q(eginning)j(ab)q(out)d(the)g(mo)q (del)h(of)f(the)g(run)o(time)g(en)o(vironmen)o(t,)h(with)f(resource)h (manager)189 1175 y(and)h(pro)q(cess)g(manager,)g(is)g(gone.)143 1269 y Fn(\017)23 b Fq(There)15 b(are)g(no)g(resource)g(ob)s(jects,)g (so)f(all)i(the)g(resource-manipulation)h(functions)f(are)f(gone.)143 1363 y Fn(\017)23 b Fq(There)15 b(is)g(only)g(one)g(surviving)h(en)o (vironmen)o(t-query)g(capabilit)o(y)l(,)g(to)e(\014nd)i(out)e(ho)o(w)h (man)o(y)f(pro-)189 1419 y(cesses)c(ma)o(y)g(usefully)h(b)q(e)g(spa)o (wned.)19 b(This)10 b(is)h(implemen)o(ted)h(as)e(an)g(attribute)g(on)g Fm(MPI)p 1667 1419 14 2 v 16 w(COMM)p 1825 1419 V 16 w(W)o(ORLD)p Fq(,)189 1476 y(and)k(allo)o(ws)g(man)o(y)g(existing)h (programs)e(to)h(b)q(e)h(p)q(orted)f(to)f(MPI)i(straigh)o(tforw)o (ardly)l(.)j(Sp)q(eci\014ca-)189 1532 y(tion)d(of)g(where)h(they)f (will)i(b)q(e)f(spa)o(wned)f(will)i(ha)o(v)o(e)e(to)g(b)q(e)h(part)f (of)g(mpirun,)h(if)f(it)h(is)g(to)f(b)q(e)h(done)189 1588 y(p)q(ortably)l(.)143 1682 y Fn(\017)23 b Fq(There)15 b(is)h(no)f(MPI)p 521 1682 V 16 w(ALLOCA)l(TE,)h(since)h(there)e(are)g (no)g(resources.)143 1776 y Fn(\017)23 b Fq(There)c(is)i(no)e(pro)q (cess)h(ob)s(ject.)32 b(Instead)20 b(pro)q(cesses)g(are)g(iden)o (ti\014ed)h(b)o(y)f(their)g(\(group,)g(rank\))189 1833 y(pair.)143 1926 y Fn(\017)j Fq(There)16 b(are)f(not)h(\(y)o(et\))e(an) o(y)i(non-blo)q(c)o(king)i(calls.)k(This)17 b(will)g(probably)g(ha)o(v) o(e)e(to)g(c)o(hange,)h(esp)q(e-)189 1983 y(cially)h(for)d(the)h(clien) o(t-serv)o(er)i(stu\013.)143 2077 y Fn(\017)23 b Fq(The)15 b(examples)h(ha)o(v)o(e)f(b)q(een)h(up)q(dated.)75 2220 y Fl(1.1)59 b(Intro)r(duction)75 2321 y Fq(The)15 b(MPI-1)g(message)g (passing)g(library)h(allo)o(ws)f(pro)q(cesses)h(in)g(a)f(parallel)h (program)e(to)h(comm)o(unicate)75 2378 y(with)21 b(one)f(another.)34 b(MPI-1)20 b(sp)q(eci\014es)i(neither)g(ho)o(w)d(the)i(pro)q(cesses)f (are)g(created,)h(nor)f(ho)o(w)g(they)75 2434 y(establish)f(comm)o (unication.)30 b(Moreo)o(v)o(er,)17 b(an)i(MPI-1)f(application)i(is)e (static,)h(that)e(is,)i(no)g(pro)q(cesses)75 2491 y(can)c(b)q(e)h (added)g(to)f(or)f(deleted)j(from)d(an)h(application)i(while)g(it)e(is) h(running.)166 2547 y(MPI)22 b(users)g(ha)o(v)o(e)g(ask)o(ed)f(that)h (the)g(MPI-1)g(mo)q(del)h(b)q(e)f(relaxed)h(to)f(allo)o(w)g(dynamic)h (pro)q(cess)75 2604 y(managemen)o(t.)17 b(The)11 b(main)f(imp)q(etus)i (comes)e(from)f(the)h(PVM)g([1])f(researc)o(h)h(e\013ort,)g(whic)o(h)h (has)f(pro)o(vided)75 2660 y(a)16 b(w)o(ealth)h(of)f(exp)q(erience)j (with)e(dynamic)h(pro)q(cess)f(managemen)o(t)f(that)g(illustrates)h (its)g(b)q(ene\014ts)h(and)964 2828 y(1)p eop %%Page: 2 4 2 3 bop 75 -100 a Fq(2)951 b Fk(CHAPTER)16 b(1.)34 b(D)o(YNAMIC)15 b(PR)o(OCESSES)75 45 y Fq(p)q(oten)o(tial)h(pitfalls.)22 b(The)15 b(reasons)g(for)g(adding)h(dynamic)g(pro)q(cess)g(managemen)o (t)e(are)h(b)q(oth)g(tec)o(hnical)75 102 y(and)g(practical.)143 188 y Fn(\017)23 b Fq(W)l(orkstation)d(net)o(w)o(ork)f(users)i (migrating)g(from)g(PVM)f(to)g(MPI)h(are)g(accustomed)g(to)f(using)189 244 y(PVM's)13 b(capabilities)k(for)d(pro)q(cess)g(and)h(resource)f (managemen)o(t.)19 b(While)d(relativ)o(ely)f(few)f(appli-)189 301 y(cations)g(are)g(truly)h(dynamic)g(or)f(require)i(features)e(not)g (in)h(MPI,)f(the)h(lac)o(k)g(of)f(these)g(features)g(is)189 357 y(a)h(practical)h(stum)o(bling)g(blo)q(c)o(k)g(to)e(migration.)143 445 y Fn(\017)23 b Fq(Imp)q(ortan)o(t)16 b(classes)h(of)g(message)f (passing)h(applications,)i(suc)o(h)e(as)f(clien)o(t-serv)o(er)i (systems)f(and)189 502 y(task)d(farming)h(jobs,)g(require)h(dynamic)g (pro)q(cess)f(con)o(trol.)143 590 y Fn(\017)23 b Fq(With)16 b(dynamic)h(resource)g(and)f(pro)q(cess)h(managemen)o(t)e(extensions,)i (it)g(w)o(ould)g(b)q(e)g(p)q(ossible)h(to)189 646 y(write)d(ma)s(jor)f (parts)g(of)h(the)g(parallel)i(programming)e(en)o(vironmen)o(t)g(in)h (MPI)f(itself.)143 735 y Fn(\017)23 b Fq(The)14 b(abilit)o(y)i(to)d (write)h(fault)h(toleran)o(t)e(applications)j(is)f(imp)q(ortan)o(t)f (in)h(unstable)g(en)o(vironmen)o(ts)189 791 y(and)23 b(for)f(commercial)i(applications.)45 b(MPI-1)23 b(do)q(es)g(not)f(pro) o(vide)i(mec)o(hanisms)f(for)g(build-)189 847 y(ing)18 b(fault-toleran)o(t)f(applications.)30 b(The)18 b(mec)o(hanisms)g (required)h(to)e(supp)q(ort)h(fault)g(tolerance)189 904 y(largely)d(o)o(v)o(erlap)g(with)h(those)f(needed)h(to)f(supp)q(ort)g (dynamic)h(pro)q(cess)g(managemen)o(t.)166 990 y(While)e(dynamic)g(pro) q(cess)f(managemen)o(t)f(is)i(essen)o(tial,)g(it)f(is)g(imp)q(ortan)o (t)g(not)f(to)h(compromise)g(the)75 1047 y(p)q(ortabilit)o(y)j(or)f(p)q (erformance)g(of)g(MPI.)g(In)h(particular:)143 1126 y Fn(\017)23 b Fq(The)f(MPI-2)f(dynamic)i(pro)q(cess)e(managemen)o(t)g (mo)q(del)i(m)o(ust)e(apply)h(to)f(the)h(v)m(ast)f(ma)s(jorit)o(y)189 1183 y(of)e(curren)o(t)h(parallel)h(en)o(vironmen)o(ts.)34 b(These)20 b(include)j(ev)o(erything)d(from)f(tigh)o(tly)h(in)o (tegrated)189 1239 y(MPPs)d(suc)o(h)i(as)f(the)g(In)o(tel)h(P)o(aragon) e(and)h(the)g(Meik)o(o)g(CS-2)h(to)e(heterogeneous)h(net)o(w)o(orks)f (of)189 1296 y(w)o(orkstations.)143 1384 y Fn(\017)23 b Fq(MPI)16 b(m)o(ust)g(not)g(tak)o(e)g(o)o(v)o(er)f(op)q(erating)i (system)f(resp)q(onsibiliti)q(es.)26 b(It)17 b(should)g(instead)g(pro)o (vide)189 1440 y(a)e(clean)h(in)o(terface)f(b)q(et)o(w)o(een)h(an)f (application)i(and)e(system)g(soft)o(w)o(are.)143 1528 y Fn(\017)23 b Fq(MPI)17 b(m)o(ust)g(con)o(tin)o(ue)h(to)f(guaran)o (tee)g(comm)o(unication)h(determinism,)h(i.e.,)e(dynamic)i(pro)q(cess) 189 1585 y(managemen)o(t)14 b(m)o(ust)h(not)f(in)o(tro)q(duce)j(una)o (v)o(oidable)f(race)f(conditions.)143 1673 y Fn(\017)23 b Fq(MPI)15 b(m)o(ust)f(not)h(con)o(tain)h(features)e(that)h (compromise)g(p)q(erformance.)143 1761 y Fn(\017)23 b Fq(MPI-1)16 b(programs)f(m)o(ust)h(w)o(ork)g(under)h(MPI-2,)f(i.e.,)g (the)h(MPI-1)f(static)g(pro)q(cess)h(mo)q(del)h(m)o(ust)189 1817 y(b)q(e)e(a)e(sp)q(ecial)j(case)f(of)e(the)i(MPI-2)f(dynamic)h(mo) q(del.)166 1897 y(The)e(MPI)g(dynamic)h(pro)q(cess)g(managemen)o(t)e (mo)q(del)i(address)f(these)h(issues)g(in)f(t)o(w)o(o)f(w)o(a)o(ys.)19 b(First,)75 1953 y(MPI)d(remains)g(primarily)h(a)f(comm)o(unication)h (library)l(.)23 b(It)16 b(do)q(es)g(not)f(manage)h(the)g(parallel)h(en) o(viron-)75 2010 y(men)o(t)d(in)h(whic)o(h)g(a)f(parallel)i(program)d (executes,)i(though)f(it)h(pro)o(vides)f(a)g(minimal)i(in)o(terface)f (b)q(et)o(w)o(een)75 2066 y(an)g(application)i(and)e(external)h (resource)f(and)h(pro)q(cess)f(managers.)166 2123 y(Second,)i(MPI-2)g (do)q(es)g(not)f(c)o(hange)h(the)g(concept)g(of)f(comm)o(unicator.)24 b(Once)18 b(a)e(comm)o(unicator)75 2179 y(is)h(built,)g(it)g(b)q(eha)o (v)o(es)f(as)g(sp)q(eci\014ed)j(in)e(MPI-1.)23 b(A)16 b(comm)o(unicator)g(is)h(nev)o(er)f(c)o(hanged)h(once)f(created,)75 2236 y(and)f(it)h(is)g(alw)o(a)o(ys)e(created)h(using)h(deterministic)h (collectiv)o(e)g(seman)o(tics.)75 2376 y Fl(1.2)59 b(The)19 b(MPI-2)h(Dynamic)f(Pro)r(cess)g(Mo)r(del)75 2478 y Fq(The)i(MPI-2)g (dynamic)h(pro)q(cess)g(mo)q(del)g(allo)o(ws)f(for)f(the)i(creation)f (and)g(destruction)h(of)f(pro)q(cesses)75 2534 y(after)14 b(an)g(MPI)h(application)h(has)e(started.)19 b(It)14 b(pro)o(vides)h(a)g(mec)o(hanism)g(to)f(establish)h(comm)o(unication)75 2591 y(b)q(et)o(w)o(een)i(the)g(newly)h(created)f(pro)q(cesses)g(and)g (the)g(existing)h(MPI)f(application.)27 b(It)17 b(also)g(pro)o(vides)g (a)75 2647 y(mec)o(hanism)f(to)e(establish)j(comm)o(unication)f(b)q(et) o(w)o(een)f(t)o(w)o(o)f(existing)i(MPI)f(applications,)i(ev)o(en)e (when)75 2704 y(one)g(did)i(not)d(\\start")g(the)h(other.)1967 46 y Fj(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 3 5 3 4 bop 75 -100 a Fk(1.2.)34 b(THE)15 b(MPI-2)g(D)o(YNAMIC)g(PR)o (OCESS)h(MODEL)775 b Fq(3)75 45 y Fi(1.2.1)49 b(Sta)o(rting)16 b(and)h(Managing)h(Pro)q(cesses)75 131 y Fq(MPI)10 b(applications)i(ma) o(y)e(start)f(new)h(pro)q(cesses)h(\(including)i(non-MPI)d(pro)q (cesses\),)h(send)g(them)g(signals,)75 187 y(and)17 b(\014nd)h(out)e (when)i(they)f(die)h(or)e(b)q(ecome)i(unreac)o(hable.)26 b(They)17 b(do)g(this)g(through)g(an)f(in)o(terface)i(to)75 244 y(an)d(external)h(pro)q(cess)g(manager,)f(whic)o(h)h(can)g(range)f (from)g(a)g(parallel)i(op)q(erating)f(system)f(\(CMOST\))75 300 y(to)g(la)o(y)o(ered)g(soft)o(w)o(are)e(\(POE\))i(to)g(an)g Fh(rsh)g Fq(command)g(\(P4\).)166 357 y(There)g(are)g(three)h(w)o(a)o (ys)e(to)g(start)g(new)i(pro)q(cesses.)166 413 y Fm(MPI)p 251 413 14 2 v 16 w(SP)l(A)-5 b(WN)23 b Fq(is)f(a)g(simple)h(in)o (terface)g(that)e(allo)o(ws)h(starts)f(MPI)h(pro)q(cesses)g(and)h (establishes)75 470 y(comm)o(unication)16 b(with)f(them,)g(returning)h (an)f(in)o(tercomm)o(unicator.)166 526 y(The)23 b(t)o(w)o(o)f(other)h (routines)g(are)g(used)h(in)g(conjunction)g(with)g Fm(MPI)p 1378 526 V 16 w(GROUP)p 1546 526 V 17 w(INIT)f Fq(to)f(start)g(a)75 583 y(non)o(uniform)16 b(collection)h(of)d(pro)q(cesses.)166 639 y Fm(MPI)p 251 639 V 16 w(GROUP)p 419 639 V 17 w(ST)l(ART)p 578 639 V 18 w(A)l(TT)l(A)o(CH)24 b Fq(starts)e(MPI)h(pro)q(cesses)h (and)g(establishes)g(comm)o(unication)75 695 y(with)16 b(them,)e(returning)i(an)f(in)o(tercomm)o(unicator.)166 752 y Fm(MPI)p 251 752 V 16 w(GROUP)p 419 752 V 17 w(ST)l(ART)p 578 752 V 18 w(INDEPENDENT)c Fq(starts)e(new)j(pro)q(cesses,)f(whic)o (h)h(ma)o(y)e(or)g(ma)o(y)h(not)f(com-)75 808 y(m)o(unicate)18 b(amoung)f(eac)o(h)h(other,)g(but)f(don't)h(comm)o(unicate)g(with)g (their)g(paren)o(t.)27 b(This)18 b(function)g(is)75 865 y(required)e(for)f(starting)f(non-MPI)i(tasks)e(and)i(is)f(useful)i (for)d(starting)h(an)g(SPMD)g(application.)166 921 y(MPI)f(uses)h(the)g (existing)g(group)g(abstraction)f(to)g(represen)o(t)g(pro)q(cesses.)20 b(A)15 b(pro)q(cess)g(is)g(iden)o(ti\014ed)75 978 y(b)o(y)g(a)g (\(group,)f(rank\))h(pair.)75 1099 y Fi(1.2.2)49 b(The)16 b(Runtime)g(Environment)g(\(w)o(as:)21 b(Resources\))75 1185 y Fq(MPI)10 b(allo)o(ws)g(an)g(application)i(to)e(start)f(new)h (pro)q(cesses)h(with)f(the)g(routines)h Fm(MPI)p 1453 1185 V 16 w(SP)l(A)-5 b(WN)p Fq(,)10 b Fm(MPI)p 1724 1185 V 16 w(ST)l(ART)p 1882 1185 V 18 w(A)l(TT)l(A)o(CH)p Fq(,)75 1242 y(and)24 b Fm(MPI)p 257 1242 V 15 w(ST)l(ART)p 414 1242 V 18 w(INDEPENDENT)p Fq(.)f(The)h(in)o(terface)g(for)e (starting)h(new)h(pro)q(cesses)g(is)g(relativ)o(ely)75 1298 y(straigh)o(tforw)o(ard.)18 b(The)d(hard)g(part)g(is)h(to)e(sp)q (ecify)j Fg(wher)n(e)e Fq(the)g(new)h(pro)q(cesses)f(start.)166 1355 y(The)f(di\016cult)o(y)h(is)f(that)f(there)h(is)g(an)g(enormous)g (range)f(of)g(run)o(time)h(en)o(vironmen)o(ts)g(and)g(applica-)75 1411 y(tion)i(requiremen)o(ts,)g(and)g(MPI)g(m)o(ust)f(not)h(b)q(e)g (tailored)h(to)e(an)o(y)g(particular)i(one.)22 b(Examples)16 b(of)g(suc)o(h)75 1468 y(en)o(vironmen)o(ts)f(are:)143 1574 y Fn(\017)23 b Ff(MPP)15 b(managed)i(b)o(y)e(a)i(batc)o(h)g (queueing)f(system)p Fq(.)j(Batc)o(h)13 b(queueing)j(systems)e (generally)189 1630 y(allo)q(cate)j(resources)g(b)q(efore)g(an)g (application)i(b)q(egins,)f(enforce)f(limits)h(on)f(resource)g(use)g (\(CPU)189 1687 y(time,)f(memory)g(use,)g(etc\),)g(and)g(do)h(not)e (allo)o(w)i(a)f(c)o(hange)g(in)h(resource)f(allo)q(cation)i(after)d(a)h (job)189 1743 y(b)q(egins.)21 b(Moreo)o(v)o(er,)13 b(man)o(y)i(MPPs)f (ha)o(v)o(e)h(sp)q(ecial)h(limitations)h(or)d(extensions,)i(suc)o(h)f (as)g(a)g(limit)189 1800 y(the)c(n)o(um)o(b)q(er)h(of)f(pro)q(cesses)h (that)f(ma)o(y)g(run)h(on)g(one)f(pro)q(cessor,)h(or)f(the)h(abilit)o (y)h(to)e(gang-sc)o(hedule)189 1856 y(pro)q(cesses)k(of)g(a)g(parallel) i(application.)143 1950 y Fn(\017)23 b Ff(Net)o(w)o(ork)17 b(of)i(w)o(orkstations)g(with)h(PVM)p Fq(.)15 b(PVM)h(\(P)o(arallel)h (Virtual)h(Mac)o(hine\))e(allo)o(ws)h(a)189 2006 y(user)f(to)g(create)g (a)g(\\virtual)h(mac)o(hine")g(out)f(of)g(a)g(net)o(w)o(ork)f(of)h(w)o (orkstations.)23 b(An)16 b(application)189 2063 y(ma)o(y)i(extend)j (the)e(virtual)h(mac)o(hine)h(or)e(manage)g(pro)q(cesses)h(\(create,)f (kill,)j(redirect)f(output,)189 2119 y(etc.\))26 b(through)18 b(the)f(PVM)g(library)l(.)28 b(Requests)19 b(to)e(manage)g(the)g(mac)o (hine)i(or)e(pro)q(cesses)h(ma)o(y)189 2176 y(b)q(e)e(in)o(tercepted)g (and)f(handled)i(b)o(y)e(an)g(external)h(resource)f(manager.)143 2270 y Fn(\017)23 b Ff(Net)o(w)o(ork)10 b(of)i(w)o(orkstations)f (managed)i(b)o(y)e(a)i(load)g(balancing)h(system)p Fq(.)j(A)10 b(load)h(balanc-)189 2326 y(ing)h(system)f(ma)o(y)f(c)o(ho)q(ose)i(the) f(lo)q(cation)h(of)f(spa)o(wned)h(pro)q(cesses)f(based)h(on)f(dynamic)i (quan)o(tities,)189 2382 y(suc)o(h)19 b(as)f(load)h(a)o(v)o(erage.)29 b(It)18 b(ma)o(y)g(transparen)o(tly)g(migrate)h(pro)q(cesses)g(from)e (one)i(mac)o(hine)h(to)189 2439 y(another)14 b(when)i(a)f(resource)g(b) q(ecomes)h(una)o(v)m(ailable.)143 2533 y Fn(\017)23 b Ff(Large)j(SMP)g(with)g(Unix)p Fq(.)43 b(Applications)24 b(are)f(run)g(directly)h(b)o(y)e(the)h(user.)42 b(They)23 b(are)189 2589 y(sc)o(heduled)15 b(at)d(a)h(lo)o(w)g(lev)o(el)h(b)o(y)f (the)g(op)q(erating)h(system.)k(Pro)q(cesses)13 b(ma)o(y)g(ha)o(v)o(e)g (sp)q(ecial)h(sc)o(hedul-)189 2646 y(ing)g(c)o(haracteristics)h (\(gang-sc)o(heduling,)g(pro)q(cessor)g(a\016nit)o(y)l(,)f(deadline)i (sc)o(heduling,)g(pro)q(cessor)-32 46 y Fj(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 4 6 4 5 bop 75 -100 a Fq(4)951 b Fk(CHAPTER)16 b(1.)34 b(D)o(YNAMIC)15 b(PR)o(OCESSES)189 45 y Fq(lo)q(c)o(king,)i(etc.\))24 b(and)16 b(b)q(e)i(sub)s(ject)e(to)g(OS)h(resource)f(limits)i(\(n)o(um) o(b)q(er)f(of)f(pro)q(cesses,)h(amoun)o(t)e(of)189 102 y(memory)l(,)f(etc.\).)166 203 y(MPI)f(assumes,)g(implicitly)l(,)k(the) c(existence)h(of)f(an)g(en)o(vironmen)o(t)h(in)g(whic)o(h)g(an)f (application)i(runs.)75 260 y(It)k(do)q(es)g(not)g(pro)o(vide)g(\\op)q (erating)g(system")f(services,)i(suc)o(h)g(as)e(a)h(general)g(abilit)o (y)h(to)e(query)i(what)75 316 y(pro)q(cesses)i(are)f(running,)j(to)c (kill)k(arbitrary)c(pro)q(cesses,)j(to)e(\014nd)h(out)f(prop)q(erties)h (of)f(the)h(run)o(time)75 373 y(en)o(vironmen)o(t)d(\(ho)o(w)f(man)o(y) g(pro)q(cessors,)g(ho)o(w)g(m)o(uc)o(h)h(memory)l(,)f(etc\).)30 b(Complex)19 b(in)o(teraction)g(of)f(an)75 429 y(MPI)i(application)h (with)f(its)g(run)o(time)h(en)o(vironmen)o(t)f(should)h(b)q(e)f(done)g (through)g(an)g(en)o(vironmen)o(t-)75 486 y(sp)q(eci\014c)d(API.)166 625 y Fe(Discussion:)52 b Fd(An)19 b(example)e(of)h(suc)o(h)h(an)g(API) g(w)o(ould)e(b)q(e)i(the)h(PVM)f(task)f(and)h(mac)o(hine)e(manage-)75 681 y(men)o(t)e(routines)i({)f Fc(pvm)p 450 681 14 2 v 15 w(addhosts)p Fd(,)f Fc(pvm)p 734 681 V 15 w(config)p Fd(,)g Fc(pvm)p 974 681 V 16 w(tasks)p Fd(,)g(etc.,)i(p)q(ossibly)f(mo) q(di\014ed)f(to)h(return)i(an)e(MPI)75 738 y(\(group,rank\))e(when)g(p) q(ossible.)k(A)c(Condor)g(or)g(PBS)g(API)g(w)o(ould)g(b)q(e)g(another)g (p)q(ossibilit)o(y)m(.)166 877 y Fq(A)o(t)i(some)h(lo)o(w)f(lev)o(el,)i (ob)o(viously)l(,)g(MPI)f(m)o(ust)f(b)q(e)h(able)h(to)e(in)o(teract)h (with)g(the)g(run)o(time)g(system,)75 933 y(but)h(the)f(in)o(teraction) h(is)g(not)f(visible)j(at)d(the)h(application)h(lev)o(el)g(and)e(the)h (details)g(are)g(not)f(sp)q(eci\014ed)75 990 y(b)o(y)e(the)g(MPI)h (standard.)166 1129 y Fe(Discussion:)34 b Fd(It)14 b(has)g(b)q(een)h (requested)h(that)e(the)g(MPI)g(forum)e(sp)q(ecify)j(this)f(in)o (terface)g(explicitly)m(.)j(MPI)75 1185 y(has)f(so)g(far)g(shied)h(a)o (w)o(a)o(y)e(from)f(sp)q(ecifying)i(implemen)o(tation)d(details,)j(for) f(go)q(o)q(d)h(reasons.)26 b(It)16 b(w)o(ould)f(b)q(e)i(nice,)75 1242 y(ho)o(w)o(ev)o(er,)d(if)f(implemen)o(tors)e(agreed)k(on)f(a)f (minim)o(al)e(standard)j(in)o(terface)g(to)g(allo)o(w)e(in)o(terop)q (erabilit)o(y)h(-)h(w)o(cs)166 1381 y Fq(MPI)f(do)q(es)g(not)f(require) i(the)f(existence)h(of)e(an)h(underlying)h(\\virtual)f(mac)o(hine")h (mo)q(del,)g(in)f(whic)o(h)75 1437 y(there)h(is)g(a)g(consisten)o(t)g (global)g(view)h(of)e(an)h(MPI)g(application,)h(and)f(an)g(implicit)i (\\op)q(erating)e(system")75 1494 y(managing)k(resources)f(and)h(pro)q (cesses.)28 b(F)l(or)17 b(instance,)i(pro)q(cesses)f(spa)o(wned)g(b)o (y)g(one)f(task)g(ma)o(y)g(not)75 1550 y(b)q(e)i(visible)h(to)e (another;)h(additional)h(hosts)e(added)h(to)e(the)i(run)o(time)f(en)o (vironmen)o(t)h(b)o(y)f(one)h(pro)q(cess)75 1607 y(ma)o(y)13 b(not)h(b)q(e)h(visible)i(in)e(another)e(pro)q(cess;)i(tasks)e(spa)o (wned)i(b)o(y)f(di\013eren)o(t)g(pro)q(cesses)h(ma)o(y)e(not)h(b)q(e)h (au-)75 1663 y(tomatically)g(distributed)h(o)o(v)o(er)d(a)o(v)m (ailable)k(resources.)i(MPI)c(do)q(es)f(require,)h(ho)o(w)o(ev)o(er,)f (that)g(a)g(pro)q(cess)75 1720 y(b)q(e)k(a)o(w)o(are)e(of)h(its)g(o)o (wn)g(c)o(hanges)h(to)e(the)i(run)o(time)f(en)o(vironmen)o(t.)27 b(F)l(or)17 b(instance,)h(it)g(can)f(b)q(e)h(noti\014ed)75 1776 y(when)e(its)f(c)o(hildren)i(die;)f(tasks)e(whic)o(h)i(it)g(spa)o (wns)f(will)i(b)q(e)e(distributed)i(in)f(some)f(sane)g(manner,)g(etc.) 189 1867 y Fg(A)n(dvic)n(e)22 b(to)h(implementors.)86 b Fq(A)23 b(virtual)g(mac)o(hine)h(abstraction)e(is)h(appropriate)g(in) h(most)189 1923 y(en)o(vironmen)o(ts)15 b(and)g(should)g(b)q(e)h(part)e (of)g(a)h(\\high-qualit)o(y")h(implemen)o(tation.)21 b(In)15 b(other)f(cases,)189 1980 y(pro)o(viding)19 b(the)g(virtual)g (mac)o(hine)g(abstraction)f(ma)o(y)g(add)g(to)g(m)o(uc)o(h)h(baggage)e (\(e.g.,)h(daemon)189 2036 y(pro)q(cesses\))d(or)g(inhibit)i(p)q (ortabilit)o(y)l(.)k(\()p Fg(End)16 b(of)g(advic)n(e)g(to)h (implementors.)p Fq(\))166 2127 y(In)o(teraction)12 b(b)q(et)o(w)o(een) g(MPI)g(and)g(the)f(run)o(time)i(en)o(vironmen)o(t)f(is)g(limited)i(to) d(the)h(follo)o(wing)g(areas:)143 2229 y Fn(\017)23 b Fq(A)15 b(pro)q(cess)g(ma)o(y)g(start)f(new)h(pro)q(cesses)h(with)f Fm(MPI)p 1082 2229 V 16 w(SP)l(A)-5 b(WN)p Fq(,)16 b Fm(MPI)p 1359 2229 V 16 w(ST)l(ART)p 1517 2229 V 17 w(A)l(TT)l(A)o(CH)p Fq(,)g(or)189 2285 y Fm(MPI)p 274 2285 V 15 w(ST)l(ART)p 431 2285 V 18 w(INDEPENDENT)p Fq(.)e(It)h(ma)o(y)e(kill)k(pro)q(cesses) d(it)h(has)f(spa)o(wned)h(\(and)f(p)q(ossibly)i(oth-)189 2342 y(ers\))e(and)i(ma)o(y)e(b)q(e)i(noti\014ed)g(when)g(pro)q(cesses) g(die)g(or)e(b)q(ecome)i(unreac)o(hable.)189 2499 y Fe(Discussion)o(:) 38 b Fd(W)m(e)14 b(don't)g(ha)o(v)o(e)h Fb(MPI)p 815 2499 13 2 v 14 w(ALLOCA)m(TE)e Fd(an)o(y)i(more,)e(since)i(the)h(run)o (time)d(en)o(vironmen)o(t)h(has)189 2555 y(dissolv)o(ed)k(in)o(to)g (the)h(bac)o(kground)g(\(i.e.,)f(w)o(e)h(ha)o(v)o(e)f(no)g(handle)h(to) f(it\).)32 b(Resources)20 b(can)f(b)q(e)g(allo)q(cated)189 2612 y(through)14 b(the)g(run)o(time-en)o(vironmen)o(t)e(API,)h(or)h (through)g Fc(mpirun)p Fd(.)1967 46 y Fj(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 5 7 5 6 bop 75 -100 a Fk(1.2.)34 b(THE)15 b(MPI-2)g(D)o(YNAMIC)g(PR)o (OCESS)h(MODEL)775 b Fq(5)143 45 y Fn(\017)23 b Fq(When)c(a)g(pro)q (cess)g(spa)o(wns)f(a)h(c)o(hild)i(pro)q(cess,)e(it)g(optionlly)i (passes)e(a)f(string)h(to)g(the)g(run)o(time)189 102 y(en)o(vironmen)o(t)14 b(to)g(indicate)h(where)g(the)f(pro)q(cess)g (should)i(b)q(e)e(started.)19 b(The)c(string)f(is)g(opaque)h(to)189 158 y(MPI.)143 252 y Fn(\017)23 b Fq(A)13 b(prede\014ned)i(atrribute,)f Fm(MPI)p 743 252 14 2 v 16 w(UNIVERSE)p 973 252 V 17 w(SIZE)p Fq(,)f(of)g Fm(MPI)p 1241 252 V 16 w(COMM)p 1399 252 V 16 w(W)o(ORLD)h Fq(can)f(b)q(e)i(queried)189 308 y(to)f(\014nd)i(out)f(ho)o(w)g(man)o(y)g(pro)q(cesses)g(can)h (usefully)h(b)q(e)f(started)e(altogether.)20 b(One)c(can)f(subtract)189 365 y(the)d(size)i(of)e Fm(MPI)p 482 365 V 16 w(COMM)p 640 365 V 16 w(W)o(ORLD)h Fq(from)f(this)h(v)m(alue)g(to)f(\014nd)i (out)e(ho)o(w)g(man)o(y)g(pro)q(cesses)h(migh)o(t)189 421 y(usefully)k(b)q(e)e(started)g(in)h(addition)g(to)f(those)g (already)g(running.)166 603 y Fe(Discussion:)41 b Fd(It)16 b(seems)f(that)h(a)f(pro)q(cess)j(should)d(b)q(e)i(able)e(to)h(query)g (the)g(run)o(time)e(en)o(vironmen)o(t)h(in)g(a)75 653 y(v)o(ery)g(simple)e(w)o(a)o(y)m(.)20 b(The)15 b(main)e(thing)h(I'm)f (w)o(orried)i(ab)q(out)f(is)h(that)g(90\045)f(of)g(\\dynamic")f (programs)g(will)g(w)o(an)o(t)75 703 y(to)j(kno)o(w)f(only)g(ho)o(w)g (man)o(y)f(pro)q(cesses)19 b(to)c(spa)o(wn)h(\(i.e.,)f(ho)o(w)g(man)o (y)f(a)o(v)n(ailable)g(pro)q(cessors\).)26 b(The)16 b(next)g(most)75 753 y(common)c(thing)i(ma)o(y)f(b)q(e)j(ho)o(w)e(man)o(y)f(hosts)i(are) h(there)g(of)e(a)g(giv)o(en)h(arc)o(hitecture.)22 b(The)16 b(next)f(migh)o(t)e(b)q(e)i(what)75 803 y(are)g(their)g(names.)20 b(A)15 b(slipp)q(ery)g(slop)q(e.)21 b(It)15 b(w)o(ould)f(b)q(e)h(nice)h (to)e(do)h(a)o(w)o(a)o(y)e(with)i(all)e(resource)k(issues,)f(but)f(w)o (ould)75 853 y(w)o(e)f(wind)g(up)g(with)f(zero)i(p)q(ortabilit)o(y?)i (-)c(w)o(cs)166 902 y(My)f(p)q(osition)f(on)h(the)g(slipp)q(ery)g(slop) q(e)g(is)g(on)g(the)g(\014rst)h(step:)18 b(the)13 b(pro)o(vision)e(of)g (the)h Fb(MPI)p 1566 902 13 2 v 15 w(UNIVERSE)p 1779 902 V 13 w(SIZE)75 952 y Fd(attribute.)19 b(-)13 b(RL)75 1157 y Fi(1.2.3)49 b(Applications)18 b(Requiring)f(Direct)e (Communication)i(with)g(the)e(Runtime)h(System)75 1242 y Fq(The)f(existing)g(MPI)g(sp)q(eci\014cation)i(is)e(adequate)f(for)g (most)g(parallel)i(applications.)21 b(In)16 b(these)e(applica-)75 1299 y(tions,)i(run)o(time)h(en)o(vironmen)o(t,)f(whether)g(simple)i (or)d(elab)q(orate,)h(allo)q(cates)h(resources)f(and)h(manages)75 1355 y(user)k(pro)q(cesses)g(without)g(in)o(teracting)g(with)g(the)g (application)h(program.)36 b(In)21 b(other)g(applications,)75 1412 y(ho)o(w)o(ev)o(er,)c(it)h(is)f(necessary)h(that)f(the)g Fg(user)i(level)i Fq(of)c(the)g(application)i(comm)o(unicate)f(with)g (the)g(run-)75 1468 y(time)c(en)o(vironmen)o(t.)19 b(Here)14 b(w)o(e)f(describ)q(e)i(three)e(broad)g(classes)h(of)f(suc)o(h)h (applications.)21 b(In)14 b(Section)g(1.5)75 1525 y(w)o(e)h(will)i(giv) o(e)e(concrete)h(examples)g(of)e(eac)o(h)i(of)e(these)i(classes.)75 1645 y Fm(T)l(ask)c(F)o(a)o(rming)43 b Fq(By)11 b(a)g(\\task)f(farm")g (application)i(w)o(e)f(mean)g(a)f(program)g(that)g(manages)h(the)g (execution)75 1701 y(of)j(a)g(set)g(of)g(other,)g(p)q(ossibly)i(sequen) o(tial,)g(programs.)i(This)d(situation)g(often)f(arises)h(when)g(one)f (w)o(an)o(ts)75 1758 y(to)19 b(run)h(the)g(same)f(sequen)o(tial)i (program)e(man)o(y)g(times)h(with)g(v)m(arying)h(input)f(data.)33 b(W)l(e)20 b(call)h(eac)o(h)75 1814 y(in)o(v)o(o)q(cation)14 b(of)e(the)i(sequen)o(tial)g(program)e(a)h Fg(task)p Fq(.)19 b(It)13 b(is)h(often)f(simplest)h(to)f(\\parallelize")i(the)e (existing)75 1871 y(sequen)o(tial)h(program)d(b)o(y)i(writing)g(a)f (parallel)j(\\harness")d(program)f(that)h(in)i(turn)e(dev)o(otes)h(a)f (separate,)75 1927 y(transien)o(t)j(pro)q(cess)g(to)g(eac)o(h)g(task.)k (When)d(one)f(task)f(\014nishes,)i(a)f(new)g(pro)q(cess)h(is)f(started) g(to)f(execute)75 1983 y(the)19 b(next)f(one.)30 b(Ev)o(en)19 b(if)g(the)f(resources)h(allo)q(cated)g(to)f(the)h(job)f(are)g (\014xed,)i(the)f(\\harness")f(pro)q(cess)75 2040 y(m)o(ust)13 b(in)o(teract)g(frequen)o(tly)h(with)g(the)f(pro)q(cess)h(manager)f (\(ev)o(en)g(if)h(this)g(is)g(just)f Fh(rsh)p Fq(,)g(to)g(start)f(the)i (new)75 2096 y(pro)q(cesses)i(with)f(the)h(new)f(input)h(data\).)j(In)d (man)o(y)f(cases)g(this)h(harness)f(can)g(b)q(e)h(written)g(in)g(a)e (simple)75 2153 y(scripting)k(language)f(lik)o(e)g Fh(csh)g Fq(or)f Fh(perl)p Fq(,)g(but)h(some)f(users)h(prefer)g(to)f(use)h(F)l (ortran)e(or)h(C.)g(Note)h(that)75 2209 y(it)f(is)g(an)g(explicit)i (goal)d(of)h(the)f(MPI)h(dynamic)h(pro)q(cess)f(arc)o(hitecture)g(to)f (allo)o(w)h(the)f(managemen)o(t)g(of)75 2266 y(non-MPI)h(pro)q(cesses.) 75 2386 y Fm(Dynamic)f(numb)q(er)i(of)e(p)o(ro)q(cesses)j(in)e(pa)o (rallel)f(job)46 b Fq(The)16 b(program)f(wishes)i(to)e(decide)j Fg(inside)g Fq(the)e(pro-)75 2442 y(gram)e(to)g(adjust)g(the)g(n)o(um)o (b)q(er)h(of)f(pro)q(cesses)h(to)f(\014t)h(the)f(size)i(of)e(the)h (problem.)20 b(F)l(urthermore,)14 b(it)h(ma)o(y)75 2499 y(con)o(tin)o(ue)k(to)g(add)g(and)g(subtract)f(pro)q(cesses)h(during)h (the)f(computation)g(to)f(\014t)h(separate)f(phases)h(of)75 2555 y(the)g(computation,)h(some)f(of)g(whic)o(h)h(ma)o(y)f(b)q(e)h (more)f(parallel)i(than)e(others.)32 b(In)20 b(order)f(to)g(do)g(this,) 75 2612 y(the)i(application)i(program)d(will)j(ha)o(v)o(e)e(to)g(in)o (teract)g(with)h(the)f(resource)g(manager)g(\(ho)o(w)o(ev)o(er)f(it)h (is)75 2668 y(implemen)o(ted\))e(to)d(request)i(and)f(acquire)h(or)f (return)g(computational)h(resources.)26 b(It)17 b(will)i(also)e(ha)o(v) o(e)-32 46 y Fj(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 6 8 6 7 bop 75 -100 a Fq(6)951 b Fk(CHAPTER)16 b(1.)34 b(D)o(YNAMIC)15 b(PR)o(OCESSES)75 45 y Fq(to)h(in)o(teract)h(with)g(the)g(pro)q(cess)g (manager)g(to)f(request)h(that)f(pro)q(cesses)h(b)q(e)h(started)e(and)h (in)h(order)f(to)75 102 y(mak)o(e)12 b(the)g(new)h(pro)q(cesses)f(kno)o (wn)g(to)g(the)g(message-passing)g(library)h(so)f(that)g(the)g(larger)g (\(or)g(smaller\))75 158 y(group)j(of)g(pro)q(cesses)g(can)h(comm)o (unicate.)166 214 y(An)21 b(imp)q(ortan)o(t)f(t)o(yp)q(e)h(of)f (dynamic)h(application)i(is)e(a)f(sca)o(v)o(enger)g(application.)38 b(A)20 b(sca)o(v)o(enger)75 271 y(application)j(is)g(\\em)o (barassingly)f(parallel")h(in)f(the)g(sense)g(that)f(it)h(p)q(erforms)g (a)f(large)h(n)o(um)o(b)q(er)g(of)75 327 y(completely)17 b(indep)q(enden)o(t)i(tasks.)i(If)c(the)f(n)o(um)o(b)q(er)g(of)g(tasks) f(is)i(large)f(enough,)g(suc)o(h)g(an)g(application)75 384 y(can)k(mak)o(e)g(use)h(of)f(an)o(y)g(resources)g(that)g(b)q(ecome) h(a)o(v)m(ailable.)37 b(Con)o(v)o(ersely)l(,)21 b(it)g(can)g(easily)g (giv)o(e)f(up)75 440 y(resources)i(to)g(another)g(application.)42 b(Sca)o(v)o(enger)22 b(applications)i(are)e(excellen)o(t)i(for)e (\014lling)i(in)f(the)75 497 y(\\holes")15 b(on)g(a)g(space-shared)h (parallel)h(mac)o(hine,)f(allo)o(wing)g(it)f(to)g(ac)o(hiev)o(e)g(v)o (ery)g(high)h(utilization.)75 617 y Fm(Client/Server)46 b Fq(This)15 b(situation)h(is)f(the)g(opp)q(osite)h(of)e(the)h (situations)g(ab)q(o)o(v)o(e,)g(where)g(pro)q(cesses)g(come)75 673 y(and)21 b(go)f(up)q(on)h(request.)37 b(In)21 b(the)g(clien)o (t/serv)o(er)g(mo)q(del,)i(one)e(set)f(of)g(pro)q(cesses)h(is)g (relativ)o(ely)h(p)q(er-)75 730 y(manen)o(t)17 b(\(the)g(serv)o(er,)g (whic)o(h)h(w)o(e)f(assume)g(here)h(ma)o(y)e(b)q(e)i(a)f(parallel)i (program\).)24 b(A)o(t)17 b(unpredictable)75 786 y(times,)e(another)f (\(p)q(ossibly)h(parallel\))h(program)d(\(the)i(clien)o(t\))g(b)q (egins)h(execution)f(and)g(m)o(ust)f(establish)75 843 y(comm)o(unication)k(with)f(the)g(serv)o(er.)26 b(In)17 b(this)h(case)f(the)g(pro)q(cess)g(manager)g(m)o(ust)f(pro)o(vide)i(a)f (w)o(a)o(y)f(for)75 899 y(the)e(clien)o(t)h(to)e(lo)q(cate)h(the)g (serv)o(er)f(and)h(comm)o(unicate)g(to)f(the)g(message-passing)h (library)h(that)e(it)h(m)o(ust)75 956 y(no)o(w)h(supp)q(ort)g(comm)o (unications)h(with)f(a)g(new)h(collection)h(of)d(pro)q(cesses.)166 1012 y(It)19 b(is)g(curren)o(tly)h(p)q(ossible)g(to)f(write)g(the)g (parallel)h(clien)o(ts)g(and)f(serv)o(ers)g(in)g(MPI,)g(but)g(b)q (ecause)75 1068 y(MPI)g(do)q(es)g(not)g(pro)o(vide)h(the)f(necessary)g (in)o(terfaces)g(b)q(et)o(w)o(een)h(the)f(application)i(program)d(and)h (the)75 1125 y(resource)f(manager)g(or)g(pro)q(cess)h(manager,)f(other) g(nonp)q(ortable,)h(mac)o(hine)g(sp)q(eci\014c)i(libraries)f(m)o(ust)75 1181 y(b)q(e)15 b(called)h(in)f(order)f(for)g(the)g(clien)o(t)h(and)g (serv)o(er)f(to)f(comm)o(unicate)i(with)g(one)f(another.)19 b(On)c(the)f(other)75 1238 y(hand,)g(MPI)g(do)q(es)g(con)o(tain)g(sev)o (eral)f(features)h(that)f(mak)o(e)g(it)h(relativ)o(ely)h(easy)e(to)g (add)h(suc)o(h)g(in)o(terfaces,)75 1294 y(and)h(w)o(e)g(prop)q(ose)h(b) q(oth)f(a)g(simple)h(in)o(terface)g(and)f(a)g(more)g(complex)h(but)f (\015exible)j(one.)75 1437 y Fl(1.3)59 b(Pro)r(cess)19 b(Manager)h(Interface)75 1541 y Fi(1.3.1)49 b(Pro)q(cesses)15 b(in)i(MPI)75 1626 y Fq(A)e(pro)q(cess)h(is)g(represen)o(ted)g(in)g (MPI)f(b)o(y)g(a)g(\(group,)g(rank\))f(pair.)21 b(A)15 b(pro)q(cess)h(ma)o(y)f(or)f(ma)o(y)h(not)g(b)q(e)h(an)75 1683 y(\\MPI)h(pro)q(cess")g(in)h(that)f(it)g(ma)o(y)f(or)h(ma)o(y)g (not)f(call)j Fm(MPI)p 1102 1683 14 2 v 15 w(INIT)p Fq(.)e(One)h(can't) e(comm)o(unicate)i(directly)75 1739 y(with)d(a)f(pro)q(cess)h(\(that)e (requires)j(a)e(comm)o(unicator\),)f(but)i(signals)g(can)g(b)q(e)g(sen) o(t)g(to)e(one.)20 b(Note)14 b(that)g(a)75 1796 y(\(group,)g(rank\))h (iden)o(ti\014cation)i(is)f(not)e(unique)j(b)q(ecause)f(a)f(pro)q(cess) h(ma)o(y)e(b)q(elong)i(to)f(sev)o(eral)g(groups.)166 1935 y Fe(Discussion:)59 b Fd(The)21 b(alternativ)o(e)f(is)g(to)g(ha)o (v)o(e)g(an)h Fb(MPI)p 1114 1935 13 2 v 14 w(Pro)q(cess)f Fd(ob)r(ject.)38 b(A)21 b(stra)o(w)f(v)o(ote)h(sho)o(w)o(ed)f(a)75 1991 y(preference)c(for)e(the)g(\(group,)g(rank\))g(alternativ)o(e.)k (W)m(e)13 b(will)g(regret)i(this.)j({)13 b(RL.)166 2131 y Fq(MPI)i(pro)o(vides)h(no)f(guaran)o(tees)g(on)g(the)g(order)g(of)g (op)q(erations)h(b)q(et)o(w)o(een)f(messages)g(and)h(signals.)75 2187 y(If)e(a)f(message)h(and)g(a)f(signal)h(are)g(sen)o(t)f(to)g(a)h (pro)q(cess,)g(the)g(signal)g(ma)o(y)f(arriv)o(e)h(or)f(b)q(e)h(pro)q (cessed)h(b)q(efore)75 2243 y(the)g(message,)g(dep)q(ending)i(on)e (hardw)o(are)g(and)g(MPI)g(implemen)o(tation)i(details.)166 2383 y Fe(Discussion:)34 b Fd(A)14 b(bad)g(programming)d(practice)k (that)f(has)g(b)q(een)h(seen)g(b)o(y)f(naiv)o(e)g(users)h(of)f(systems) g(that)75 2439 y(pro)o(vide)f(b)q(oth)g(messages)h(and)f(signals)f(is)i (to)f(send)h(a)f(pro)q(cess)i(a)e(message)g(and)g(then)h(signal)e(it)h (to)g(w)o(ak)o(e)g(up)g(and)75 2495 y(receiv)o(e)i(the)g(message.)i (This)d(needs)h(to)f(b)q(e)h(plainly)d(forbidden.)18 b(-)13 b(A)o(G.)1967 46 y Fj(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 7 9 7 8 bop 75 -100 a Fk(1.3.)34 b(PR)o(OCESS)16 b(MANA)o(GER)f(INTERF)-5 b(A)o(CE)916 b Fq(7)75 45 y Fi(1.3.2)49 b(Sta)o(rting)16 b(Pro)q(cesses)f(-)h(Simple)h(Interface)75 131 y Fq(The)e(follo)o(wing) h(routine)g(is)g(the)f(simplest)h(w)o(a)o(y)e(to)h(start)f(MPI)h(pro)q (cesses.)75 282 y Fm(MPI)p 160 282 14 2 v 16 w(SP)l(A)-5 b(WN\(executable,)17 b(a)o(rguments,)e(n,)g(comm,)e(intercomm\))117 359 y Fd(IN)155 b Fm(executable)391 b Fd(executable)15 b(\014le)f(con)o(taining)f(program)f(to)i(b)q(e)g(run)117 433 y(IN)155 b Fm(a)o(rguments)391 b Fd(argumen)o(ts)13 b(for)h(the)g(program)117 508 y(IN)155 b Fm(n)564 b Fd(n)o(um)o(b)q(er) 13 b(of)h(pro)q(cesses)i(to)e(start)117 582 y(IN)155 b Fm(comm)470 b Fd(comm)o(unicator)11 b(of)j(group)f(of)h(spa)o(wning)f (pro)q(cesses)117 657 y(OUT)108 b Fm(intercomm)384 b Fd(in)o(tercomm)o(unicator)7 b(b)q(et)o(w)o(een)k(original)c(group)i (and)g(the)h(newly)905 713 y(spa)o(wned)15 b(group)166 838 y Fm(MPI)p 251 838 V 16 w(SP)l(A)-5 b(WN)15 b Fq(is)g(collectiv)o (e)h(o)o(v)o(er)e Fm(comm)f Fq(and)h(the)h(newly)g(created)g(pro)q (cesses.)20 b(It)14 b(starts)g Fm(n)h Fq(iden-)75 894 y(tical)k(copies)g(of)f Fm(executable)i Fq(in)f(a)f(w)o(a)o(y)f(whic)o (h)i(is)g(determined)g(b)o(y)f(the)h(run)o(time)f(en)o(vironmen)o(t)h (\(i.e.,)75 950 y(the)c(user)g(has)g(no)g(con)o(trol)f(o)o(v)o(er)g (where)i(the)f(pro)q(cesses)g(go,)f(though)h(a)f(high)i(qualit)o(y)g (implemen)o(tation)75 1007 y(will)j(distribute)g(them)f(ev)o(enly)h(o)o (v)o(er)e(a)o(v)m(ailable)i(resources\).)27 b(The)18 b(spa)o(wned)g(pro)q(cesses)g(are)g Fg(r)n(e)n(quir)n(e)n(d)75 1063 y Fq(to)i(call)h Fm(MPI)p 309 1063 V 16 w(INIT)p Fq(,)e(whic)o(h)i(is)g(collectiv)o(e)h(with)f Fm(MPI)p 1025 1063 V 16 w(SP)l(A)-5 b(WN)21 b Fq(in)g(the)f(paren)o(t.)35 b(The)21 b(in)o(tercomm)o(u-)75 1120 y(nicator)h(is)g(kno)o(wn)f(in)i (the)e(c)o(hildren)j(as)d Fm(MPI)p 909 1120 V 16 w(COMM)p 1067 1120 V 16 w(P)l(ARENT)p Fq(.)i(The)f(c)o(hildren)h(ha)o(v)o(e)e (their)h(o)o(wn)75 1176 y Fm(MPI)p 160 1176 V 16 w(COMM)p 318 1176 V 16 w(W)o(ORLD)15 b Fq(whic)o(h)h(is)g(separate)f(from)f (that)h(of)f(the)i(paren)o(t.)166 1315 y Fe(Discussion:)49 b Fd(The)19 b(\\pre-allo)q(cated)e(resources")j(to)e(b)q(e)g(used)h(to) f(run)g(spa)o(wned)g(pro)q(cesses)j(could)c(b)q(e)75 1372 y(sp)q(eci\014ed)g(in)d(a)h(p)q(ortable)h(w)o(a)o(y)e(if)g(w)o(e)i (standardize)g Fc(mpirun)p Fd(.)21 b(A)15 b(set)h(of)e(argumen)o(ts)h (or)g(a)g(\014le)g(argumen)o(t)f(could)75 1428 y(describ)q(e)i(a)d (virtual)g(mac)o(hine.)75 1632 y Fi(1.3.3)49 b(Sta)o(rting)16 b(Pro)q(cesses)f(-)h(F)o(ull)h(Interface)75 1718 y Fq(While)g Fm(MPI)p 293 1718 V 15 w(SP)l(A)-5 b(WN)16 b Fq(is)g(su\016cien)o(t)g (for)f(most)f(cases,)h(it)g(do)q(es)h(not)f(allo)o(w)g(the)g(follo)o (wing:)143 1808 y Fn(\017)23 b Fq(Spa)o(wning)16 b(of)g(non-MPI)g(pro)q (cesses)h(or)e(MPI)i(pro)q(cesses)f(whic)o(h)h(do)f(not)g(wish)g(to)g (\(or)f(need)i(to\))189 1864 y(comm)o(unicate)e(with)h(their)f(paren)o (t.)143 1957 y Fn(\017)23 b Fq(Spa)o(wning)15 b(of)g(m)o(ultiple)i (binaries)g(whic)o(h)f(all)g(ha)o(v)o(e)f(the)g(same)g Fm(MPI)p 1369 1957 V 16 w(COMM)p 1527 1957 V 16 w(W)o(ORLD)p Fq(.)143 2049 y Fn(\017)23 b Fq(Spa)o(wning)15 b(of)g(the)g(same)g (binary)h(with)g(di\013eren)o(t)f(argumen)o(ts.)166 2139 y(An)f(additional)i(complication)g(is)f(that)e(on)i(man)o(y)e(curren)o (t)i(parallel)g(mac)o(hines,)g(comm)o(unication)75 2195 y(b)q(et)o(w)o(een)h(pro)q(cesses)f(spa)o(wned)g(at)g(di\013eren)o(t)g (times)h(ma)o(y)e(b)q(e)i(slo)o(w.)166 2252 y(The)e(general)g(mec)o (hanism)h(for)e(starting)h(pro)q(cesses)g(requires)h(t)o(w)o(o)d (steps.)20 b(First,)13 b(pro)q(cess)h(groups)75 2308 y(are)20 b(initialized)k(with)c(one)h(or)e(more)h(calls)h(to)f Fm(MPI)p 999 2308 V 16 w(GROUP)p 1167 2308 V 18 w(INIT)p Fq(.)f(Eac)o(h)h(of)g(these)g(calls)i(tak)o(es)d(an)75 2365 y(executable,)c(an)e(argumen)o(t)g(list,)h(and)f(a)g(string)h (indicating)h(where)f(the)f(pro)q(cesses)h(should)g(b)q(e)g(started,)75 2421 y(returning)21 b(a)f(group)g(handle.)37 b(The)21 b(string)g(is)g(opaque)f(to)g(MPI)h(but)f(ma)o(y)g(b)q(e)h(meaningful)h (to)e(the)75 2478 y(run)o(time)c(system.)j(Groups)c(can)g(b)q(e)h (joined)g(together)f(using)g(MPI-1)h(group)e(functions.)166 2534 y(The)j(second)h(step)f(starts)e(the)i(pro)q(cesses)h(sp)q (eci\014ed)h(b)o(y)e(a)f(group)h(handle)h(and)f(places)h(them)f(all)75 2591 y(in)f(a)f(single)h Fm(MPI)p 377 2591 V 16 w(COMM)p 535 2591 V 17 w(W)o(ORLD)p Fq(.)166 2647 y(The)10 b(t)o(w)o(o-step)f (pro)q(cess)i(not)e(only)i(allo)o(ws)f(for)g(all)h(the)f(c)o(hildren)i (to)e(ha)o(v)o(e)g(the)g(same)g Fm(MPI)p 1667 2647 V 15 w(COMM)p 1824 2647 V 17 w(W)o(ORLD)p Fq(,)75 2704 y(but)17 b(it)g(also)g(giv)o(es)g(the)h(user)f(the)g(\015exibilit)o(y)i (needed)g(to)d(run)h(applications)i(across)d(a)h(net)o(w)o(ork.)24 b(The)-32 46 y Fj(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 8 10 8 9 bop 75 -100 a Fq(8)951 b Fk(CHAPTER)16 b(1.)34 b(D)o(YNAMIC)15 b(PR)o(OCESSES)75 45 y Fq(simple)i(example)g(is)f(w)o(ere)f (executables)i(m)o(ust)e(b)q(e)h(matc)o(hed)g(with)g(particular)g(w)o (orkstations)e(but)i(the)75 102 y(user)f(also)h(w)o(an)o(ts)e(all)i (the)f(pro)q(cesses)h(on)f(all)h(the)f(w)o(orkstations)f(w)o(orking)h (together.)166 158 y(There)c(are)f(t)o(w)o(o)g(options)h(at)f(the)h (second)g(step.)18 b(If)11 b(the)g(paren)o(t)f(pro)q(cess)h(is)h (starting)e(MPI)h(pro)q(cesses)75 214 y(that)21 b(will)j(comm)o (unicate)e(bac)o(k)f(with)i(the)e(paren)o(t)h(pro)q(cess)g(at)f(some)h (p)q(oin)o(t)g(in)h(the)e(future,)j(then)75 271 y(the)19 b(application)h(should)g(call)g Fm(MPI)p 715 271 14 2 v 16 w(ST)l(ART)p 873 271 V 17 w(A)l(TT)l(A)o(CH)p Fq(.)g(This)f (function)h(blo)q(c)o(ks)f(un)o(til)h(the)f(c)o(hildren)75 327 y(establish)g(an)f(in)o(ter-comm)o(unicator)h(with)f(the)g(paren)o (t)g(group.)29 b(The)18 b(syc)o(hronization)h(o)q(ccurs)f(when)75 384 y(the)12 b(c)o(hildren)j(call)e Fm(MPI)p 486 384 V 16 w(INIT)p Fq(.)e(When)i Fm(MPI)p 827 384 V 16 w(INIT)f Fq(is)h(called,)h(the)e(v)m(arables)h Fm(MPI)p 1456 384 V 16 w(COMM)p 1614 384 V 17 w(W)o(ORLD)f Fq(and)75 440 y Fm(MPI)p 160 440 V 16 w(COMM)p 318 440 V 16 w(P)l(ARENT)17 b Fq(\(in)o(ter-comm)o(unicator\))d(are)h(set)g(in)h(the)f(c)o (hildren.)166 497 y(The)i(adv)m(an)o(tage)f(of)h(using)h Fm(MPI)p 737 497 V 15 w(INIT)f Fq(in)h(this)f(w)o(a)o(y)f(is)h(that)g (MPI-1)f(programs)g(can)h(b)q(e)h(started)75 553 y(with)e(the)f(new)g (MPI-2)g(functions)h(without)f(mo)q(di\014cation)i(to)d(these)i (programs.)166 610 y(The)f(other)f(option)h(in)g(the)f(second)i(step)e (is)h(to)f(call)i Fm(MPI)p 1157 610 V 15 w(ST)l(ART)p 1314 610 V 18 w(INDEPENDENT)p Fq(.)e(This)h(func-)75 666 y(tion)d(do)q(es)f(not)g(w)o(ait)g(for)g(the)g(c)o(hildren)i(and)f (is)g(exp)q(ected)g(to)f(b)q(e)h(used)g(when)g(the)f(c)o(hildren)j(are) d(non-MPI)75 723 y(pro)q(cesses)18 b(or)f(when)i(the)e(c)o(hildren)j (only)e(comm)o(unicate)g(among)g(themselv)o(es)g(and)g(not)f(bac)o(k)h (to)f(the)75 779 y(paren)o(ts.)29 b(In)18 b(the)h(latter)f(case,)g(the) h(MPI)p 817 779 V 16 w(COMM)p 985 779 V 16 w(P)l(ARENT)g(in)o(tercomm)o (unicator)f(is)h(not)f(de\014ned)75 835 y(on)d(return)g(from)g(MPI)p 477 835 V 16 w(INIT.)166 892 y(Note)f(that)f Fm(MPI)p 457 892 V 16 w(GROUP)p 625 892 V 18 w(INIT)g Fq(and)i Fm(MPI)p 915 892 V 16 w(GROUP)p 1083 892 V 17 w(ST)l(ART)p 1242 892 V 18 w(INDEPENDENT)f Fq(are)g(called)i(on)e(a)75 948 y(single)e(pro)q(cess,)g(while)h Fm(MPI)p 565 948 V 16 w(GROUP)p 733 948 V 18 w(ST)l(ART)p 893 948 V 17 w(A)l(TT)l(A)o(CH)f Fq(is)g(collectiv)o(e)g(o)o(v)o(er)f(a)g(paren)o(t) f(comm)o(unicator)75 1005 y(and)15 b(the)h(spa)o(wned)f(pro)q(cesses.) 75 1156 y Fm(MPI)p 160 1156 V 16 w(GROUP)p 328 1156 V 18 w(INIT\(executable,)g(a)o(rguments,)g(n,)g(where,)h(group\))117 1233 y Fd(IN)155 b Fm(executable)391 b Fd(executable)15 b(\014le)f(con)o(taining)f(program)f(to)i(b)q(e)g(run)117 1308 y(IN)155 b Fm(a)o(rguments)391 b Fd(argumen)o(ts)13 b(for)h(the)g(program)117 1383 y(IN)155 b Fm(n)564 b Fd(n)o(um)o(b)q(er)13 b(of)h(pro)q(cesses)i(to)e(start)117 1458 y(IN)155 b Fm(where)477 b Fd(A)13 b(string)f(telling)g(the)h(run)o (time)e(system)h(where)i(to)e(start)h(the)905 1515 y(pro)q(cesses)k (\(opaque)d(to)f(MPI,)h(or)g(NULL\))117 1590 y(OUT)108 b Fm(group)479 b Fd(iden)o(ti\014es)15 b(the)f(group)g(of)f(pro)q (cesses)75 1809 y Fm(MPI)p 160 1809 V 16 w(GROUP)p 328 1809 V 18 w(ST)l(ART)p 488 1809 V 17 w(A)l(TT)l(A)o(CH\(mycomm,)f (group,)j(new)o(comm\))117 1886 y Fd(IN)155 b Fm(mycomm)412 b Fd(comm)o(unicator)11 b(of)j(paren)o(ts)g(group)117 1961 y(IN)155 b Fm(group)479 b Fd(group)14 b(returned)i(b)o(y)d Fb(MPI)p 1326 1961 13 2 v 15 w(GROUP)p 1483 1961 V 13 w(INIT)117 2036 y Fd(OUT)108 b Fm(new)o(comm)397 b Fd(in)o(ter-comm)o (unicator)11 b(b)q(et)o(w)o(een)16 b(paren)o(t)e(and)g(c)o(hild)f (groups)75 2255 y Fm(MPI)p 160 2255 14 2 v 16 w(GROUP)p 328 2255 V 18 w(ST)l(ART)p 488 2255 V 17 w(INDEPENDENT\(group\))117 2333 y Fd(IN)155 b Fm(group)479 b Fd(group)14 b(returned)i(b)o(y)d Fb(MPI)p 1326 2333 13 2 v 15 w(GROUP)p 1483 2333 V 13 w(INIT)75 2522 y Fi(1.3.4)49 b(Pro)q(cess)15 b(Utilities)75 2608 y Fq(A)h(pro)q(cess)g(represen)o(ted)g(b)o(y)g(a)f(\(group,rank\)) f(pair)i(cannot)g(b)q(e)g(comm)o(unicated)h(with)f(directly)l(,)h(un)o (til)75 2665 y(a)d(comm)o(unicator)g(is)h(constructed)f(con)o(taining)i (it.)j(It)c(need)g(not)f(b)q(e)h(an)f(\\MPI)g(pro)q(cess")h(in)g(the)f (sense)1967 46 y Fj(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 9 11 9 10 bop 75 -100 a Fk(1.3.)34 b(PR)o(OCESS)16 b(MANA)o(GER)f(INTERF)-5 b(A)o(CE)916 b Fq(9)75 45 y(that)15 b(it)h(migh)o(t)g(not)f(call)i Fm(MPI)p 601 45 14 2 v 16 w(INIT)p Fq(.)d(On)j(the)f(other)f(hand,)h (it)g(allo)o(ws)g(out-of-band)g(comm)o(unication,)75 102 y(suc)o(h)g(as)e(signals,)i(and)f(migh)o(t)g(b)q(e)h(a)f(useful)h (concept)g(for)f(dealing)h(with)g(failures.)166 158 y(T)l(o)f(ask)g (the)g(pro)q(cess)g(manager)g(to)f(deliv)o(er)j(signals)f(to)f(pro)q (cesses,)g(w)o(e)g(use)75 309 y Fm(MPI)p 160 309 V 16 w(GROUP)p 328 309 V 18 w(SIGNAL\(group,)f(signal\))117 386 y Fd(IN)155 b Fm(group)479 b Fd(group)14 b(of)f(pro)q(cesses)k(to)c (signal)117 461 y(IN)155 b Fm(signal)480 b Fd(signal)13 b(t)o(yp)q(e)h(\(in)o(t\))166 586 y Fq(It)22 b(is)g(the)f(resp)q (onsibilit)o(y)j(of)d(an)h(implemen)o(tation)h(to)e(translate)g(b)q(et) o(w)o(een)h(signals;)j(in)d(other)75 642 y(w)o(ords,)17 b(a)h Fh(SIGINT)f Fq(that)g(has)g(v)m(alue)i Fh(3)f Fq(on)g(system)f(A) h(m)o(ust)f(b)q(e)h(deliv)o(ered)i(as)d(a)h Fh(SIGINT)f Fq(on)g(system)75 699 y(B,)e(ev)o(en)h(if)f(system)g(B)h(uses)f(the)g (v)m(alue)i Fh(5)e Fq(for)g Fh(SIGINT)p Fq(.)f(If)h(the)h(signal)g(can) f(not)g(b)q(e)h(deliv)o(ered)h(b)q(ecause)75 755 y(there)e(is)h(no)e (corresp)q(onding)i(signal,)g(the)f(error)f(co)q(de)i(is)f Fm(MPI)p 1157 755 V 16 w(ERR)p 1258 755 V 17 w(INV)l(ALID)p 1447 755 V 16 w(SIGNAL)p Fq(.)g(In)h(a)e(qualit)o(y)75 812 y(implemen)o(tation,)20 b(the)e(full)h(range)f(of)g(POSIX)h (signals)g(will)h(b)q(e)e(delev)o(erable,)j(but)d(it)g(is)h(mandatory) 75 868 y(that)14 b(at)h(least)g(the)h(KILL)g(signal)g(\()p Fm(MPI)p 769 868 V 16 w(SIGNAL)p 940 868 V 17 w(KILL)p Fq(\))d(b)q(e)j(supp)q(orted.)75 990 y Fi(1.3.5)49 b(Noti\014cation)75 1076 y Fq(There)19 b(needs)g(to)f(b)q(e)h(some)g(w)o(a)o(y)e(of)h (\014nding)i(out)e(when)h(a)g(pro)q(cess)f(\014nishes.)32 b(In)19 b(MPI)g(w)o(e)f(ha)o(v)o(e)g(no)75 1132 y(mec)o(hanism)g(for)g (async)o(hronous)f(noti\014cation.)29 b(Therefore)17 b(the)h(b)q(est)g(w)o(e)g(can)g(do)f(is)i(to)e(construct)g(a)75 1188 y(request)f(that)g(can)h(b)q(e)g(tested)f(and)h(w)o(aited)f(on.)24 b(If)17 b(pro)q(cesses)g(w)o(ere)f(represen)o(ted)h(b)o(y)f(requests,)h (then)75 1245 y(w)o(e)h(could)g(w)o(ait)g(on)g(them)f(directly)l(.)30 b(Since)19 b(w)o(e)f(ha)o(v)o(e)f(in)o(tro)q(duced)i(the)f(notion)g(of) g Fm(MPI)p 1636 1245 V 15 w(Pro)q(cess)p Fq(,)h(w)o(e)75 1301 y(can)c(explicitly)j(attac)o(h)c(a)h(request)h(to)e(it)i(in)g (order)f(to)f(w)o(ait)h(on)g(it.)75 1452 y Fm(MPI)p 160 1452 V 16 w(NOTIFY\(group,)g(n,)g(a)o(rra)o(y)p 641 1452 V 14 w(of)p 692 1452 V 16 w(requests\))117 1530 y Fd(IN)155 b Fm(group)479 b Fd(a)14 b(group)g(of)f(pro)q(cesses)117 1605 y(IN)155 b Fm(n)564 b Fd(n)o(um)o(b)q(er)13 b(of)h(requests)i(in)d (follo)o(wing)e(arra)o(y)117 1680 y(OUT)108 b Fm(a)o(rra)o(y)p 416 1680 V 15 w(of)p 468 1680 V 16 w(requests)272 b Fb(MPI)p 982 1680 13 2 v 15 w(Request)p Fd(s)15 b(to)f(b)q(e)g(tested)i(and/or)d (w)o(aited)h(on)166 1880 y Fe(Discussion:)34 b Fd(This)14 b(used)g(to)g(b)q(e:)166 1930 y(PVM)i(users)g(ha)o(v)o(e)f(found)g (that)h(a)f(\015exible)g(`notify')e(function)i(is)g(imp)q(ortan)o(t)f (to)h(building)f(fault)g(toleran)o(t)75 1980 y(applications.)j(W)m(e)c (prop)q(ose)i(to)f(supply)g(similar)d(functionalit)o(y)h(in)i(MPI.)75 2124 y Fb(MPI)p 152 2124 V 14 w(NOTIFY\(what,)g(n,)f(a)o(rra)o(y)m(,)g (request\))117 2195 y Fd(IN)154 b Fb(what)504 b Fd(a)12 b(\015ag)f(sp)q(ecifying)h(what)f(to)h(b)q(e)g(noti\014ed)g(ab)q(out)g (\(see)h(b)q(elo)o(w\))117 2257 y(IN)154 b Fb(n)567 b Fd(n)o(um)o(b)q(er)13 b(of)h(ob)r(jects)h(in)e(follo)o(wing)e(arra)o(y) 117 2319 y(IN)154 b Fb(a)o(rra)o(y)503 b Fd(arra)o(y)14 b(of)f(MPI)h(ob)r(jects)h(\(dep)q(ends)h(on)e(v)n(alue)f(of)g(`what'\)) 117 2382 y(OUT)107 b Fb(request)465 b Fd(mpifuncMPI)p 1136 2382 V 14 w(Request)15 b(to)e(b)q(e)i(tested)g(and/or)f(w)o(aited) f(on)166 2499 y(Initially)m(,)7 b(w)o(e)j(prop)q(osed)g(that)f(the)h Fb(what)g Fd(can)f(tak)o(e)h(on)f(t)o(w)o(o)g(v)n(alues:)15 b(MPI)p 1306 2499 V 15 w(Pro)q(cess)p 1455 2499 V 17 w(Exit)9 b(and)g(MPI)p 1716 2499 V 16 w(Resource)p 1893 2499 V 16 w(F)m(ailure.)75 2549 y(In)14 b(these)h(cases)g Fb(a)o(rra)o(y)f Fd(con)o(tains)g(MPI)g(Pro)q(cess)i(ob)r(jects)f(and)f (MPI)g(resource)i(ob)r(jects)f(resp)q(ectiv)o(ely)m(.)166 2599 y(These)f(are)g(the)g(t)o(w)o(o)e(most)g(ob)o(vious)h(v)n(alues)g (for)f(`what'.)17 b(Are)d(there)h(other)e(v)n(alues)g(w)o(e)h(w)o(an)o (t)e(to)h(consider?)75 2649 y(PVM)21 b(con)o(tains)g(the)g(equiv)n (alen)o(t)f(of)h(MPI)p 783 2649 V 15 w(Resource)p 959 2649 V 16 w(Add)g(case,)i(but)e(w)o(e)g(ha)o(v)o(e)g(restricted)i(MPI)e (resource)75 2699 y(o)o(wnership)14 b(suc)o(h)h(that)f(the)g(MPI)p 608 2699 V 16 w(Resource)p 785 2699 V 16 w(Add)g(case)h(is)f(not)f(imp) q(ortan)o(t)f(in)i(MPI.)g(-A)o(G)-32 46 y Fj(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 10 12 10 11 bop 75 -100 a Fq(10)928 b Fk(CHAPTER)16 b(1.)34 b(D)o(YNAMIC)15 b(PR)o(OCESSES)166 45 y Fd(I)f(ha)o(v)o(e)g(c)o(hanged) h(it)e(b)q(ecause)j(there)g(are)e(no)g(pro)q(cesses)j(an)o(y)d(more)f (and)h(there)h(is)f(no)g(resource)j(o)o(wnership)75 102 y(or)d(ev)o(en)g(resources)i(at)e(all.)j(Therefore)e(w)o(e)f(can)g(mak) o(e)e(do)h(with)h(the)g(sp)q(ecial)g(case)h(of)e(w)o(aiting)f(for)i (pro)q(cesses)i(to)75 158 y(terminate.)i(I)c(am)f(not)h(sure)h(that)f (this)g(restriction)h(is)f(desirable,)g(ho)o(w)o(ev)o(er,)g(and)g(p)q (erhaps)i(w)o(e)e(should)g(put)g(the)75 214 y(\\what")e(bac)o(k)h(in)o (to)e(the)j(argumen)o(t)d(list)h(for)h(MPI)p 866 214 13 2 v 15 w(NOTIFY)g(as)g(a)f(placeholder)h(for)f(future)h(extensions.) 19 b({)12 b(RL)166 354 y Fq(W)l(e)j(can)g(think)g(of)g(this)g(as)g(MPI) g(in)o(terface)g(to)f(the)h(Pro)q(cess)g(Manager's)e(handling)k(of)d (the)h(signal)75 410 y Fh(SIGCHILD)p Fq(.)e(The)i(exit)g(co)q(de)g (from)f(the)g(pro)q(cess)h(\(from)f(a)g Fh(return)23 b(n)14 b Fq(or)g Fh(exit\(n\))g Fq(in)h(C)f(or)g Fh(STOP)24 b(n)14 b Fq(in)75 467 y(F)l(ortran\))g(can)h(b)q(e)h(retriev)o(ed)g (from)e(the)h Fm(MPI)p 863 467 14 2 v 16 w(status)i Fq(\014lled)g(in)f (the)g Fm(MPI)p 1332 467 V 15 w(W)l(AIT)p Fq(.)166 523 y(Note)h(that)g(this)h(lev)o(el)h(of)e(pro)q(cess)h(managemen)o(t)f (allo)o(ws)h(us)g(to)f(manage)g(non-MPI)h(pro)q(cesses,)75 579 y(since)e(comm)o(unicators)f(are)g(not)g(in)o(v)o(olv)o(ed.)75 723 y Fl(1.4)59 b(A)n(ttaching)19 b(to)g(Indep)r(endent)e(Pro)r(cesses) 75 907 y Fe(Discussion:)38 b Fd(This)14 b(has)h(not)g(b)q(een)h(up)q (dated)g(in)e(sev)o(eral)h(v)o(ersions,)g(and)g(needs)h(w)o(ork.)21 b(In)15 b(particular,)f(older)75 963 y(v)o(ersions)f(had)g(more)f (\015exibilit)o(y)m(,)f(and)h(the)i(non-blo)q(c)o(king)d(v)o(ersions)j (of)e(the)i(calls)e(ma)o(y)f(b)q(e)i(required)h(in)f(order)g(for)75 1020 y(a)h(serv)o(er)h(to)f(accept)h(new)f(clien)o(ts)g(while)g (serving)g(old)f(ones.)166 1159 y Fq(So)e(far)f(w)o(e)h(ha)o(v)o(e)f (co)o(v)o(ered)h(the)g(case)g(of)f(creating)h(new)h(pro)q(cesses.)18 b(F)l(or)11 b(clien)o(t-serv)o(er)h(applications,)75 1215 y(the)i(situation)g(is)g(di\013eren)o(t,)f(b)q(ecause)i(the)e(pro) q(cesses)h(in)h(question)f(already)f(exist,)h(and)g(what)f(w)o(e)g (need)75 1272 y(is)g(a)g(comm)o(unicator)f(to)g(b)q(e)h(used)h(b)o(y)e (them)h(to)f(comm)o(unicate)h(with)g(one)g(another.)18 b(Similar)d(situations)75 1328 y(arise)20 b(in)h(other)f(applications)h (as)f(w)o(ell.)35 b(F)l(or)19 b(example,)j(a)e(visualization)i(to)q(ol) e(ma)o(y)f(w)o(an)o(t)g(to)g(start)75 1385 y(up)e(and)f(attac)o(h)f(to) h(a)f(running)j(sim)o(ulation,)f(or)e(t)o(w)o(o)g(parts)h(of)g(a)f (large)i(application)g(ma)o(y)f(b)q(e)h(started)75 1441 y(separatly)e(at)g(t)o(w)o(o)f(di\013eren)o(t)h(sites)g(and)h(then)f(w) o(an)o(t)f(to)h(comm)o(unicate)h(with)f(eac)o(h)g(other.)166 1498 y(This)22 b(section)g(attempts)f(to)g(pro)o(vide)h(the)g (functions)g(needed)h(to)e(solv)o(e)h(the)g(general)g(case)f(of)75 1554 y(creating)12 b(an)f(in)o(tercomm)o(unicator)g(b)q(et)o(w)o(een)h (t)o(w)o(o)e(MPI)h(pro)q(cesses)h(with)g(no)f(kno)o(wledge)h(of)f(eac)o (h)h(other.)75 1676 y Fi(1.4.1)49 b(Registration)17 b(and)g(Connection) 75 1762 y Fq(The)e(follo)o(wing)h(four)f(functions)h(de\014ne)g(the)g (in)o(terface.)75 1912 y Fm(MPI)p 160 1912 V 16 w(PROCESS)p 373 1912 V 18 w(REGISTER\()g(name,)e(handle\))117 1990 y Fd(IN)155 b Fm(name)485 b Fd(string)14 b(used)h(for)e(con)o(tacting) 117 2065 y(OUT)108 b Fm(handle)465 b Fd(asso)q(ciated)15 b(with)e(the)i(name)166 2189 y Fq(The)i(form)g(of)f(the)i Fm(name)f Fq(argumen)o(t)f(has)h(sev)o(eral)g(p)q(ossibilitie)q(s.)28 b(The)18 b(most)e(ob)o(vious)h(is)h(to)f(use)75 2246 y(the)c Fh(net-address:port-number)d Fq(format)h(that)h(curren)o(t)h (systems)f(will)j(\014nd)e(most)f(straigh)o(tforw)o(ard.)75 2302 y(Ho)o(w)o(ev)o(er,)i(in)i(the)f(long)h(run,)f(name)g(serv)o(ers)g (of)f(v)m(arious)i(kinds)g(ma)o(y)f(require)h(more)f(\015exibilit)o(y)l (.)1967 46 y Fj(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 11 13 11 12 bop 75 -100 a Fk(1.4.)34 b(A)l(TT)l(A)o(CHING)15 b(TO)h(INDEPENDENT)e(PR)o(OCESSES)645 b Fq(11)75 45 y Fm(MPI)p 160 45 14 2 v 16 w(PROCESS)p 373 45 V 18 w(A)o(CCEPT\(mycomm,) 12 b(handle,)k(ro)q(ot,)e(new)o(comm\))117 122 y Fd(IN)155 b Fm(mycomm)412 b Fd(comm)o(unicator)11 b(o)o(v)o(er)j(whic)o(h)g(this) g(call)f(is)h(collectiv)o(e)117 197 y(IN)155 b Fm(handle)465 b Fd(asso)q(ciated)15 b(with)e(registered)j(name)117 273 y(IN)155 b Fm(ro)q(ot)508 b Fd(the)15 b(pro)q(cess)g(that)f (registered)i(the)e(name)117 348 y(OUT)108 b Fm(new)o(comm)397 b Fd(new)12 b(in)o(ter-comm)o(unicator,)d(whic)o(h)i(includes)h(the)g (pro)q(cesses)905 404 y(of)h(the)i(remote)e(group)75 623 y Fm(MPI)p 160 623 V 16 w(PROCESS)p 373 623 V 18 w(CONT)l(A)o(CT\(mycomm,)f(name,)j(new)o(comm\))117 700 y Fd(IN)155 b Fm(mycomm)412 b Fd(comm)o(unicator)11 b(o)o(v)o(er)j (whic)o(h)g(this)g(call)f(is)h(collectiv)o(e)117 775 y(IN)155 b Fm(name)485 b Fd(name)8 b(b)o(y)h(whic)o(h)h(remote)e(pro)q (cess)j(can)f(b)q(e)g(con)o(tacted)g(\(string\))117 851 y(OUT)108 b Fm(new)o(comm)397 b Fd(new)12 b(in)o(ter-comm)o(unicator,)d (whic)o(h)i(includes)h(the)g(pro)q(cesses)905 907 y(of)h(the)i(remote)e (group)75 1126 y Fm(MPI)p 160 1126 V 16 w(FREE)p 285 1126 V 17 w(NAME\()i(handle\))117 1203 y Fd(IN)155 b Fm(handle)465 b Fd(asso)q(ciated)15 b(with)e(registered)j(name)166 1328 y Fq(W)l(e)h(will)h(illustrate)g(the)f(use)g(of)f(these)h (functions)g(with)g(a)g(clien)o(t/serv)o(er)g(example.)25 b(The)17 b(serv)o(er)75 1384 y(w)o(ould)12 b(register)f(a)h(\\name")f (b)o(y)g(whic)o(h)i(it)f(w)o(an)o(ts)e(to)h(b)q(e)h(kno)o(wn.)19 b(The)11 b(Register)i(function)f(w)o(ould)g(return)75 1441 y(an)k(error)g(if)g(there)g(is)h(a)f(name)g(con\015ict.)24 b(The)16 b(serv)o(er)g(group)g(calls)h(MPI)p 1350 1441 V 16 w(PR)o(OCESS)p 1578 1441 V 17 w(A)o(CCEPT)f(and)75 1497 y(the)f(clien)o(t)h(group)f(calls)h(MPI)p 597 1497 V 16 w(PR)o(OCESS)p 825 1497 V 18 w(CONT)l(A)o(CT.)e(The)h(output)g(of) f(these)i(t)o(w)o(o)d(collectiv)o(e)k(calls)75 1553 y(is)j(an)g(in)o (ter-comm)o(unicator)g(b)q(et)o(w)o(een)g(the)f(t)o(w)o(o)g(groups.)33 b(No)o(w)19 b(an)o(y)g(pro)q(cess)h(in)h(the)e(clien)o(t)i(group)75 1610 y(can)g(comm)o(unicate)h(with)f(an)o(y)g(pro)q(cess)h(in)g(the)f (serv)o(er)g(group)g(and)g(vice)i(v)o(ersa,)e(using)h(the)g(in)o(ter-) 75 1666 y(comm)o(unicator.)166 1723 y(Disconnection)11 b(o)q(ccurs)g(when)g(pro)q(cesses)f(call)h Fm(MPI)p 1057 1723 V 16 w(COMM)p 1215 1723 V 17 w(FREE)f Fq(on)g(the)h(in)o(ter-comm) o(unicator.)166 1862 y Fe(Discussion:)30 b Fd(I'm)9 b(not)h(clear)h(ho) o(w)f(the)h(serv)o(er)h(can)f(con)o(tin)o(ue)g(to)f(service)i(clien)o (ts)f(it)f(is)g(already)g(connected)75 1918 y(to,)17 b(and)g(still)f(A)o(CCEPT)h(new)h(clien)o(ts.)27 b(Do)17 b(w)o(e)g(need)h(to)f(mak)o(e)e(A)o(ttac)o(h)i(and)g(Con)o(tact)g (non-blo)q(c)o(king)f(calls?)75 1975 y(-A)o(G)166 2190 y Fe(Discussion:)48 b Fd(A)o(G-)17 b(This)h(is)f(an)g(old)g(discussion) h(but)g(it)f(w)o(as)g(left)h(in)f(b)q(ecause)i(W)o(CS)e(mak)o(es)f(sev) o(eral)75 2240 y(imp)q(ortan)o(t)c(observ)n(ations)i(in)g(it.)166 2290 y(After)h(going)e(through)h(the)g(exercise)i(of)e(replacing)g (connections)h(with)f(pro)q(cesses,)i(I)e(think)g(I)g(understand)75 2339 y(wh)o(y)j(connections)h(w)o(ere)f(prop)q(osed)h(in)f(the)g (\014rst)h(place.)27 b(The)17 b(problem)f(is)h(that)f(y)o(ou)h(ma)o(y)e (w)o(an)o(t)h(to)h(prev)o(en)o(t)75 2389 y(the)g(serv)o(er)h(from)d (blo)q(c)o(king)h(no)g(matter)g(what)g(the)h(clien)o(t)g(do)q(es.)27 b(As)17 b(prop)q(osed)g(here,)h(the)f(serv)o(er)h(will)d(blo)q(c)o(k)75 2439 y(while)i(A)m(TT)m(A)o(CHing)f(if)g(the)i(clien)o(t)f(do)q(esn't)h (do)f(a)g(matc)o(hing)e(A)m(TT)m(A)o(CH.)h(A)h(\\connection")h(w)o(as)f (more)f(than)75 2489 y(a)j(simple)e(pro)q(cess)k(in)e(that)g (establishing)g(an)g(in)o(tercomm)o(unicator)e(from)g(an)i Fb(MPI)p 1447 2489 13 2 v 14 w(Connection)h Fd(\(using)f(RE-)75 2539 y(MOTE)p 205 2539 V 16 w(A)m(TT)m(A)o(CH\))13 b(w)o(as)h(a)f Fa(lo)n(c)n(al)g Fd(op)q(eration,)h(not)g(requiring)f(sync)o (hronization)h(with)g(the)g(remote)g(pro)q(cesses.)166 2589 y(I)f(main)o(tain)d(that)j(this)g(only)f(sw)o(ept)i(the)g(problem) d(under)j(the)g(rug.)k(In)13 b(e\013ect,)h(establishing)f(a)f (connection)75 2638 y(with)g(ICONNECT/IA)o(CCEPT)h(w)o(as)g(a)f(a)g (non)o(blo)q(c)o(king)f(collectiv)o(e)i(op)q(eration)f(that)h (established)g(comm)o(unica-)75 2688 y(tion.)19 b(REMOTE)p 367 2688 V 15 w(A)m(TT)m(A)o(CH)14 b(w)o(as)g(really)g(only)g(a)g(lo)q (cal)f(lo)q(okup)h(of)g(the)h(already-established)f(comm)o(unicator.) -32 46 y Fj(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 12 14 12 13 bop 75 -100 a Fq(12)933 b Fk(CHAPTER)16 b(1.)29 b(D)o(YNAMIC)15 b(PR)o(OCESSES)75 45 y Fd(A)j(more)f(direct)i(solution) e(is)g(\(IMHO\))i(to)f(pro)o(vide)g(a)f(non-blo)q(c)o(king)g(v)o (ersion)h(of)f Fb(MPI)p 1495 45 13 2 v 14 w(PROCESS)p 1692 45 V 14 w(A)m(TT)m(A)o(CH)p Fd(.)75 95 y(This)d(has)g(all)e(the)j (same)e(asso)q(ciated)i(issues)g(of)e(non)o(blo)q(c)o(king)f(collectiv) o(e)i(op)q(erations,)g(but)g(seems)g(cleaner.)166 145 y(More)21 b(generally)m(,)g(a)f(serv)o(er)i(m)o(ust)e(alw)o(a)o(ys)f(a) o(v)o(oid)g(collectiv)o(e)i(comm)o(unicatio)o(n)d(with)i(a)g(clien)o(t) h(\(unless)75 195 y(non)o(blo)q(c)o(king)11 b(collectiv)o(e)h(is)g(a)o (v)n(ailable\))f(and)h(m)o(ust)f(alw)o(a)o(ys)g(use)i(non-blo)q(c)o (king)e(p)q(oin)o(t-to-p)q(oin)o(t)g(comm)o(unication)75 244 y(if)f(it)h(wishes)h(to)g(a)o(v)o(oid)e(deadlo)q(c)o(k)h(due)h(to)f (an)g(unco)q(op)q(erativ)o(e/incorrect)i(clien)o(t.)k(Because)c(there)g (are)f(\(curren)o(tly\))75 294 y(no)f(non-blo)q(c)o(king)e(collectiv)o (e)i(op)q(erations,)g(and)f(b)q(ecause)j(these)f(w)o(ould)e(probably)g (not)h(b)q(e)g(to)q(o)g(useful)f(to)h(a)f(serv)o(er)75 344 y(in)16 b(an)o(y)f(case)i(\(?\))25 b(W)m(e)15 b(ma)o(y)f(w)o(an)o (t)i(to)f(a)o(v)o(oid)g(the)i(issue)f(en)o(tirely)g(and)g(pro)o(vide)g (only)f(a)h(lo)q(cal)f(\(non-collectiv)o(e\))75 394 y(non-blo)q(c)o (king)e(v)o(ersion)h(of)f Fb(MPI)p 586 394 V 14 w(PROCESS)p 783 394 V 13 w(IA)m(TT)m(A)o(CH)g Fd(\(i.e,)g(without)g(the)i Fb(comm)d Fd(and)h Fb(ro)q(ot)h Fd(argumen)o(ts.)j(\).)166 444 y(If)12 b(w)o(e)h(do)f(ha)o(v)o(e)g(a)g(non)o(blo)q(c)o(king)f (collectiv)o(e)h(op)q(eration,)g(it)g(will)f(need)j(to)e(function)g (correctly)i(when)e(sev)o(eral)75 493 y(are)i(o)o(v)o(erlapp)q(ed.)k (Is)d(this)f(p)q(ossible?)75 550 y(w)o(cs)75 776 y Fl(1.5)59 b(Examples)75 877 y Fm(Manager-w)o(o)o(rk)o(er)13 b(example,)i(using)h (SP)l(A)-5 b(WN.)75 971 y Fh(/*)24 b(manager)e(*/)75 1028 y(#include)h()75 1084 y(main\(int,)g(argc,)g(char)g (*arg[]\))75 1140 y({)147 1197 y(MPI_Status)f(status;)147 1253 y(int)h(count,)g(world_size,)f(flag;)147 1310 y(MPI_Comm)g (everyone;)262 b(/*)23 b(intercommunicator)f(*/)147 1423 y(MPI_Init\(&argc,)f(&argv\);)147 1479 y(MPI_Comm_size\(MPI_COMM_)o (WORLD,)f(&world_size\);)147 1536 y(if)j(\(world_size)g(!=)g(1\))242 1592 y(error\("Top)g(heavy)g(with)g(management"\);)147 1649 y(MPI_Attr_get\(MPI_COMM_W)o(ORLD,)e(MPI_UNIVERSE_SIZE,)g(&count,) i(&flag\))147 1761 y(if)g(\(count)g(==)h(1\))242 1818 y(error\("No)f(room)g(to)h(start)f(workers"\);)147 1931 y(MPI_Spawn\("worker",)e(NULL,)i(count-1,)g(MPI_COMM_SELF,)f (&everyone\);)147 2044 y(/*)170 2100 y(*)i(Parallel)f(code)g(here.)g (The)h(communicator)e("everyone")h(can)g(be)h(used)170 2157 y(*)g(to)g(communicate)e(with)h(the)h(spawned)f(processes,)f (which)h(have)h(ranks)f(0,..count-2)170 2213 y(*)h(in)g(the)f(remote)g (group)g(of)h(the)f(intercommunicator)f("everyone".)170 2270 y(*/)147 2382 y(MPI_Finalize\(\);)75 2439 y(})75 2552 y(/*)i(worker)f(*/)75 2665 y(#include)g()1967 46 y Fj(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 13 15 13 14 bop 75 -100 a Fk(1.5.)29 b(EXAMPLES)1398 b Fq(13)75 45 y Fh(main\(int)23 b(argc,)g(char)g(*argv[]\))75 102 y({)147 158 y(MPI_Init\(&argc,)e(&argv\);)147 271 y(/*)170 327 y(*)j(Parallel)f(code)g(here.)170 384 y(*)h(The)g(manager)f(is)g (represented)f(as)i(the)g(process)e(with)i(rank)f(0)h(in)f(\(the)h (remote)170 440 y(*)g(group)f(of\))h(MPI_COMM_PARENT.)45 b(If)24 b(the)f(workers)g(need)h(to)f(communicate)g(among)170 497 y(*)h(themselves,)f(they)g(can)g(either)g(extract)g(the)h(local)f (group)g(of)h(the)170 553 y(*)g(intercommunicator)e(MPI_COMM_PARENT,)f (or)j(use)f(MPI_COMM_WORLD.)46 b(They)170 610 y(*)24 b(should)f(be)h(the)f(same.)170 666 y(*/)147 779 y(MPI_Finalize\(\);)75 835 y(})75 1068 y Fm(Manager-w)o(o)o(rk)o(er)13 b(example,)i(using)h(A) l(TT)l(A)o(CH.)75 1162 y Fh(/*)24 b(manager)e(*/)75 1219 y(#include)h()75 1275 y(main\(int,)g(argc,)g(char)g(*arg[]\))75 1332 y({)147 1388 y(int)g(count,)g(world_size,)f(flag;)147 1445 y(MPI_Comm)g(everyone;)333 b(/*)24 b(intercommunictor)d(*/)147 1501 y(MPI_Group)h(worker_group;)147 1614 y(MPI_Init\(&argc,)f (&argv\);)147 1670 y(MPI_Comm_size\(MPI_COMM_)o(WORLD,)f (&world_size\);)147 1727 y(if)j(\(world_size)g(!=)g(1\))242 1783 y(error\("Top)g(heavy)g(with)g(management"\);)147 1840 y(MPI_Attr_get\(MPI_COMM_W)o(ORLD,)e(MPI_UNIVERSE_SIZE,)g(&count,) i(&flag\);)147 1953 y(if)g(\(count)g(==)h(1\))242 2009 y(error\("No)f(room)g(to)h(start)f(workers"\);)147 2122 y(MPI_Group_init\("worker")o(,)e(NULL,)i(count-1,)g(NULL,)g (&worker_group\);)147 2178 y(MPI_Start_attach\(MPI_CO)o(MM_SELF)o(,)e (0,)j(worker_group,)e(&everyone\);)147 2291 y(/*)170 2348 y(*)i(Parallel)f(code)g(here.)g(The)h(communicator)e("everyone")h (can)g(be)h(used)170 2404 y(*)g(to)g(communicate)e(with)h(the)h (spawned)f(processes,)f(which)h(have)h(ranks)f(0,..count-2)170 2461 y(*)h(in)g(the)f(remote)g(group)g(of)h(the)f(intercommunicator)f ("everyone".)170 2517 y(*/)147 2630 y(MPI_Finalize\(\);)75 2687 y(})-32 46 y Fj(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 14 16 14 15 bop 75 -100 a Fq(14)933 b Fk(CHAPTER)16 b(1.)29 b(D)o(YNAMIC)15 b(PR)o(OCESSES)75 102 y Fh(/*)24 b(worker)f(*/)75 214 y(#include)g()75 271 y(main\(int)g(argc,)g(char)g(*argv[]\)) 75 327 y({)147 384 y(MPI_Init\(&argc,)e(&argv\);)147 497 y(/*)170 553 y(*)j(Parallel)f(code)g(here.)170 610 y(*)h(Manager)f(is)h(represented)e(as)i(the)f(process)g(with)g(rank)h (0)f(in)h(\(the)f(remote)g(group)170 666 y(*)h(of\))g(MPI_COMM_PARENT.) 45 b(The)24 b(workers)f(can)g(communicate)f(with)i(one)f(another)170 723 y(*)h(using)f(MPI_COMM_WORLD.)170 779 y(*/)147 892 y(MPI_Finalize\(\);)75 948 y(})75 1181 y Fm(T)l(ask)16 b(fa)o(rm)d(example.)45 b Fq(This)16 b(example)g(sho)o(ws)e(ho)o(w)h (to)f(manage)h(a)g(set)g(of)g(non-MPI)g(pro)q(cesses.)75 1288 y Fh(#include)23 b()75 1344 y(#define)g(MAXPROC)g(1000)75 1401 y(main\(int)g(argc,)g(char)g(*argv[]\))75 1457 y({)147 1513 y(MPI_Group)f(workers[MAXPROC];)147 1570 y(MPI_Group)g (worker_group;)147 1626 y(MPI_Request)g(obituaries[MAXPROC];)147 1683 y(MPI_Status)g(statuses[MAXPROC];)45 b(/*)24 b(see)f(comment)g (below)g(*/)147 1739 y(int)g(count,)g(nslots,)g(i,)h(deadone;)147 1796 y(char)f(**args;)147 1909 y(ThingToDo)f(*thingstodo;)147 1965 y(int)h(nthingstodo,)f(nthingsdone;)147 2078 y(MPI_Init\(&argc,)f (&argv\);)147 2134 y(MPI_Attr_get\(MPI_COMM_W)o(ORLD,)g (MPI_UNIVERSE_SIZE,)g(&count,)i(&flag\);)147 2191 y(nslots)g(=)147 2247 y(GetThingsToDo\(&thingsto)o(do,)e(nthingstodo\);)147 2360 y(/*)i(need)g(to)h(initialize)f(these)g(for)g(Waitany/all\(\))f (below)h(to)h(work)f(*/)147 2417 y(for)g(\(i)h(=)f(0;)h(i)g(<)f (MAXPROC;)g(i++\))g(obituaries[i])g(=)g(MPI_REQUEST_NULL;)147 2530 y(/*)g(start)g(up)h(initial)f(set)g(of)h(tasks)f(*/)147 2586 y(for)g(\(nthingsdone)f(=)i(0;)g(nthingsdone)e(<)i(nslots)f(&&)g (nthingsdone)g(<)g(nthingstodo;)266 2643 y(nthingsdone++\))147 2699 y({)1967 46 y Fj(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 15 17 15 16 bop 75 -100 a Fk(1.5.)29 b(EXAMPLES)1398 b Fq(15)218 45 y Fh(SetArgs\(thingstodo[nthingsd)o(one],)21 b(&args\);)218 102 y(MPI_Group_init\("cow",)g(args,)i(1,)h(NULL,)f (&workers[nthingsdone]\);)147 158 y(})147 214 y(merge_groups\(workers,) e(&worker_group\);)236 b(/*)24 b(merge)f(singleton)g(groups)1340 271 y(into)g(one)h(big)f(group)g(*/)147 327 y(MPI_Group_start_indepen)o (dent\(wo)o(rker_gro)o(up\);)147 384 y(MPI_Notify\(worker_group)o(,)e (nslots,)i(obituaries\);)147 497 y(/*)g(spawn)g(new)h(ones)f(as)h(old)f (ones)g(finish)h(*/)147 553 y(while\(nthingsdone)d(<)j(nthingstodo\)) 147 610 y({)218 666 y(MPI_Waitany\(nslots,)d(obituaries,)i(&deadone,)f (&status[0]\);)218 723 y(SetArgs\(thingstodo[nthingsd)o(one],)f (&args\);)218 779 y(MPI_Group_init\("cow",)g(args,)i(1,)h (&workers[nthingsdone]\);)218 835 y(MPI_Group_start_independent)o (\(workers)o([nthing)o(sdone]\))o(;)218 892 y (MPI_Notify\(&workers[nthings)o(done],)d(1,)i (&obituaries[nthingsdone]\);)147 948 y(})147 1005 y (MPI_Waitall\(nslots,)e(obituaries,)h(statuses\);)147 1118 y(MPI_Finalize\(\);)75 1174 y(})75 1351 y Fm(PVM-st)o(yle)16 b(SPMD)f(example)45 b Fq(This)16 b(is)g(ho)o(w)f(man)o(y)f(PVM)h (programs)f(are)h(t)o(ypically)i(written.)j(There)75 1407 y(is)h(no)g(reason)g(they)g(can't)f(b)q(e)i(done)f(with)g(MPI-1,)h (but)f(in)h(case)f(users)g(w)o(an)o(t)f(the)h(app)q(earance)g(of)75 1464 y(minimal)c(c)o(hange,)e(here)g(is)h(is.)k(\(V)l(ery)15 b(similar)i(to)d(the)i(manager-w)o(ork)o(er)d(example)j(ab)q(o)o(v)o (e.\))75 1570 y Fh(/*)24 b(myprog)f(*/)75 1683 y(#include)g()75 1739 y(main\(int)g(argc,)g(char)g(*argv[]\))75 1796 y({)147 1852 y(int)g(count,)g(world_size,)f(flag;)147 1909 y(MPI_Comm)g (everyone;)238 b(/*)23 b(intercommunicator)f(*/)147 1965 y(MPI_Comm)g(new_world;)214 b(/*)23 b(corresponding)f (intracommunicator)g(*/)147 2078 y(MPI_Init\(&argc,)f(&argv\);)147 2191 y(if)i(\(MPI_COMM_PARENT)f(==)h(MPI_COMM_NULL\))46 b(/*)24 b(I'm)f(the)h(parent)f(*/)147 2247 y({)218 2304 y(MPI_Attr_get\(MPI_COMM_WORLD)o(,)e(MPI_UNIVERSE_SIZE,)h(&count,)h (&flag\);)218 2360 y(if)h(\(count)f(==)g(1\))314 2417 y(error\("No)f(room)i(to)f(start)g(workers"\);)218 2473 y(MPI_Spawn\("myprog",)e(NULL,)j(count-1,)e(MPI_COMM_SELF,)g (&everyone\);)218 2530 y(MPI_Intercomm_merge\(everyon)o(e,)f(FALSE,)i (&new_world\);)147 2586 y(})147 2643 y(else)218 2699 y(MPI_Intercomm_merge\(MPI_COM)o(M_PARENT)o(,)e(TRUE,)i(&new_world\);) -32 46 y Fj(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 16 18 16 17 bop 75 -100 a Fq(16)933 b Fk(CHAPTER)16 b(1.)29 b(D)o(YNAMIC)15 b(PR)o(OCESSES)170 102 y Fh(/*)24 b(SPMD)f(parallel)g (code)g(here,)h(using)f(new_world)f(instead)h(of)h(MPI_COMM_WORLD)e(*/) 170 214 y(MPI_Finalize\(\);)75 271 y(})75 447 y Fm(Client-server)14 b(example.)44 b Fq(This)14 b(is)f(a)f(simple)j(example;)f(the)f(serv)o (er)f(accepts)h(only)g(a)g(single)h(connection)75 503 y(at)h(a)f(time)i(and)f(serv)o(es)g(that)g(connection)h(un)o(til)g(the) g(clien)o(t)g(requests)f(to)g(b)q(e)h(disconnected.)166 560 y(Here)21 b(is)h(the)g(serv)o(er.)38 b(It)21 b(accepts)h(a)f (single)h(connection)h(and)e(then)h(pro)q(cesses)g(data)e(un)o(til)j (it)75 616 y(receiv)o(es)16 b(a)f(message)g(with)g(tag)f Fh(1)p Fq(.)20 b(A)15 b(message)g(with)h(tag)e Fh(0)h Fq(tells)h(the)g(serv)o(er)e(to)h(exit.)75 706 y Fh(#include)23 b("mpi.h")75 763 y(main\()g(int)h(argc,)f(char)g(**argv)g(\))75 819 y({)75 875 y(MPI_Comm)g(client;)75 932 y(MPI_Status)f(status;)75 988 y(double)h(buf[MAX_DATA];)75 1045 y(int)95 b(again;)75 1158 y(MPI_Init\()23 b(&argc,)g(&argv)g(\);)75 1214 y(while)g(\(1\))h ({)170 1271 y(MPI_Server_connect\()e(MPI_COMM_WORLD,)g("cave:1234",)g (&client)h(\);)170 1327 y(again)h(=)f(1;)170 1384 y(while)h(\(again\))f ({)266 1440 y(MPI_Recv\()g(buf,)g(MAX_DATA,)f(MPI_DOUBLE,)h(0,)g (MPI_ANY_TAG,)505 1496 y(client,)f(&status)h(\);)266 1553 y(switch)g(\(status.tag\))f({)361 1609 y(case)i(0:)f (MPI_Comm_free\()f(&client)h(\);)552 1666 y(MPI_Finalize\(\);)552 1722 y(return)g(0;)361 1779 y(case)h(1:)f(MPI_Comm_free\()f(&client)h (\);)552 1835 y(again)g(=)h(0;)552 1892 y(break;)361 1948 y(case)g(2:)f(/*)h(do)f(something)g(*/)361 2005 y(...)361 2061 y(default:)552 2117 y(MPI_Abort\()g(MPI_COMM_WORLD,)f ("Unexpected)g(message)h(type")g(\);)361 2174 y(})266 2230 y(})170 2287 y(})75 2343 y(})166 2433 y Fq(Here)15 b(is)h(the)f(clien)o(t.)75 2534 y Fh(#include)23 b("mpi.h")75 2591 y(main\()g(int)h(argc,)f(char)g(**argv)g(\))75 2647 y({)75 2704 y(MPI_Comm)g(server;)1967 46 y Fj(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 17 19 17 18 bop 75 -100 a Fk(1.5.)34 b(EXAMPLES)1393 b Fq(17)75 45 y Fh(double)23 b(buf[MAX_DATA];)75 158 y(MPI_Init\()g(&argc,)g (&argv)g(\);)75 214 y(MPI_Client_connect\()e(MPI_COMM_WORLD,)h ("cave:1234",)g(&server)h(\);)75 271 y(while)g(\(!done\))g({)170 327 y(tag)h(=)g(2;)f(/*)h(Action)f(to)g(perform)g(*/)170 384 y(MPI_Send\()g(buf,)g(n,)h(MPI_DOUBLE,)e(0,)i(tag,)f(server)g(\);) 170 440 y(/*)h(etc)f(*/)170 497 y(})75 553 y(MPI_Send\()g(buf,)g(0,)h (MPI_DOUBLE,)e(0,)h(1,)h(server)f(\);)75 610 y(MPI_Comm_free\()f (&client)h(\);)75 666 y(MPI_Finalize\(\);)75 723 y(})166 829 y Fq(If)17 b(the)g(serv)o(er)f(needs)i(to)e(manage)g(m)o(ultiple)i (connections)g(at)e(once,)h(it)g(can)g(use)g Fm(MPI)p 1675 829 14 2 v 16 w(IA)o(CCEPT)75 885 y Fq(instead.)j(The)c(clien)o(t) g(need)g(not)f(b)q(e)h(c)o(hanged.)-32 46 y Fj(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 18 20 18 19 bop 75 377 a Fo(Biblio)q(graph)m(y)75 600 y Fq([1])22 b(Al)d(Geist,)g(Adam)f(Beguelin,)j(Jac)o(k)e(Dongarra,)e(W)l(eic)o (heng)j(Jiang,)g(Bob)e(Manc)o(hek,)h(and)g(V)l(aidy)146 656 y(Sunderam.)g Fg(PVM:)c(Par)n(al)r(lel)g(Virtual)h(Machine|A)g (User's)f(Guide)h(and)g(T)m(utorial)g(for)g(Network)146 713 y(Par)n(al)r(lel)f(Computing)p Fq(.)20 b(MIT)15 b(Press,)g(1994.) 952 2828 y(18)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-comm@CS.UTK.EDU Fri Oct 27 18:46:19 1995 Return-Path: Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id SAA17547; Fri, 27 Oct 1995 18:46:18 -0400 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id SAA11005; Fri, 27 Oct 1995 18:40:49 -0400 Received: from cs.umn.edu by CS.UTK.EDU with ESMTP (cf v2.9s-UTK) id SAA10974; Fri, 27 Oct 1995 18:40:40 -0400 Received: from peca.cs.umn.edu (peca.cs.umn.edu [128.101.224.11]) by mail.cs.umn.edu (8.6.11/8.6.6) with ESMTP From: glo@cs.umn.edu (Gen-Ching Lo) Received: by peca.cs.umn.edu (8.6.11) id RAA11009; Fri, 27 Oct 1995 17:43:33 -0500 Message-Id: <199510272243.RAA11009@peca.cs.umn.edu> Subject: subscription request To: mpi-comm@CS.UTK.EDU, mpi-impl@mcs.anl.gov Date: Fri, 27 Oct 1995 17:43:33 -0500 (CDT) X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Dear Sir/Madam, I am MPI's new user on SGI clusters and would like to receive more information about MPI. Would you please add my name to this discussion list? Thank you very much for your help. Sincerely, Gen-Ching Lo P.S. my email address is glo@cs.umn.edu From mpi-comm-human@mcs.anl.gov Tue Nov 28 10:54:37 1995 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id KAA05027; Tue, 28 Nov 1995 10:54:36 -0500 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id JAA04226 for mpi-comm-out; Tue, 28 Nov 1995 09:44:42 -0600 Received: from donner (donner.mcs.anl.gov [140.221.5.134]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id JAA04158; Tue, 28 Nov 1995 09:42:31 -0600 Date: Tue, 28 Nov 1995 09:42:30 -0600 Message-Id: <199511281542.JAA17171@donner> From: lusk@mcs.anl.gov (Rusty Lusk) To: mpi-comm@mcs.anl.gov, mpi-core@mcs.anl.gov Subject: MPI-2 BOF at SC95 [comp.parallel.mpi #1058] Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk ------ Forwarded Article ------ From lederman@descartes.super.org (Steve Huss-Lederman) This is to announce that there will be a Birds-of-a-Feather about MPI-2 at Supercomputing '95. The session will be from 3:30-5:00 on Wednesday, December 6 in Room 11A. The first half of the session will be an overview of the current thinking in the MPI-2 standards effort; the second half will be public comments and input. To give people interested in the details of the MPI-2 standards effort a chance to review them, the MPI Forum is making the current DRAFT of the MPI-2 document available. It can be gotten from ftp://ftp.mcs.anl.gov/pub/mpi/mpi2/mpi-draft.ps.Z. To give people a flavor of the MPI-2 standards effort, the abstract for the document is attached below. Hope to see you there. The MPI Forum ---------------------------------------------------------------------- This document describes the MPI-2 effort to add extensions to the original MPI standard. The MPI-2 effort began in March 1995 and the additions to the standard are scheduled to be released for public comment at Supercomputing '96. Topics being explored for possible inclusion in MPI-2 are dynamic processes, one-sided communications, extended collective operations, external interfaces, additional language bindings, real-time communications, and miscellaneous topics. NOTE: This is the current state of some of the chapters being drafted for possible inclusion in the MPI-2 standard document. It represents the ongoing work of the MPI Forum in an incomplete and tentative form, but is being distributed with the intention of desseminating the work of the MPI Forum and to allow people attending the BOF at Supercomputing '95 to understand the topics under consideration. The proposals for MPI-2 are still very much under development. It is very likely that the final MPI-2 standard will differ in important ways from this draft. As a result, this draft should not be taken as a promise of the final form of the MPI-2 standard. ------ End of Forwarded Article From mpi-comm-human@mcs.anl.gov Thu Dec 14 18:32:21 1995 return-path: received: from antares.mcs.anl.gov (mcs.anl.gov) by netlib2.cs.utk.edu (cf v2.9t-netlib) id AA00204; Thu, 14 Dec 95 18:19:42 EST received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id QAA27492 for mpi-comm-out; Thu, 14 Dec 1995 16:45:05 -0600 received: from mcs.anl.gov (donner.mcs.anl.gov [140.221.5.134]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id QAA27484; Thu, 14 Dec 1995 16:44:56 -0600 message-id: <199512142244.QAA27484@antares.mcs.anl.gov> To: mpi-io@nas.nasa.gov Cc: mpi-comm@mcs.anl.gov subject: Re: Type contructor in-reply-to: Your message of "Thu, 14 Dec 1995 14:32:19 PST." <199512142229.OAA16848@win24.nas.nasa.gov> date: Thu, 14 Dec 1995 16:44:54 -0600 From: Rusty Lusk Sender: owner-mpi-core@mcs.anl.gov precedence: bulk | I would be delighted if we could get the non-I/O specific type | constructors adopted by MPI2, but my understanding is that MPI2 isn't | going to be released until SC96. In the interim, I think we should | add the type constructors to the MPI-IO draft (with the intention | of deleting them when/if they become part of MPI2). | | Does anyone know who we should be working with to make sure that any | common type constructors have the same syntax? There is a section of the Miscellaneous chapter of the MPI-2 draft (released at Supercomputing '95 and posted on the Web) entitled "Additional type constructors". Currently it has only one new datatype constructor, the "Contiguous struct". The time to propose new datatype constructors for MPI-2 is right now, and the place to propose them is on the MPI-2 discussion list mpi-misc@mcs.anl.gov. As it so happens, I am coordinating the chapter, and can make sure that proposals are included in the ongoing draft and discussed at the MPI-2 meetings, but it is always best if there is an "original" champion for new proposals at the meetings. Rusty Lusk P.S. You can join the above-mentioned list by sending subscribe mpi-misc to majordomo@mcs.anl.gov From owner-mpi-comm@CS.UTK.EDU Fri Mar 8 19:15:32 1996 Return-Path: Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id TAA03242; Fri, 8 Mar 1996 19:15:32 -0500 Received: from localhost by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id TAA01803; Fri, 8 Mar 1996 19:10:50 -0500 Received: from venera.isi.edu by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id TAA01786; Fri, 8 Mar 1996 19:10:47 -0500 Received: from ariel.isi.edu by venera.isi.edu (5.65c/5.61+local-22) id ; Fri, 8 Mar 1996 16:10:38 -0800 Posted-Date: Fri, 08 Mar 1996 16:12:14 -0800 Received: by ariel.isi.edu (5.65c/4.0.3-4) id ; Fri, 8 Mar 1996 16:12:14 -0800 Sender: lantang@ISI.EDU Message-Id: <3140CCDE.41C67EA6@isi.edu> Date: Fri, 08 Mar 1996 16:12:14 -0800 From: Lan Tang Organization: ISI/USC X-Mailer: Mozilla 2.0 (X11; I; SunOS 4.1.3 sun4c) Mime-Version: 1.0 To: mpi-comm@CS.UTK.EDU Subject: subscribe From mpi-comm-human@mcs.anl.gov Thu Mar 14 14:11:56 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id OAA01421; Thu, 14 Mar 1996 14:11:55 -0500 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id MAA09960 for mpi-comm-out; Thu, 14 Mar 1996 12:45:28 -0600 Received: from win24.nas.nasa.gov (win24.nas.nasa.gov [129.99.33.39]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id MAA09949; Thu, 14 Mar 1996 12:45:19 -0600 Received: from localhost (nitzberg@localhost) by win24.nas.nasa.gov (8.6.12/NAS.6.1) with SMTP id KAA15979; Thu, 14 Mar 1996 10:45:04 -0800 Message-Id: <199603141845.KAA15979@win24.nas.nasa.gov> X-Authentication-Warning: win24.nas.nasa.gov: Host localhost didn't use HELO protocol To: mpi-comm@mcs.anl.gov, mpi-core@mcs.anl.gov Cc: mpi-io@nas.nasa.gov Subject: Re: MPI-IO Standards Process In-reply-to: Your message of "Thu, 14 Mar 1996 07:40:00 PST." Date: Thu, 14 Mar 1996 10:45:03 -0800 From: Bill Nitzberg Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk Rusty, How do we pursue proposing the addition of an I/O chapter to the MPI2 forum? - bill Richard Frost writes: > > It appears from your last statement that the MPI-IO group has been more > interested (and driven by real needs) in developing an implementation than > defining a standard. > > If you still hope to be recognized as a chapter in MPI2, why not join the > standards effort now? There are 30 professionals from industry, academia, > and government labs participating in the MPI Forum. > > Rusty Lusk is maintaining a schedule of chapter discussions for the MPI > Forum and has publicly requested that the MPI-IO group join the Forum on > several occasions. > > Given the outcome of last week's MPI Forum meeting, it would appear that > the schedule could accomodate discussion of an MPI-IO chapter in the June, > July, and August meetings. > > I encourage you to pursue this opportunity. > > Richard Frost > SDSC From owner-mpi-comm@CS.UTK.EDU Fri Apr 19 11:27:47 1996 Return-Path: Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id LAA26206; Fri, 19 Apr 1996 11:27:46 -0400 Received: from localhost (root@localhost) by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id LAA05389; Fri, 19 Apr 1996 11:22:45 -0400 Received: from iraun1.ira.uka.de (iraun1.ira.uka.de [129.13.10.90]) by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id LAA05369; Fri, 19 Apr 1996 11:22:36 -0400 Received: from Orion (actually i41a2.ira.uka.de) by iraun1.ira.uka.de with SMTP (PP); Fri, 19 Apr 1996 17:22:29 +0200 Received: from localhost by Orion; (5.65v3.2/1.1.8.2/04Dec95-0429PM) id AA08869; Fri, 19 Apr 1996 17:22:27 +0200 Message-Id: <9604191522.AA08869@Orion> To: mpi-comm@CS.UTK.EDU Cc: blum@ira.uka.de Subject: SUBSCRIBE Date: Fri, 19 Apr 96 17:22:27 +0200 From: "Joachim Blum, IPD, Uni Karlsruhe" X-Mts: smtp SUBSCRIBE blum@ira.uka.de From mpi-comm-human@mcs.anl.gov Thu May 9 11:28:02 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id LAA09595; Thu, 9 May 1996 11:28:01 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id JAA23570 for mpi-comm-out; Thu, 9 May 1996 09:48:08 -0500 Received: from mcs.anl.gov (donner.mcs.anl.gov [140.221.5.134]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id JAA23565; Thu, 9 May 1996 09:48:01 -0500 Message-Id: <199605091448.JAA23565@antares.mcs.anl.gov> To: mpi-comm@mcs.anl.gov, sio-exec@cacr.caltech.edu, sio-hlapi@cacr.caltech.edu, dfk@cs.dartmouth.edu, choudhar@cat.syr.edu, mpi-io@mcs.anl.gov, mpi-io@nas.nasa.gov Subject: MPI Forum takes up I/O Date: Thu, 09 May 1996 09:47:58 -0500 From: Rusty Lusk Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk At its most recent meeting, the MPI Forum voted to start a subcommittee on parallel I/O. This will be a new effort, since I/O was not originally considered to be in the scope of MPI-2. Those interested in participating in the design of an informal standard (like MPI) for high-performance, portable I/O, particularly in an MPI context, are welcome to attend the next meeting of the MPI Forum. The meeting will be the next regularly-scheduled MPI Forum meeting, scheduled for June 5-7 in Chicago. Forum meetings occur every six weeks, in Chicago. Those interested should contact me about details. Rusty Lusk From mpi-comm-human@mcs.anl.gov Fri May 10 15:29:56 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id PAA02699; Fri, 10 May 1996 15:29:53 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id OAA20992 for mpi-comm-out; Fri, 10 May 1996 14:10:39 -0500 Received: from brahma.ticam.utexas.edu (brahma-e.ticam.utexas.edu [128.83.68.51]) by antares.mcs.anl.gov (8.6.10/8.6.10) with SMTP id MAA18845; Fri, 10 May 1996 12:44:18 -0500 Received: from agni.ticam.utexas.edu by brahma.ticam.utexas.edu (AIX 3.2/UCB 5.64/4.03) id AA51433; Fri, 10 May 1996 12:44:45 -0500 Received: by agni.ticam.utexas.edu (AIX 3.2/UCB 5.64/4.03) id AA13400; Fri, 10 May 1996 12:44:09 -0500 Date: Fri, 10 May 1996 12:44:08 -0500 (CDT) From: "Carter Edwards : carter@ticam.utexas.edu" To: mpi-1sided@mcs.anl.gov, mpi-comm@mcs.anl.gov Subject: Request Handlers, please Message-Id: Mime-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk Please post the minutes of the April 24-26, 1996 meeting. The discussion and vote to remove Request Handlers from the MPI2 spec. is of particular interest. It was noted in Chapter 6, External Interfaces, that Request Handlers will reappear in a TBD Chapter on Threads. -- I propose that Request Handler semantics be "orthogonal" to Thread semantics. An MPI user should be able to use Request Handlers without having to be concerned with Thread semantics. A simple suggestion to achieve this "orthogonality": If Threads are present, then the execution of a Request Handler preempts the Thread from which it was posted. A Request Handler could either execute within its posting Thread or block its posting Thread until the Request Handler "returns". If a MPI User wants a particular Handler or set of Handlers to execute in a separate Thread then the MPI User must first create a Thread and then post the Handler(s) within that Thread. -- This baits a larger discussion regarding MPI-2 and Threads. If an MPI implementation internally creates Threads then should these threads: 1) be transparent to the MPI user and 2) not introduce consistency problems in the presence of an MPI user's Threads. ---------------------------------------------------------------------------- Carter Edwards, carter@ticam.utexas.edu, http://www.ticam.utexas.edu/~carter ---------------------------------------------------------------------------- From mpi-comm-human@mcs.anl.gov Fri May 10 16:59:41 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id QAA04769; Fri, 10 May 1996 16:59:40 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id PAA23321 for mpi-comm-out; Fri, 10 May 1996 15:36:38 -0500 Received: from Aurora.CS.MsState.Edu (aurora.cs.msstate.edu [130.18.208.91]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id OAA22098; Fri, 10 May 1996 14:54:06 -0500 Received: from aurora.cs.msstate.edu (aurora.cs.msstate.edu [130.18.208.91]); by Aurora.CS.MsState.Edu using SMTP (SMI-8.6/7.0m-FWP-MsState); id OAA06492; Fri, 10 May 1996 14:53:38 -0500 Date: Fri, 10 May 1996 14:53:38 -0500 (CDT) From: Anthony Skjellum To: "Carter Edwards : carter@ticam.utexas.edu" cc: mpi-1sided@mcs.anl.gov, mpi-comm@mcs.anl.gov Subject: Re: Request Handlers, please In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk Carter, there are MPI implementations that use threads inside. They should certainly be transparent to user process. Windows/NT version we are doing works this way. -Tony On Fri, 10 May 1996, Carter Edwards : carter@ticam.utexas.edu wrote: > Date: Fri, 10 May 1996 12:44:08 -0500 (CDT) > From: Carter Edwards : carter@ticam.utexas.edu > To: mpi-1sided@mcs.anl.gov, mpi-comm@mcs.anl.gov > Subject: Request Handlers, please > > > Please post the minutes of the April 24-26, 1996 meeting. > The discussion and vote to remove Request Handlers from > the MPI2 spec. is of particular interest. > > It was noted in Chapter 6, External Interfaces, that > Request Handlers will reappear in a TBD Chapter on Threads. > > -- > > I propose that Request Handler semantics be "orthogonal" > to Thread semantics. An MPI user should be able to > use Request Handlers without having to be concerned > with Thread semantics. > > A simple suggestion to achieve this "orthogonality": > If Threads are present, then the execution of a > Request Handler preempts the Thread from which it > was posted. A Request Handler could either execute > within its posting Thread or block its posting > Thread until the Request Handler "returns". > > If a MPI User wants a particular Handler or set of Handlers > to execute in a separate Thread then the MPI User must first > create a Thread and then post the Handler(s) within that Thread. > > -- > > This baits a larger discussion regarding MPI-2 and Threads. > If an MPI implementation internally creates Threads then > should these threads: > 1) be transparent to the MPI user and > 2) not introduce consistency problems in the > presence of an MPI user's Threads. > > ---------------------------------------------------------------------------- > Carter Edwards, carter@ticam.utexas.edu, http://www.ticam.utexas.edu/~carter > ---------------------------------------------------------------------------- > > > > Anthony Skjellum, PhD, Asst. Professor of Computer Science; Mississippi State University, Department of Computer Science & NSF ERC Butler, Rm 300, PO Box 9637, Corner of Perry&Barr, Mississippi State,MS 39762 (601)325-8435 FAX: (601)325-8997; http://www.erc.msstate.edu/~tony; Quote: "What a rain of ashes falls on him that sees the new and cannot leave the old."-Shakespeare ; e-mail: tony@cs.msstate.edu; Try MPI! From owner-mpi-comm@CS.UTK.EDU Wed May 15 05:25:38 1996 Return-Path: Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id FAA16692; Wed, 15 May 1996 05:25:38 -0400 Received: from localhost (root@localhost) by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id FAA09232; Wed, 15 May 1996 05:22:44 -0400 Received: from goggins.bath.ac.uk (goggins.bath.ac.uk [138.38.32.13]) by CS.UTK.EDU with ESMTP (cf v2.9s-UTK) id FAA09215; Wed, 15 May 1996 05:22:41 -0400 Received: from maths.Bath.AC.UK (actually host java.maths.bath.ac.uk) by goggins.bath.ac.uk with SMTP (PP); Wed, 15 May 1996 10:22:20 +0100 Date: Wed, 15 May 1996 10:21:18 +0100 (BST) From: Mr J Rodriguez X-Sender: jr@java To: mpi-bugs@mcs.anl.gov, mpi-comm@CS.UTK.EDU, mpi-impl@mcs.anl.gov Subject: SUBSCRIBE Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: jr@maths.bath.ac.uk jr@maths.bath.ac.uk From mpi-comm-human@mcs.anl.gov Fri May 17 17:11:44 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id RAA01146; Fri, 17 May 1996 17:11:43 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id PAA19627 for mpi-comm-out; Fri, 17 May 1996 15:59:13 -0500 Received: from mcs.anl.gov (godzilla.mcs.anl.gov [140.221.5.136]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id PAA19613 for ; Fri, 17 May 1996 15:59:02 -0500 Message-Id: <199605172059.PAA19613@antares.mcs.anl.gov> To: mpi-comm@mcs.anl.gov Subject: Minutes of last meeting are ready Date: Fri, 17 May 1996 15:59:01 -0500 From: William Gropp Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk In the usual place (http://www.mcs.anl.gov/Projects/mpi/mpi2/mpi2.html). Bill From mpi-comm-human@mcs.anl.gov Tue May 28 12:19:12 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id MAA00716; Tue, 28 May 1996 12:19:11 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id LAA23665 for mpi-comm-out; Tue, 28 May 1996 11:01:50 -0500 Received: from mcs.anl.gov (donner.mcs.anl.gov [140.221.5.134]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id LAA23649; Tue, 28 May 1996 11:01:39 -0500 Message-Id: <199605281601.LAA23649@antares.mcs.anl.gov> To: mpi-core@antares.mcs.anl.gov, mpi-comm@antares.mcs.anl.gov Subject: tutorial on Shared memory issues Date: Tue, 28 May 1996 11:01:37 -0500 From: Rusty Lusk Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk Dear group, For those who would like to better understand the issues surrounding some of the discussions in the 1-sided communication subcommittee, I highly recommend the excellent paper Shared Memory Consistency Models: A Tutorial by Sarita Adve and Kourosh Gharachorloo This is quite easy to read and uses as examples just exactly the sorts of things people are going to want to do with put/get. It presents the memory model that most *users* have of shared memory, explains how modern CPUs do not support this model, and what kinds of steps (e.g. fence) need to be taken by programmers (or compilers, or libraries) to reconstruct the model while still allowing optimizations. I wish that I had read this months ago, but it is not too late. It is available from the Web page: http://www.research.digital.com/wrl/techreports/abstracts/95.7.html (A link on that page will get you the postscript of the full 22-page paper.) Regards, Rusty From mpi-comm-human@mcs.anl.gov Tue Jun 18 12:03:34 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id MAA06032; Tue, 18 Jun 1996 12:03:33 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id KAA07051 for mpi-comm-out; Tue, 18 Jun 1996 10:49:10 -0500 Received: from hermes.clab.ee.upatras.gr (hermes.clab.ee.upatras.gr [150.140.174.3]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id GAA03660 for ; Tue, 18 Jun 1996 06:47:36 -0500 Received: from miki.clab.ee.upatras.gr by hermes.clab.ee.upatras.gr with SMTP (1.37.109.16/16.2) id AA080321908; Tue, 18 Jun 1996 14:45:08 +0200 Message-Id: <31C696FC.6238@ee.upatras.gr> Date: Tue, 18 Jun 1996 14:46:05 +0300 From: "Dimitris P. Ioannides" Reply-To: ioannidi@ee.upatras.gr Organization: Computer Science Laboratory - Univeristy Of Patras X-Mailer: Mozilla 3.0b3 (Win95; I) Mime-Version: 1.0 To: mpi-comm@mcs.anl.gov Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk subscribe mpi-coll ioannidi@ee.upatras.gr From mpi-comm-human@mcs.anl.gov Tue Jun 25 17:54:20 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id RAA05127; Tue, 25 Jun 1996 17:54:19 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id QAA18929 for mpi-comm-out; Tue, 25 Jun 1996 16:43:39 -0500 Received: from igw2.watson.ibm.com (igw2.watson.ibm.com [129.34.139.6]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id QAA18924 for ; Tue, 25 Jun 1996 16:43:29 -0500 Received: from mailhub1.watson.ibm.com (mailhub1.watson.ibm.com [9.2.249.31]) by igw2.watson.ibm.com (8.7.4/8.7.1) with ESMTP id RAA23171 for ; Tue, 25 Jun 1996 17:43:50 -0400 Received: from watngi02.watson.ibm.com (watngi02.watson.ibm.com [9.2.235.124]) by mailhub1.watson.ibm.com (8.7.1/03-28-96) with SMTP id RAA53182 for ; Tue, 25 Jun 1996 17:43:26 -0400 Received: by watngi02.watson.ibm.com (IBM OS/2 SENDMAIL VERSION 1.3.17/03-09-96 ) id AA5425; Tue, 25 Jun 96 17:43:23 -0400 Message-Id: <9606252143.AA5425@watngi02.watson.ibm.com> Received: from IBM Research with "Lotus Notes Mail Gateway for SMTP" id CB5F00C5E17370A7852563540076DF35; Tue, 25 Jun 96 17:43:17 To: Barry Smith Cc: mpi-comm From: Marc Snir/Watson/IBM Research Date: 25 Jun 96 17:41:09 Subject: Re: CHANGES to MPI standard are damaging Mime-Version: 1.0 Content-Type: Text/Plain Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk No syntactical changes to MPI1 are under consideration. The basic rule is that MPI1 is a standard. MPI2 may add new functions but may not change the functions in MPI1. From mpi-comm-human@mcs.anl.gov Tue Jun 25 17:55:41 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id RAA05186; Tue, 25 Jun 1996 17:55:38 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id PAA16992 for mpi-comm-out; Tue, 25 Jun 1996 15:28:19 -0500 Received: from eagle (eagle.mcs.anl.gov [140.221.3.47]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id PAA16987 for ; Tue, 25 Jun 1996 15:28:11 -0500 From: Barry Smith Date: Tue, 25 Jun 1996 15:28:10 -0500 Message-Id: <199606252028.PAA01327@eagle> To: mpi-comm@mcs.anl.gov Subject: CHANGES to MPI standard are damaging Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk As a parallel library writer (using MPI) whose software has been downloaded, installed, and used at 100s of sites around the world, I urge the MPI 2 participants to strongly resist the urge to make ANY syntactical changes in the MPI 1.1 standard. Any change in calling sequences (no matter how trivial or well intentioned) causes havoc for libraries. People installing libraries on older MPI releases or older library releases on newer MPI releases cannot build. And most do not have the time or skills to track down the problem. Just the minor changes to the calling sequences for the attributes in going from MPI 1.0 to 1.1 meant that the PETSc libraries could not be installed without change and headaches. Library users cannot tolerate having to muck around in source figuring how to update some library due to a trivial, cosmetic change to the standard. MPI already has a solid base of thousands of users and one of its great advantages over PVM is its stability. Large MPI using libraries, 10s of thousands of lines of code, have already been written and it is not realistic to expect those to track changes in MPI 1.? calling sequences. It is far better to accept MPI 1.1 with its blemishes then to make any further changes in its syntax. MPI 2.0 may add to MPI 1.1 but it should not change. Otherwise I fear for the future of MPI. Barry Smith A concerned MPI user and fan From mpi-comm-human@mcs.anl.gov Tue Jun 25 17:57:52 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id RAA05246; Tue, 25 Jun 1996 17:57:52 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id QAA19091 for mpi-comm-out; Tue, 25 Jun 1996 16:47:03 -0500 Received: from eagle (eagle.mcs.anl.gov [140.221.3.47]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id QAA19084; Tue, 25 Jun 1996 16:46:51 -0500 From: Barry Smith Message-Id: <199606252146.QAA04687@eagle> Subject: Re: CHANGES to MPI standard are damaging To: snir@watson.ibm.com (Marc Snir/Watson/IBM Research) Date: Tue, 25 Jun 1996 16:46:49 -0500 (CDT) Cc: mpi-comm@mcs.anl.gov In-Reply-To: <9606252143.AA5423@watngi02.watson.ibm.com> from Marc Snir/Watson/IBM Research at "Jun 25, 96 05:41:09 pm" X-Mailer: ELM [version 2.4ME+ PL10 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 363 Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk That's terrific. When I see people writing about changes in MPI 1.2 I start to worry, since 1.1 had some unneccessary cosmetic changes I hate to see that repeated in 1.2. > No syntactical changes to MPI1 are under consideration. The basic rule is that > MPI1 is a standard. MPI2 may add new functions but may not change the > functions in MPI1. > From mpi-comm-human@mcs.anl.gov Tue Jun 25 18:24:09 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id SAA05493; Tue, 25 Jun 1996 18:24:08 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id RAA19685 for mpi-comm-out; Tue, 25 Jun 1996 17:10:44 -0500 Received: from sgi.sgi.com (SGI.COM [192.48.153.1]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id RAA19680 for ; Tue, 25 Jun 1996 17:10:35 -0500 Received: from mrjones.engr.sgi.com ([150.166.49.18]) by sgi.sgi.com (950413.SGI.8.6.12/950213.SGI.AUTOCF) via ESMTP id PAA00321 for <@sgi.engr.sgi.com:mpi-comm@mcs.anl.gov>; Tue, 25 Jun 1996 15:10:32 -0700 Received: by mrjones.engr.sgi.com (950413.SGI.8.6.12/940406.SGI.AUTO) for mpi-comm@mcs.anl.gov id PAA16752; Tue, 25 Jun 1996 15:10:31 -0700 From: "Eric Salo" Message-Id: <9606251510.ZM16750@mrjones.engr.sgi.com> Date: Tue, 25 Jun 1996 15:10:31 -0700 In-Reply-To: Barry Smith "CHANGES to MPI standard are damaging" (Jun 25, 3:28pm) References: <199606252028.PAA01327@eagle> X-Mailer: Z-Mail (3.2.3 08feb96 MediaMail) To: mpi-comm@mcs.anl.gov Subject: Re: CHANGES to MPI standard are damaging Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk Barry, is there anything specific in the current MPI-2 chapters that concerns you? I am unaware of any proposals to change MPI-1.1 semantics, and am quite certain that the MPI Forum would resoundingly vote down anything of the sort. -- Eric Salo Silicon Graphics Inc. "Do you know what the (415)933-2998 2011 N. Shoreline Blvd, 8U-808 last Xon said, just salo@sgi.com Mountain View, CA 94043-1389 before he died?" From owner-mpi-comm@CS.UTK.EDU Tue Jul 9 14:39:30 1996 Return-Path: Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id OAA04637; Tue, 9 Jul 1996 14:39:29 -0400 Received: from localhost (root@localhost) by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id OAA00564; Tue, 9 Jul 1996 14:34:45 -0400 Received: from drawbridge.ctc.com (drawbridge.ctc.com [147.160.199.35]) by CS.UTK.EDU with ESMTP (cf v2.9s-UTK) id OAA00556; Tue, 9 Jul 1996 14:34:32 -0400 Received: by drawbridge.ctc.com (951211.SGI.8.6.12.PATCH1042/951211.SGI) for <@drawbridge.ctc.com:mpi-comm@cs.utk.edu> id OAA11263; Tue, 9 Jul 1996 14:34:11 -0400 Received: from sgi10.ctc.com(147.160.31.8) by drawbridge.ctc.com via smap (V1.3) id sma011230; Tue Jul 9 14:31:55 1996 Received: from sgi15.ctc.com by sgi10.ctc.com via SMTP (931110.SGI/940406.SGI.AUTO) for @drawbridge.ctc.com:mpi-comm@cs.utk.edu id AA06045; Tue, 9 Jul 96 14:31:49 -0400 Received: by sgi15.ctc.com id OAA03872; Tue, 9 Jul 1996 14:31:48 -0400 From: "Rodrigo Bisbal" Message-Id: <9607091431.ZM3870@sgi15.ctc.com> Date: Tue, 9 Jul 1996 14:31:47 -0400 Reply-To: bisbal@ctc.com X-Mailer: Z-Mail (3.2.2 10apr95 MediaMail) To: mpi-comm@CS.UTK.EDU Subject: subscribe Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii bisbal@ctc.com From owner-mpi-comm@CS.UTK.EDU Tue Jul 9 14:53:02 1996 Return-Path: Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id OAA05051; Tue, 9 Jul 1996 14:53:02 -0400 Received: from localhost (root@localhost) by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id OAA02389; Tue, 9 Jul 1996 14:54:33 -0400 Received: from drawbridge.ctc.com (drawbridge.ctc.com [147.160.199.35]) by CS.UTK.EDU with ESMTP (cf v2.9s-UTK) id OAA02374; Tue, 9 Jul 1996 14:54:26 -0400 Received: by drawbridge.ctc.com (951211.SGI.8.6.12.PATCH1042/951211.SGI) for <@drawbridge.ctc.com:mpi-comm@cs.utk.edu> id OAA11454; Tue, 9 Jul 1996 14:54:13 -0400 Received: from sgi10.ctc.com(147.160.31.8) by drawbridge.ctc.com via smap (V1.3) id sma011445; Tue Jul 9 14:52:36 1996 Received: from sgi15.ctc.com by sgi10.ctc.com via SMTP (931110.SGI/940406.SGI.AUTO) for @drawbridge.ctc.com:mpi-comm@cs.utk.edu id AA06548; Tue, 9 Jul 96 14:52:30 -0400 Received: by sgi15.ctc.com id OAA03955; Tue, 9 Jul 1996 14:52:29 -0400 From: "Rodrigo Bisbal" Message-Id: <9607091452.ZM3953@sgi15.ctc.com> Date: Tue, 9 Jul 1996 14:52:29 -0400 Reply-To: bisbal@ctc.com X-Mailer: Z-Mail (3.2.2 10apr95 MediaMail) To: mpi-comm@CS.UTK.EDU Subject: buffering with mpich Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii MPI version : 1.0.12 (February 1, 1996) Implementation : mpich Uname -s : IRIX extreme 5.3 08031226 IP22 mips SGI hinv: 1 200 MHZ IP22 Processor FPU: MIPS R4010 Floating Point Chip Revision: 0.0 CPU: MIPS R4400 Processor Chip Revision: 6.0 On-board serial ports: 2 On-board bi-directional parallel port Data cache size: 16 Kbytes Instruction cache size: 16 Kbytes Secondary unified instruction/data cache size: 2 Mbytes Main memory size: 96 Mbytes EISA bus: adapter 0 Integral Ethernet: ec0, version 1 Integral SCSI controller 1: Version WD33C93B, revision D Integral SCSI controller 0: Version WD33C93B, revision D PROBLEM: I am pushing the limits of the buffer and message size for the nonblocking buffered send. I need to send a lot of data and continue computing in an overlapping fashion. This is what I've done: . I create a buffer of 500K and attach it: buffer=malloc(500000); MPI_Buffer_attach(500000); . then I create a message of 50K bytes size message=malloc(50000); . the receiver process won't start receiving until after 10 seconds allowing to test if the sender buffer works. . then I start sending 10 messages, it fills up up to 100K (2 messages), even though I attached a 500K buffer ?? The rest of the messages are not delivered until the receiver starts receiving. . I am using the MPI_Bsend and MPI_Ibsend calls. I have concluded that these buffered calls are not using the user specified buffer. They must be using a default MPI buffering scheme. QUESTIONS: 1. Is there a limit for the used-defined buffer size? (can it be more that 100K?) 2. Besides attaching a buffer, what else do you need to increase the buffer size ? 3. Is there additional flags when calling MPI_Bsend and/or MPI_Ibsend to take full advantage of the buffer ? 4. Any ideas, comments ? 5. Any experiences buffering within LAM or CHIMP? (I am using mpich) ANY COMMENTS ARE APPRECIATED: =================================== * Rodrigo Bisbal * * Concurrent Technologies Corp. * * bisbal@ctc.com * =================================== From mpi-comm-human@mcs.anl.gov Mon Aug 19 17:26:07 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id RAA02762; Mon, 19 Aug 1996 17:26:05 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id OAA01855 for mpi-comm-out; Mon, 19 Aug 1996 14:46:34 -0500 Received: from Aurora.CS.MsState.Edu (aurora.cs.msstate.edu [130.18.208.91]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id OAA01697; Mon, 19 Aug 1996 14:39:19 -0500 Received: from localhost (tony@localhost); by Aurora.CS.MsState.Edu using SMTP (SMI-8.6/7.0m-FWP-MsState); id OAA26346; Mon, 19 Aug 1996 14:39:17 -0500 Date: Mon, 19 Aug 1996 14:39:16 -0500 (CDT) From: Tony Skjellum To: Tony Kimball cc: mpi-core@mcs.anl.gov, mpi-comm@mcs.anl.gov Subject: Re: event driven chapter In-Reply-To: <199608191904.OAA26197@compound.Think.COM> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk Mr. Kimball: You should take this up with the MPI-CORE reflector. It is the policy to reflect the postscript, which is then archived. I understand the concerns you raise, and so I have taken the initiative to forward to the mpi-core group, and mpi-comm group, so it can be discussed by the appropriate people, rather than aiming it at a subset. This is my final communication with you on this subject. -Tony Anthony Skjellum, PhD, Asst. Professor of Computer Science; Mississippi State University, Department of Computer Science & NSF ERC Butler, Rm 300, PO Box 9637, Corner of Perry&Barr, Mississippi State,MS 39762 (601)325-8435 FAX: (601)325-8997; http://www.erc.msstate.edu/~tony; Quote: "What a rain of ashes falls on him that sees the new and cannot leave the old."-Shakespeare ; e-mail: tony@cs.msstate.edu; Try MPI! On Mon, 19 Aug 1996, Tony Kimball wrote: > Quoth Tony Skjellum on Mon, 19 August: > : OK, you don't understand the MPI reflector list. We send out the > : postscript. The fact that I reflected it was unintentional. I do think > : you know how to hit delete, so ... > > I object strongly to the practice. This is a bandwidth-profligate > distribution technique, and therefore of detriment to the Internet > community as a whole. Moreover, it does material damage to those who > must pay for the packets to recieve the unwanted drafts. If it were > desirable to distribute drafts by email, which I strongly contend it > is not, it would nonetheless be no more than commonsense to separate > the distribution lists into drafts-only and discussion-only subsets. > I have no problem hitting delete. That is not an issue for me. > Locking down my Ethernet LAN for as much as a second to deliver unwanted > materials is a significant detriment, made worse by a factor of twenty > when I have twenty independent recipients on the LAN. Worse yet > is locking down the T1 external interface of my network to the > backbone for that time. Worse still is locking down my V.34 modem > port for 3 full minutes, particularly if I am paying premium rates > to read my mail in flight. > > > From mpi-comm-human@mcs.anl.gov Thu Aug 29 17:52:34 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id RAA23471; Thu, 29 Aug 1996 17:52:32 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id PAA07870 for mpi-comm-out; Thu, 29 Aug 1996 15:34:45 -0500 Received: from mail.think.com (Mail1.Think.COM [131.239.33.245]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id MAA02813; Thu, 29 Aug 1996 12:39:57 -0500 Received: from Early-Bird.Think.COM (Early-Bird-1.Think.COM [131.239.146.105]) by mail.think.com (8.7.5/m3) with ESMTP id NAA20651; Thu, 29 Aug 1996 13:39:06 -0400 (EDT) Received: from compound.Think.COM (fergus-2.dialup.prtel.com [206.10.99.132]) by Early-Bird.Think.COM (8.7.5/e1) with ESMTP id NAA26694; Thu, 29 Aug 1996 13:39:02 -0400 (EDT) Received: (from alk@localhost) by compound.Think.COM (8.7.5/8.7.3) id MAA01723; Thu, 29 Aug 1996 12:39:19 -0500 (CDT) Date: Thu, 29 Aug 1996 12:39:19 -0500 (CDT) From: Tony Kimball Message-Id: <199608291739.MAA01723@compound.Think.COM> To: tony@aurora.cs.msstate.edu Cc: mpi-core@mcs.anl.gov, mpi-comm@mcs.anl.gov Subject: Re: Newest realtime draft References: <199608291609.LAA01626@compound.Think.COM> Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk Quoth Tony Skjellum on Thu, 29 August: : ... express your concerns directly to : the entire MPI Forum body... I presume the CC list is appropriate. I must confess some reluctance to address such a large audience without a offer of proposal, but you are most *encouraging* on this point;-). My own concerns would be met by splitting the various lists into discussion and draft-distribution groups. Thus, I would request this, were it not that others may find the measure inadequate, and the issue, a procedural annoyance, rather than substantive and germane, may be raised again -- which we would all prefer to avoid, yes? Specifically: : ...connected to us by cellular modem, and reads his : e-mail by downloading while flying cross country... In consideration of such recipients, the obvious suggestion is to establish a repository for submission of drafts, accessible by http and ftp (and, by implication, ftp-mail), and adopt the convention that announcements of draft submissions, and not the drafts themselves, should be broadcast. Absent the volunteer of such a repository, this preferred solution fails, however. Were it to exist, the slight elaboration of using it as an archive for submissions to the previously mention draft-distribution lists would serve the interests of *all* readers. I cannot volunteer a site, for everything which I manage personally is in experimental/developmental use, and subject to random downtime. : I clearly explained these things to you before. My apologies: I must have somehow overlooked that explanation. Regardless, I should prefer that personal criticisms -- and "atavistic" is rather pejorative, you must admit -- pardon me while I struggle to walk erect -- should be expressed in private correspondence when possible, for otherwise there is a strong temptation to defend oneself to an entirely uninterested audience. Atavistic though I may be, not to mention curmudgeonly, I bear no doubt that my concerns, however atavistically inflated, are shared by less vocal recipients. From mpi-comm-human@mcs.anl.gov Thu Aug 29 18:03:29 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id SAA23982; Thu, 29 Aug 1996 18:03:27 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id PAA07810 for mpi-comm-out; Thu, 29 Aug 1996 15:32:11 -0500 Received: from Aurora.CS.MsState.Edu (aurora.cs.msstate.edu [130.18.208.91]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id LAA01537; Thu, 29 Aug 1996 11:34:47 -0500 Received: from localhost (tony@localhost); by Aurora.CS.MsState.Edu using SMTP (SMI-8.6/7.0m-FWP-MsState); id LAA19890; Thu, 29 Aug 1996 11:34:47 -0500 Date: Thu, 29 Aug 1996 11:34:47 -0500 (CDT) From: Tony Skjellum To: Tony Kimball cc: mpi-core@mcs.anl.gov, mpi-comm@mcs.anl.gov Subject: Re: Newest realtime draft In-Reply-To: <199608291609.LAA01626@compound.Think.COM> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk Mr. Kimball: I cannot thank you enough for reiterating your concern, but I remind you that your subscription is voluntary, but the method of sharing information is the convention of the Forum, of which I am but one voice among many. Once again, I have propagated your concern to the Forum. I cannot stop doing transmission in this way until the MPI Forum changes its policy. Please address comments to those bodies, and unsubscribe in the meantime to control bandwidth to your desktop. You will find periodic updates at the URLs below, and you can poll for them. If you were subscribing to other lists, you would get their postscript too. Perhaps you do, and are haranging others than me. If so, I feel that you are missing the opportunity to express your concerns directly to the entire MPI Forum body, which might listen to you. I clearly explained these things to you before. If you do not agree with the terms of our voluntary organization's activities, please complain to the group at large, not to one of those following its procedures. I personally am time writing this letter, because you are from a company that might contribute to the standard; not because i have not explained this before. To be frank, if you were a random lurker, I would have blown you off already, after having responded clearly twice before. However, I do find all of this tiresome and atavistic. Your employer would be better served by your contributing a technical response to the documents, rather than pursuing a crusade. The only other person who complained about this is connected to us by cellular modem, and reads his e-mail by downloading while flying cross country... Tony Skjellum On Thu, 29 Aug 1996, Tony Kimball wrote: > Date: Thu, 29 Aug 1996 11:09:32 -0500 (CDT) > From: Tony Kimball > To: tony@erc.msstate.edu > Subject: Newest realtime draft > > Quoth Tony Skjellum on Wed, 28 August: > : See also anonymous ftp > : as ftp://aurora.cs.msstate.edu/pub/mpi/rt-2-28aug96.ps > : or ftp://aurora.cs.msstate.edu/pub/mpi/rt-2-28aug96.ps.Z > : > > I very much appreciate the urls, and thank you for them > most enthusiastically; however, I must strenuously protest > the redudant inclusion of 203 kilobytes of uuencoded > binary data. > Anthony Skjellum, PhD, Associate Professor of Computer Science; Mississippi State University, Department of Computer Science & NSF ERC Butler, Rm 300, PO Box 9637, Corner of Perry&Barr, Mississippi State,MS 39762 (601)325-8435 FAX: (601)325-8997; http://www.erc.msstate.edu/~tony; Quote: "What a rain of ashes falls on him that sees the new and cannot leave the old."-Shakespeare ; e-mail: tony@cs.msstate.edu; Try MPI! From mpi-comm-human@mcs.anl.gov Fri Sep 6 15:58:38 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id PAA24498; Fri, 6 Sep 1996 15:58:35 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id NAA17711 for mpi-comm-out; Fri, 6 Sep 1996 13:37:55 -0500 Received: from Aurora.CS.MsState.Edu (aurora.cs.msstate.edu [130.18.208.91]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id NAA17495; Fri, 6 Sep 1996 13:28:05 -0500 Received: from localhost (tony@localhost); by Aurora.CS.MsState.Edu using SMTP (SMI-8.6/7.0m-FWP-MsState); id NAA22419; Fri, 6 Sep 1996 13:28:16 -0500 Date: Fri, 6 Sep 1996 13:28:16 -0500 (CDT) From: Tony Skjellum To: mpi-core@mcs.anl.gov, mpi-comm@mcs.anl.gov cc: mpi-coll@mcs.anl.gov, mpi-realtime@mcs.anl.gov Subject: FYI Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk MPI Forum: I just wanted to thank everyone who expressed concern about my recent illness at the MPI Meeting. I am much better now, and I will be in touch with those of you who covered for me on Collective and Realtime. These will proceed apace for the next meeting: Andrew will send minutes for Collective soon, and Arkady for Realtime. I had more visitors and helpers at the hospital from the MPI Forum than could have possibly been expected ... it was some experience to be sure. Anyway, I was allowed to leave without staying their too long, and I was OK to travel on Thursday, and am 99% recovered today [Friday]. For realtime aficionados, we will be setting an interim MPI/RT meeting for late September. Arkady will announce in minutes on the realtime reflector. People interested in channels under the collective chapter should also refer to the channel activity in the realtime chapter to see how further evolution has gone on there. Thanks again for your concern, Tony Anthony Skjellum, PhD, Associate Professor of Computer Science; Mississippi State University, Department of Computer Science & NSF ERC Butler, Rm 300, PO Box 9637, Corner of Perry&Barr, Mississippi State,MS 39762 (601)325-8435 FAX: (601)325-8997; http://www.erc.msstate.edu/~tony; Quote: "What a rain of ashes falls on him that sees the new and cannot leave the old."-Shakespeare ; e-mail: tony@cs.msstate.edu; Try MPI! From mpi-comm-human@mcs.anl.gov Tue Sep 17 10:48:44 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id KAA16655; Tue, 17 Sep 1996 10:48:41 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id IAA20181 for mpi-comm-out; Tue, 17 Sep 1996 08:59:55 -0500 Received: from igw2.watson.ibm.com (igw2.watson.ibm.com [129.34.139.6]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id IAA20160 for ; Tue, 17 Sep 1996 08:59:38 -0500 Received: from mailhub1.watson.ibm.com (mailhub1.watson.ibm.com [9.2.249.31]) by igw2.watson.ibm.com (8.7.4/8.7.1) with ESMTP id KAA08049 for ; Tue, 17 Sep 1996 10:00:04 -0400 Received: from watngi01.watson.ibm.com (watngi01.watson.ibm.com [9.2.235.20]) by mailhub1.watson.ibm.com (8.7.1/09-08-96) with SMTP id JAA690040 for ; Tue, 17 Sep 1996 09:59:35 -0400 Received: by watngi01.watson.ibm.com (IBM OS/2 SENDMAIL VERSION 1.3.17/07-11-96 ) id AA0226; Tue, 17 Sep 96 09:59:34 -0400 Message-Id: <9609171359.AA0226@watngi01.watson.ibm.com> Received: from IBM Research with "Lotus Notes Mail Gateway for SMTP" id BEDF13D896ED9DD6852563A8004CC303; Tue, 17 Sep 96 09:59:33 To: mpi-comm Cc: mpi-core From: Marc Snir/Watson/IBM Research Date: 16 Sep 96 10:40:05 Subject: Request for information on Fortran systems that use MPI Mime-Version: 1.0 Content-Type: Text/Plain Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk The current proposal for 1sided communication in MPI2 makes the use of some functions (lock/unlock, for shared memory emulation) contingent upon the availability of dynamic memory allocation and C-like pointers. The type of code that is assumed to work in Fortran is shown below: REAL A POINTER (P, A(100,100)) ! declare P to be an (integer) pointer to array A CALL MPI_MEM_ALLOC(400,comm,P,ierr) ! allocate space for array A and returns in P address of allocated memory; ! now array A can be used This is not part of standard Fortran or Fortran 90 (Fortran 90 pointers cannot be used for this type of code), but is widely available in many Fortran environments. Questions: 1. Are there Fortran systems that do not support (integer) pointers at all? 2. Are there systems that support such pointers, but require a syntax that is different than the one shown in the example above? 3. On 64 bit systems, what is the datatype used for (integer) pointers? I.e., what should be the type of the 3rd formal argument of MPI_MEM_ALLOC? If your pet Fortran has problems with this kind of code, please answer; otherwise you will be stuck with an MPI feature that you cannot use on your system. From mpi-comm-human@mcs.anl.gov Tue Sep 17 13:44:51 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id NAA19037; Tue, 17 Sep 1996 13:44:50 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id MAA25293 for mpi-comm-out; Tue, 17 Sep 1996 12:16:47 -0500 Received: from felix.dircon.co.uk (felix.dircon.co.uk [193.128.224.10]) by antares.mcs.anl.gov (8.6.10/8.6.10) with SMTP id MAA24938; Tue, 17 Sep 1996 12:02:21 -0500 Received: by felix.dircon.co.uk id AA03732 (5.67b/IDA-1.5); Tue, 17 Sep 1996 17:35:51 +0100 Message-Id: <199609171635.AA03732@felix.dircon.co.uk> Received: from gw2-151.pool.dircon.co.uk(194.112.35.151) by amnesiac via smap (V1.3) id sma003698; Tue Sep 17 17:35:20 1996 Received: from localhost by jim (SMI-8.6) id RAA03777; Tue, 17 Sep 1996 17:41:04 +0100 To: Marc Snir/Watson/IBM Research Cc: mpi-comm , mpi-core Subject: Re: Request for information on Fortran systems that use MPI In-Reply-To: Your message of "16 Sep 1996 10:40:05." <9609171359.AA0226@watngi01.watson.ibm.com> Date: Tue, 17 Sep 1996 17:41:04 +0100 From: James Cownie Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk > 1. Are there Fortran systems that do not support (integer) pointers at all? I believe that g77 is such a system. GNU F77 version 2.7.2 (sparc) compiled by GNU C version 2.7.2. GNU Fortran Front End version 0.5.16 compiled: May 30 1996 14:00:35 jcownie@jim: g77 -o foo foo.f foo.f: In program `prof': foo.f:4: POINTER (P, A(100,100)) ^ Unimplemented or invalid form of statement at (^) -- statement ignored (this is a catchall diagnostic that currently applies to a wide variety of errors, including things like invalid ordering of statements) -- Jim James Cownie Dolphin Interconnect Solutions Phone : +44 117 9071438 E-Mail: jcownie@dolphinics.com From mpi-comm-human@mcs.anl.gov Wed Sep 18 12:04:55 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id MAA27520; Wed, 18 Sep 1996 12:04:52 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id KAA14744 for mpi-comm-out; Wed, 18 Sep 1996 10:37:49 -0500 Received: from Aurora.CS.MsState.Edu (aurora.cs.msstate.edu [130.18.208.91]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id KAA14503; Wed, 18 Sep 1996 10:29:51 -0500 Received: from localhost (tony@localhost); by Aurora.CS.MsState.Edu using SMTP (SMI-8.6/7.0m-FWP-MsState); id KAA10555; Wed, 18 Sep 1996 10:29:36 -0500 Date: Wed, 18 Sep 1996 10:29:36 -0500 (CDT) From: Tony Skjellum To: mpi-core@mcs.anl.gov, mpi-comm@mcs.anl.gov Subject: JOD as separate document Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk Folks, I make the following proposal. Since the JOD document is the core of MPI-3, and since it may need chapter stratification, I suggest that it be separated from MPI-2 document, and be physically attached, rather than being jointly managed. Reasons: 1) Reduction of complexity 2) Reduction of work for Steve Lederman 3) Ability to begin structuring work for MPI-3 4) Ability to continue work on JOD contents without strictures of SC'96 deadline associated with rest of MPI-2 There is no loss of generality, provided the JOD gets distributed at Supercomputing with rest of document. Comments? Tony From mpi-comm-human@mcs.anl.gov Wed Sep 18 12:09:02 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id MAA27594; Wed, 18 Sep 1996 12:09:01 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id KAA14714 for mpi-comm-out; Wed, 18 Sep 1996 10:36:40 -0500 Received: from Aurora.CS.MsState.Edu (aurora.cs.msstate.edu [130.18.208.91]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id KAA14409 for ; Wed, 18 Sep 1996 10:27:01 -0500 Received: from Ardra.CS.MsState.Edu (ardra.cs.msstate.edu [192.208.157.74]); by Aurora.CS.MsState.Edu using SMTP (SMI-8.6/7.0m-FWP-MsState); id KAA10535; Wed, 18 Sep 1996 10:27:14 -0500 From: Tony Skjellum Received: by Ardra.CS.MsState.Edu (SMI-8.6/6.0c-FWP); id KAA16204; Wed, 18 Sep 1996 10:26:43 -0500 Date: Wed, 18 Sep 1996 10:26:43 -0500 Message-Id: <199609181526.KAA16204@Ardra.CS.MsState.Edu> To: mpi-comm@mcs.anl.gov, mpi-core@mcs.anl.gv Subject: JOD document Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk I make the following proposal. Since the JOD document is the core of MPI-3, and since it may need chapter stratification, I suggest that it be separated from MPI-2 document, and be physically attached, rather than being jointly managed. Reasons: 1) Reduction of complexity 2) Reduction of work for Steve Lederman 3) Ability to begin structuring work for MPI-3 There is no loss of generality, provided the JOD gets distributed at Supercomputing with rest of document. Comments? Tony From mpi-comm-human@mcs.anl.gov Fri Sep 20 02:09:57 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id CAA24235; Fri, 20 Sep 1996 02:09:56 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id XAA28114 for mpi-comm-out; Thu, 19 Sep 1996 23:23:43 -0500 Received: from mcs.anl.gov (donner.mcs.anl.gov [140.221.5.134]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id XAA28086; Thu, 19 Sep 1996 23:22:42 -0500 Message-Id: <199609200422.XAA28086@antares.mcs.anl.gov> To: mpi-comm@antares.mcs.anl.gov, mpi-core@antares.mcs.anl.gov Subject: New Misc chapters Date: Thu, 19 Sep 1996 23:22:42 -0500 From: Rusty Lusk Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk The next two messages contain updates of the Misc 1.2 and Misc 2.0 chapters that reflect decisions made during the last meeting and email discussion since then. This chapter is highly miscellaneous, and I am not sure that I have correctly captured every issue that has been raised. If there is something that you have been following in Misc, please check that it has been properly addressed. Rusty From mpi-comm-human@mcs.anl.gov Fri Sep 20 05:07:37 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id FAA11317; Fri, 20 Sep 1996 05:07:28 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id XAA28198 for mpi-comm-out; Thu, 19 Sep 1996 23:27:31 -0500 Received: from mcs.anl.gov (donner.mcs.anl.gov [140.221.5.134]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id XAA28155; Thu, 19 Sep 1996 23:26:48 -0500 Message-Id: <199609200426.XAA28155@antares.mcs.anl.gov> To: mpi-comm@antares.mcs.anl.gov, mpi-core@antares.mcs.anl.gov Subject: New Misc-2.0 Date: Thu, 19 Sep 1996 23:26:46 -0500 From: Rusty Lusk Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk %!PS-Adobe-2.0 %%Creator: dvips 5.528 Copyright 1986, 1994 Radical Eye Software %%Title: temp.dvi %%CreationDate: Thu Sep 19 23:25:17 1996 %%Pages: 21 %%PageOrder: Ascend %%BoundingBox: 0 0 612 792 %%EndComments %DVIPSCommandLine: dvips -o temp.ps temp %DVIPSParameters: dpi=300, comments removed %DVIPSSource: TeX output 1996.09.19:2325 %%BeginProcSet: tex.pro /TeXDict 250 dict def TeXDict begin /N{def}def /B{bind def}N /S{exch}N /X{S N}B /TR{translate}N /isls false N /vsize 11 72 mul N /hsize 8.5 72 mul N /landplus90{false}def /@rigin{isls{[0 landplus90{1 -1}{-1 1} ifelse 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale isls{landplus90{VResolution 72 div vsize mul 0 exch}{Resolution -72 div hsize mul 0}ifelse TR}if Resolution VResolution vsize -72 div 1 add mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@landscape{/isls true N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder}N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail}B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{ pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get}B /ch-xoff{128 ch-data dup length 3 sub get sub}B /ch-yoff{ch-data dup length 2 sub get 127 sub}B /ch-dx{ch-data dup length 1 sub get}B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ ch-image}imagemask restore}B /D{/cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D }B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto /V matrix currentmatrix dup 1 get dup mul exch 0 get dup mul add .99 lt{/QV}{/RV}ifelse load def pop pop}N /eop{SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if pop /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put}for 65781.76 div /vsize X 65781.76 div /hsize X}N /p{show}N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V {}B /RV statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval dup(Display)eq exch 0 4 getinterval(NeXT)eq or}{pop false} ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /QV{gsave transform round exch round exch itransform moveto rulex 0 rlineto 0 ruley neg rlineto rulex neg 0 rlineto fill grestore}B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail} B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{ 3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{ 3 2 roll p a}B /bos{/SS save N}B /eos{SS restore}B end %%EndProcSet TeXDict begin 40258431 52099146 1000 300 300 (/tmp_mnt/Net/antireo/antireo6/lusk/mpi2/report2/temp.dvi) @start /Fa 1 1 df0 D E /Fb 2 111 df<00180038001000000000000000000000000001C0022004300430086000600060 006000C000C000C000C001800180018001806300E300C60078000D1D80960E>106 D<383C0044C6004702004602008E06000C06000C06000C0C00180C00180C401818401818 80300880300F00120E7F8D15>110 D E /Fc 4 52 df<07C018303018701C600C600CE0 0EE00EE00EE00EE00EE00EE00EE00EE00E600C600C701C30181C7007C00F157F9412>48 D<03000700FF000700070007000700070007000700070007000700070007000700070007 00070007007FF00C157E9412>I<0F8030E040708030C038E03840380038007000700060 00C00180030006000C08080810183FF07FF0FFF00D157E9412>I<0FE030306018701C70 1C001C00180038006007E000300018000C000E000EE00EE00EC00C401830300FE00F157F 9412>I E /Fd 52 122 df<00E001C0038007000E000E001C001C003800380038007000 700070007000E000E000E000E000E000E000E000E000E000E000E000E000700070007000 70003800380038001C001C000E000E000700038001C000E00B2A7E9E10>40 DI<018001C0018001806186F99F7DBE1FF8 07E007E01FF87DBEF99F61860180018001C0018010127E9E15>I44 DI<03C00FF01FF83C3C381C700E700E700EE007 E007E007E007E007E007E007E007E007E007E007E007E007700E700E700E381C3C3C1FF8 0FF007E0101D7E9B15>48 D<010007003F00FF00C7000700070007000700070007000700 07000700070007000700070007000700070007000700070007000700FFF8FFF80D1C7C9B 15>I<07E01FF83C3C700E700EE007E007E007E007700E700E3C3C1FF807E01FF83C3C70 0E700EE007E007E007E007E007E007700E781E3C3C1FF807E0101D7E9B15>56 D58 D<7FFFFFC0FFFFFFE000 00000000000000000000000000000000000000000000000000000000000000FFFFFFE07F FFFFC01B0C7E8F20>61 D<001C0000003E0000003E0000002E0000006700000067000000 E7800000C7800000C3800001C3C0000183C0000181C0000381E0000381E0000700F00007 00F0000600F0000E0078000FFFF8000FFFF8001C003C001C003C0018003C0038001E0038 001E0070001F0070000F0070000F00E0000780191D7F9C1C>65 DI<003FC000FFF003C0F00780300F00001E00003C00003C00 00780000780000780000F00000F00000F00000F00000F00000F00000F00000F00000F000 007800007800007800003C00003C00001E00000F000807801803C07800FFF0003F80151F 7D9D1B>IIII<00 3F8001FFF003C0F80780380F00181E00003C00003C0000780000780000780000F00000F0 0000F00000F00000F00000F00000F007F8F007F8F000387800387800387800383C00383C 00381E00380F003807803803C0F801FFF0003F80151F7D9D1C>III75 DIII<003F000001 FFE00003FFF00007C0F8000F807C001E001E003E001F003C000F00780007807800078078 000780F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F8 0007C078000780780007807C000F803C000F003E001F001F003E000F807C0007C0F80003 FFF00001FFE000003F00001A1F7E9D1F>II82 D<03F8000FFE001C0F00 380700700300600000E00000E00000E00000E00000F000007800007F00003FE0001FFC00 07FE0001FF00001F800007800003C00003C00001C00001C00001C00001C0C00180E00380 F007007C0E001FFC0007F000121F7E9D17>III87 D89 D<7FFFF07FFFF00001E00003E00003C00007C000 0780000F00001F00001E00003E00003C0000780000F80000F00001F00001E00003C00007 C0000780000F80000F00001E00003E00003C00007C0000780000FFFFF0FFFFF0141D7E9C 19>I<0FC03FF07FF87038401C001C001C00FC0FFC3FFC781CE01CE01CE01CF07C7FFC7F DC3F1C0E127E9114>97 DI<07E00FF81FFC3C1C70047000E000E000E000E000E000E000700070043C1C1FFC 0FF807E00E127E9112>I<000E000E000E000E000E000E000E000E000E000E000E0F8E1F EE3FFE7C3E700E700EE00EE00EE00EE00EE00EE00EF00E701E7C3E3FFE1FEE0F8E0F1D7E 9C15>I<07C01FE03FF078787018601CFFFCFFFCFFFCE000E000E000700070043C1C3FFC 1FF807E00E127E9112>I<00FC01FC03FC07000E000E000E000E000E000E000E00FFE0FF E00E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E1D80 9C0D>I<03C3C00FFFC01FFFC01C3800381C00381C00381C00381C00381C001C38001FF8 001FF0003BC0003800003800001FFC001FFF003FFF80700780E001C0E001C0E001C0F003 C07C0F803FFF001FFE0007F800121B7F9115>III107 DIII<03F0000FFC001FFE003C0F007807 80700380E001C0E001C0E001C0E001C0E001C0F003C07003807807803C0F001FFE000FFC 0003F00012127F9115>II114 D<1FC03FF07FF0F030E000E000F0007F003FC01FE000F0003800388038F0 78FFF07FE01FC00D127F9110>I<1C001C001C001C001C001C00FFE0FFE01C001C001C00 1C001C001C001C001C001C001C001C001C001C201FF00FF007C00C187F970F>I118 D121 D E /Fe 48 123 df46 D<01F00007FC000FFE001F1F003C0780 3C07807803C07803C07803C07001C0F001E0F001E0F001E0F001E0F001E0F001E0F001E0 F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001E07803C07803C07803C07803C0 3C07803C07801F1F000FFE0007FC0001F00013237EA118>48 D<00C001C007C0FFC0FFC0 FBC003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C0 03C003C003C003C003C003C003C003C0FFFFFFFFFFFF10227CA118>I<03F0000FFC001F FE003C1F003007807007C06003C0E003E0C001E04001E04001E00001E00001E00001E000 03C00003C0000780000780000F00001E00003C0000780000F00001E00001C00003800007 00000E00001C0000380000700000FFFFE0FFFFE0FFFFE013227EA118>I<01F00007FC00 1FFF003E0F003807807003C02003C02003C00003C00003C00003C0000780000780000F00 001E0003FC0003F80003FE00000F000007800003C00003C00001E00001E00001E00001E0 0001E08001E0C003C0E003C07007803C0F801FFF000FFC0003F00013237EA118>I<001F 00001F00002F00002F00006F0000EF0000CF0001CF0001CF00038F00038F00078F00070F 000F0F000E0F001E0F003C0F003C0F00780F00780F00F00F00FFFFF8FFFFF8FFFFF8000F 00000F00000F00000F00000F00000F00000F00000F00000F0015217FA018>I<3FFF803F FF803FFF803C00003C00003C00003C00003C00003C00003C00003C00003C00003CF8003F FE003FFF003F0F803E07803C03C03803C00001E00001E00001E00001E00001E00001E000 01E04003C04003C0E003C07007807C1F003FFE000FFC0003F00013227EA018>I<007E00 01FF0003FF0007C1000F00001E00001E00003C00003C000078000078000078F800F1FE00 F7FF00FF0F80FC0780F803C0F803C0F801C0F001E0F001E0F001E0F001E0F001E07001E0 7001E07801E07803C03803C03C03801C07801F0F000FFE0007FC0001F00013237EA118> II<01 F00007FC000FFE001E0F003C07803C07807803C07803C07803C07803C07803C03803803C 07801E0F000F1E0007FC0003F8000FFE001E0F003C07807803C07803C0F001E0F001E0F0 01E0F001E0F001E0F001E07803C07803C03C07803E0F801FFF0007FC0001F00013237EA1 18>I<01F00007FC000FFE001E0F003C07003807807803807803C0F003C0F001C0F001E0 F001E0F001E0F001E0F001E0F001E07003E07803E07803E03C07E03E1FE01FFDE00FF1E0 03E3C00003C00003C0000380000780000700000F00101E00183C003FF8001FF00007C000 13237EA118>I<001F0000001F0000003F8000003F8000003B8000007BC0000073C00000 71C00000F1E00000F1E00000E0E00001E0F00001E0F00001C0F00003C0780003C0780003 80780007803C0007803C0007003C000F001E000F001E000FFFFE001FFFFF001FFFFF001C 000F003C0007803C00078038000780780003C0780003C0700003C0F00001E0F00001E0E0 0001E01B237EA220>65 DI<000FF000007FFC0000FFFF0001F01F0003C0070007 8000000F0000001E0000003E0000003C0000003C000000780000007800000078000000F0 000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F000000078 00000078000000780000003C0000003C0000003E0000001E0000000F0000000780008003 C0038001F00F8000FFFF00007FFC00000FF00019257DA31F>III<000FF0 00003FFE0000FFFF8001F80F8003E00380078000000F0000001E0000001E0000003C0000 003C000000780000007800000078000000F0000000F0000000F0000000F0000000F00000 00F0000000F000FFC0F000FFC0F000FFC0780003C0780003C0780003C03C0003C03C0003 C01E0003C01E0003C00F0003C0078003C003E003C001F807C000FFFFC0003FFF00000FF8 001A257DA321>71 D73 D77 D<001FC000007FF00001FFFC0003F07E00 07C01F000F800F801F0007C01E0003C03C0001E03C0001E0780000F0780000F0780000F0 70000070F0000078F0000078F0000078F0000078F0000078F0000078F0000078F0000078 F0000078780000F0780000F0780000F07C0001F03C0001E03E0003E01E0003C01F0007C0 0F800F8007C01F0003F07E0001FFFC00007FF000001FC0001D257DA324>79 DI82 D<00FE0003FFC007FFE00F81E01E00603C00003C00007800007800 007800007800007800007C00003C00003F00001FC0000FFC0007FF0001FF80003FC00007 E00001F00000F00000F8000078000078000078000078000078000078C000F0E000F0F801 E07E07C03FFF800FFF0001FC0015257EA31B>II<07E01FF83FFC381E201E000F00 0F000F000F00FF07FF1FFF3E0F780FF00FF00FF00FF00FF83F7FFF3FEF1F8F10167E9517 >97 DI<01FC0007FF000FFF801F03803C0180780000780000700000F00000F000 00F00000F00000F00000F000007800007800007800003C00401F03C00FFFC007FF8001FC 0012167E9516>I<0003C00003C00003C00003C00003C00003C00003C00003C00003C000 03C00003C00003C00003C003E3C00FFBC01FFFC03F0FC03C07C07803C07803C0F003C0F0 03C0F003C0F003C0F003C0F003C0F003C0F003C07803C07803C03C07C03E0FC01FFFC00F FBC003E3C012237EA219>I<03F00007FC001FFE003E0F003C0780780380780380F001C0 FFFFC0FFFFC0FFFFC0F00000F00000F000007000007800007800003C00801F07800FFF80 07FF0001F80012167E9516>I<003F00FF01FF03C0038007800780078007800780078007 800780FFF8FFF8FFF8078007800780078007800780078007800780078007800780078007 8007800780078007800780102380A20F>I<01F07807FFF80FFFF81F1F001E0F003C0780 3C07803C07803C07803C07801E0F001F1F000FFE001FFC0019F0003800003800003C0000 1FFE001FFFC01FFFE03FFFF07801F07800F8F00078F00078F00078F000787800F03E03E0 1FFFC00FFF8001FC0015217F9518>III<00F000F000F000F000 000000000000000000000000000000000000F000F000F000F000F000F000F000F000F000 F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F080 F0E3E0FFE07FC01F000C2D83A20D>I108 DII<01FC0007FF000FFF801F 07C03C01E07800F07800F0700070F00078F00078F00078F00078F00078F000787800F078 00F07C01F03E03E01F07C00FFF8007FF0001FC0015167F9518>II<03E3C007FBC00FFFC01F0FC03C07 C07C03C07803C0F803C0F003C0F003C0F003C0F003C0F003C0F003C0F803C07803C07803 C03C07C03F0FC01FFFC00FFBC003E3C00003C00003C00003C00003C00003C00003C00003 C00003C00003C00003C012207E9519>II<07F01FFC 3FFE3C0E7806780078007C003F003FF01FF80FFC01FE001F000F000F000FC00FF81EFFFE 3FFC0FF010167F9513>I<0F000F000F000F000F000F00FFF8FFF8FFF80F000F000F000F 000F000F000F000F000F000F000F000F000F000F000F080F1C07FC07F803E00E1C7F9B12 >II119 D<7801F07C01E03E03C01E07C00F0780078F0007DE0003FC0001FC0000F800 00700000F80001FC0003DC00039E00078F000F07801E07801E03C03C01E07800F0F800F8 1516809516>II<7FFFC07FFFC07FFFC0000F80000F00001E00003E00007C0000780000F00001F00003 E00003C0000780000F80001F00001E00003C00007C0000FFFFC0FFFFC0FFFFC012167F95 15>I E /Ff 23 123 df<70F8F8F87005057C840D>58 D<70F8FCFC7404040408081010 2040060E7C840D>I<000001C00000078000001E00000078000001E00000078000000E00 000038000000F0000003C000000F0000003C000000F0000000F00000003C0000000F0000 0003C0000000F0000000380000000E0000000780000001E0000000780000001E00000007 80000001C01A1A7C9723>I62 D<0FFFFFFC1E03C0381803C0181003C0082003C008 20078008600780084007800840078008800F0010000F0000000F0000000F0000001E0000 001E0000001E0000001E0000003C0000003C0000003C0000003C00000078000000780000 007800000078000000F0000000F0000000F0000000F0000001F000007FFFC0001E1F7F9E 1B>84 D<00F1800389C00707800E03801C03803C0380380700780700780700780700F00E 00F00E00F00E00F00E10F01C20F01C20703C20705C40308C400F078014147E9318>97 D<07803F8007000700070007000E000E000E000E001C001C001CF01D0C3A0E3C0E380F38 0F700F700F700F700FE01EE01EE01EE01CE03CE038607060E031C01F0010207E9F14>I< 0000780003F80000700000700000700000700000E00000E00000E00000E00001C00001C0 00F1C00389C00707800E03801C03803C0380380700780700780700780700F00E00F00E00 F00E00F00E10F01C20F01C20703C20705C40308C400F078015207E9F18>100 D<007C01C207010E011C013C013802780C7BF07C00F000F000F000F00070007001700230 04183807C010147E9315>I<00007C0000CE00019E00039E00030C000700000700000700 000700000E00000E00000E0000FFF0000E00000E00001C00001C00001C00001C00001C00 00380000380000380000380000380000700000700000700000700000700000E00000E000 00E00000E00000C00001C000318000798000F300006200003C000017297E9F16>I<00E0 01E001E000C000000000000000000000000000000E001300238043804380438087000700 07000E000E001C001C001C20384038403840388019000E000B1F7E9E10>105 D<03C01FC0038003800380038007000700070007000E000E000E000E001C001C001C001C 0038003800380038007000700070007100E200E200E200E200640038000A207E9F0E> 108 D<1E07C07C00231861860023A032030043C034030043803803804380380380870070 07000700700700070070070007007007000E00E00E000E00E00E000E00E00E000E00E01C 101C01C01C201C01C038201C01C038401C01C0184038038018801801800F0024147E9328 >I<1E07802318C023A06043C0704380704380708700E00700E00700E00700E00E01C00E 01C00E01C00E03821C03841C07041C07081C03083803101801E017147E931B>I<007C00 01C3000301800E01C01E01C01C01E03C01E07801E07801E07801E0F003C0F003C0F003C0 F00780F00700700F00700E0030180018700007C00013147E9316>I<03C1E00462180474 1C08781C08701E08701E10E01E00E01E00E01E00E01E01C03C01C03C01C03C01C0380380 780380700380E003C1C0072380071E000700000700000E00000E00000E00000E00001C00 001C0000FFC000171D819317>I<1E1E0023210023C38043C78043878043830087000007 00000700000700000E00000E00000E00000E00001C00001C00001C00001C000038000018 000011147E9315>114 D<007C018203010603060706060E00078007F803FC01FE001F00 077007F006F006E004400820301FC010147E9315>I<00C000E001C001C001C001C00380 0380FFF8038007000700070007000E000E000E000E001C001C001C001C10382038203820 384018800F000D1C7F9B10>I<0F00601180702180E021C0E041C0E04380E08381C00701 C00701C00701C00E03800E03800E03800E03840E07080C07080C07080E0F1006131003E1 E016147E931A>I<03C1C00C62201034701038F02038F020386040700000700000700000 700000E00000E00000E00000E02061C040F1C040F1C080E2C080446300383C0014147E93 1A>120 D<0F00601180702180E021C0E041C0E04380E08381C00701C00701C00701C00E 03800E03800E03800E03800E07000C07000C07000E0F00061E0003EE00000E00000E0000 1C0078180078380070700060600021C0001F0000141D7E9316>I<01E02003F04007F8C0 0C1F8008010000020000040000080000100000600000C000010000020000040080080100 1003003F060061FC0040F80080700013147E9315>I E /Fg 26 90 df<3078F8787005057C840D>46 D<007E0001C3000301800701C00E00C00E00E01C00E0 1C00E03C01E03801E07801E07801E07801E07801E07801E0F003C0F003C0F003C0F003C0 F003C0F00380F00780E00780E00700E00700E00E00600E00701C003038003870000FC000 131F7C9D17>48 D<000C001C00FC0F380038003800380038003800700070007000700070 007000E000E000E000E000E000E001C001C001C001C001C001C0038003C0FFFE0F1E7C9D 17>I<003F8000C1E00100F00200780400780400780F007C0F807C0F807C0F0078060078 0000F80000F00001E00001C0000380000700000E00001C0000380000600000C000018000 0300200600200800401000403FFFC07FFF80FFFF80161E7E9D17>I<007F000183C00201 E00400F00700F00F00F00F01F00F01F00001E00001E00003C0000380000700000E0000F8 00000E000007000007800007C00003C00007C03007C07807C0F807C0F807C0F00780800F 00400E00201C0018780007E000141F7D9D17>I<01803001FFE003FFC003FF0003FC0002 0000020000020000040000040000040000047C000587000603800C01800801C00001C000 01E00001E00001E00001E07003C0F803C0F003C0E00380800780400700400E00201C0018 700007C000141F7D9D17>53 D<0000100000001800000038000000380000007800000078 000000FC000001BC0000013C0000033C0000023C0000063C0000043E0000081E0000081E 0000101E0000101E0000201E0000200F0000400F0000400F0000FFFF0000800F0001000F 8001000780020007800200078004000780040007800C0007C03E0007C0FF807FFC1E207E 9F22>65 D<07FFFF00007C01C0003C01E0003C00F0007800F8007800F8007800F8007800 F8007800F8007800F000F001F000F001E000F003C000F00F8000FFFE0000F00F0001E007 C001E003C001E003E001E001E001E001E001E001E003C001E003C003E003C003E003C003 C003C007C003C00F8007800F0007803E00FFFFF0001D1F7E9E20>I<0001F808000E0618 00380138007000F801E0007803C0007007800030078000300F0000301F0000301E000030 3E0000203C0000007C0000007C0000007C0000007C000000F8000000F8000000F8000000 F8000000F80000007800004078000080780000803C0000803C0001001C0002000E000200 06000C000300100001C0E000003F00001D217B9F21>I<07FFFF00007C01E0003C00F000 3C00780078003C0078003C0078001E0078001E0078001E0078001F00F0001F00F0001F00 F0001F00F0001F00F0001F00F0001F01E0001E01E0003E01E0003E01E0003E01E0003C01 E0007C03C0007803C000F003C000F003C001E003C003C003C0078007800F0007803C00FF FFE000201F7E9E23>I<07FFFFF8007C0078003C0038003C001800780018007800080078 000800780008007800080078080800F0100000F0100000F0100000F0300000FFF00000F0 700001E0200001E0200001E0200001E0200001E0000801E0001003C0001003C0001003C0 002003C0002003C0006003C000C0078001C0078007C0FFFFFF801D1F7E9E1F>I<07FFFF F8007C0078003C0038003C00180078001800780008007800080078000800780008007800 0800F0100000F0100000F0100000F0300000F0700000FFF00001E0600001E0200001E020 0001E0200001E0200001E0000003C0000003C0000003C0000003C0000003C0000003C000 000780000007C00000FFFE00001D1F7E9E1E>I<0001FC04000F030C003C009C0070007C 00E0003C01C0003803800018078000180F0000181F0000181E0000183E0000103C000000 7C0000007C0000007C0000007C000000F8000000F8000000F8007FFCF80003E0780001E0 780001E0780003C0780003C03C0003C03C0003C01C0003C00E0007C007000B8003801180 01E06080003F80001E217B9F24>I<07FFC7FFC0007C00F800003C007800003C00780000 7800F000007800F000007800F000007800F000007800F000007800F00000F001E00000F0 01E00000F001E00000F001E00000FFFFE00000F001E00001E003C00001E003C00001E003 C00001E003C00001E003C00001E003C00003C007800003C007800003C007800003C00780 0003C007800003C007800007800F000007C00F8000FFF8FFF800221F7E9E22>I<07FFE0 007C00003C00003C0000780000780000780000780000780000780000F00000F00000F000 00F00000F00000F00001E00001E00001E00001E00001E00001E00003C00003C00003C000 03C00003C00003C00007800007C000FFFC00131F7F9E10>I<07FFF000007E0000003C00 00003C000000780000007800000078000000780000007800000078000000F0000000F000 0000F0000000F0000000F0000000F0000001E0000001E0000001E0000001E0000001E000 8001E0010003C0010003C0010003C0030003C0020003C0060003C0060007801E0007807C 00FFFFFC00191F7E9E1C>76 D<07FC0000FFC0007C0000F800003C00017800003C000178 00004E0002F000004E0002F000004E0004F000004E0004F000004E0008F000004E0008F0 0000870011E00000870011E00000870021E00000870021E00000870041E00000838041E0 0001038083C00001038083C00001038103C00001038203C0000101C203C0000101C403C0 000201C40780000201C80780000201C80780000201D00780000200F00780000600E00780 000600E00F00000F00C00F8000FFE0C1FFF8002A1F7E9E2A>I<07FC01FFC0003E003E00 003E001800003E001800004F001000004F001000004780100000478010000043C0100000 43C010000083C020000081E020000081E020000080F020000080F0200000807820000100 78400001007C400001003C400001003C400001001E400001001E400002000F800002000F 800002000F800002000780000200078000060003800006000300000F00010000FFE00100 00221F7E9E22>I<0003F800001E0E000038070000E0038001C001C003C001E0078000E0 0F0000F00F0000F01E0000F01E0000F83E0000F83C0000F87C0000F87C0000F87C0000F8 7C0000F8F80001F0F80001F0F80001F0F80001F0F80003E0780003E0780003C0780007C0 7C0007803C000F003C001E001E001C000E0038000700F00003C3C00000FE00001D217B9F 23>I<07FFFF00007C03C0003C01E0003C00F0007800F0007800F8007800F8007800F800 7800F8007800F000F001F000F001E000F003C000F0078000F00F0000FFF80001E0000001 E0000001E0000001E0000001E0000001E0000003C0000003C0000003C0000003C0000003 C0000003C000000780000007C00000FFFC00001D1F7E9E1F>I<07FFFC00007C0700003C 03C0003C01E0007801E0007801F0007801F0007801F0007801F0007801E000F003E000F0 03C000F0078000F00F0000F03C0000FFF00001E0300001E0380001E01C0001E01C0001E0 1C0001E01E0003C03E0003C03E0003C03E0003C03E0003C03E0003C03E0207803E0407C0 1F04FFFC0F18000003E01F207E9E21>82 D<003F040060CC01803C03801C03001C070018 0600080E00080E00080E00080E00000F00000F80000FE00007FE0003FF8001FFC0007FE0 0007E00001E00000E00000F00000F04000E04000E04000E04000E06000C0600180E00380 F80300C60C0081F80016217D9F19>I<3FFFFFF03C0780F03007803060078030400F0010 400F0010C00F0010800F0010800F0010800F0010001E0000001E0000001E0000001E0000 001E0000001E0000003C0000003C0000003C0000003C0000003C0000003C000000780000 007800000078000000780000007800000078000000F0000001F800007FFFE0001C1F7A9E 21>II87 D 89 D E /Fh 37 123 df<387CFEFEFE7C3807077C860F>46 D<01FC0007FF001F07C01E 03C03E03E07C01F07C01F07C01F0FC01F8FC01F8FC01F8FC01F8FC01F8FC01F8FC01F8FC 01F8FC01F8FC01F8FC01F8FC01F8FC01F87C01F07C01F07C01F03E03E01E03C01F8FC007 FF0001FC00151D7E9C1A>48 D<00E00001E0000FE000FFE000F3E00003E00003E00003E0 0003E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E0 0003E00003E00003E00003E00003E00003E00003E000FFFF80FFFF80111D7C9C1A>I<07 F0001FFE00383F007C1F80FE0FC0FE0FC0FE0FE0FE07E07C07E03807E0000FE0000FC000 0FC0001F80001F00003E0000780000F00000E00001C0000380600700600E00601C00E01F FFC03FFFC07FFFC0FFFFC0FFFFC0131D7D9C1A>I<01FC0007FF000E0F801E0FC03F07E0 3F07E03F07E03F07E01E0FC0000FC0000F80001F0001FC0001FC00000F800007C00003E0 0003F00003F83803F87C03F8FE03F8FE03F8FE03F0FC03F07807E03C0FC01FFF8003FC00 151D7E9C1A>I<0001C00003C00007C00007C0000FC0001FC0003BC00073C00063C000C3 C00183C00383C00703C00E03C00C03C01803C03803C07003C0E003C0FFFFFEFFFFFE0007 C00007C00007C00007C00007C00007C000FFFE00FFFE171D7F9C1A>I<3803803FFF803F FF003FFE003FFC003FF0003F800030000030000030000030000033F80037FE003C1F0038 0F801007C00007C00007E00007E07807E0FC07E0FC07E0FC07E0FC07C0780FC0600F8038 1F001FFC0007F000131D7D9C1A>I<003F0001FFC007E0E00F81E01F03F01E03F03E03F0 7C03F07C01E07C0000FC1000FCFF00FDFFC0FD03E0FE01F0FE01F0FC01F8FC01F8FC01F8 FC01F87C01F87C01F87C01F83C01F03E01F01E03E00F07C007FF8001FE00151D7E9C1A> I<6000007FFFF87FFFF87FFFF07FFFE07FFFC0E00180C00300C00300C00600000C000018 0000380000380000780000700000F00000F00001F00001F00001F00001F00003F00003F0 0003F00003F00003F00003F00001E00000C000151E7D9D1A>I<01FC0007FF000F07801E 03C01C01E03C01E03C01E03E01E03F01E03FC3C01FE3801FFF000FFE0007FF8007FFC01F FFE03C3FF0780FF07803F8F001F8F000F8F00078F00078F000707800707C00E03E03C00F FF8003FC00151D7E9C1A>I<01FC000FFF001F07803E03C07C03E07C01E0FC01F0FC01F0 FC01F0FC01F8FC01F8FC01F8FC01F87C03F87C03F83E05F81FFDF807F9F80041F80001F0 3C01F07E01F07E03E07E03E07E07C03C0780381F001FFC0007F000151D7E9C1A>I<07F8 001FFE00381F80780F80FC0FC0FC0FC0FC0FC0780FC0301F80001F00003E00007C000070 0000E00000E00000C00000C00000C00000C00000C00000C0000000000000000000000000 0001C00003E00007F00007F00007F00003E00001C00012207D9F19>63 D<0007FC02003FFF0E00FE03DE03F000FE07E0003E0FC0001E1F80001E3F00000E3F0000 0E7F0000067E0000067E000006FE000000FE000000FE000000FE000000FE000000FE0000 00FE0000007E0000007E0000067F0000063F0000063F00000C1F80000C0FC0001807E000 3803F0007000FE01C0003FFF800007FC001F1F7D9E26>67 D69 D73 D77 D80 D<07FC001FFF003F0F803F07C03F03E03F03E00C03E00003 E0007FE007FBE01F03E03C03E07C03E0F803E0F803E0F803E0FC05E07E0DE03FF8FE0FE0 7E17147F9319>97 DI<01FE0007FF801F0FC03E0FC03E0FC07C0FC07C0300FC0000FC0000FC0000FC00 00FC0000FC00007C00007E00003E00603F00C01F81C007FF0001FC0013147E9317>I<01 FE0007FF800F83C01E01E03E00F07C00F07C00F8FC00F8FFFFF8FFFFF8FC0000FC0000FC 00007C00007C00003E00181E00180F807007FFE000FF8015147F9318>101 D<001F8000FFC001F3E003E7E003C7E007C7E007C3C007C00007C00007C00007C00007C0 00FFFC00FFFC0007C00007C00007C00007C00007C00007C00007C00007C00007C00007C0 0007C00007C00007C00007C00007C00007C0003FFC003FFC0013207F9F10>I<01FC3C07 FFFE0F079E1E03DE3E03E03E03E03E03E03E03E03E03E01E03C00F07800FFF0009FC0018 00001800001C00001FFF800FFFF007FFF81FFFFC3C007C70003EF0001EF0001EF0001E78 003C78003C3F01F80FFFE001FF00171E7F931A>I<1C003E007F007F007F003E001C0000 0000000000000000000000FF00FF001F001F001F001F001F001F001F001F001F001F001F 001F001F001F001F001F00FFE0FFE00B217EA00E>105 D<0038007C00FE00FE00FE007C 003800000000000000000000000001FE01FE003E003E003E003E003E003E003E003E003E 003E003E003E003E003E003E003E003E003E003E003E303E783EFC3CFC7C78783FF01FC0 0F2A83A010>I108 DII<01FF0007FFC01F83F03E00F83E00F87C007C7C007CFC007EFC007EFC007EFC007EFC 007EFC007E7C007C7C007C3E00F83E00F81F83F007FFC001FF0017147F931A>II<01F81807FE381F87783F01F83E01F87E00 F87C00F8FC00F8FC00F8FC00F8FC00F8FC00F8FC00F87C00F87E00F87E00F83F01F81F87 F80FFEF803F8F80000F80000F80000F80000F80000F80000F80000F80007FF0007FF181D 7E931C>II< 0FE63FFE701E600EE006E006F800FFC07FF83FFC1FFE03FE001FC007C007E007F006F81E FFFCC7F010147E9315>I<01800180018003800380038007800F803F80FFFCFFFC0F800F 800F800F800F800F800F800F800F800F800F860F860F860F860F8607CC03F801F00F1D7F 9C14>II120 D<3FFFE03FFFE03C07C0380F80701F80603F00603E 00607C0000F80001F80003F00003E06007C0600F80601F80E03F00C03E01C07C03C0FFFF C0FFFFC013147F9317>122 D E /Fi 37 120 df<00003F03E00000C386700001878CF0 0003879CF00003031860000700380000070038000007003800000E003800000E00700000 0E007000000E00700000FFFFFF80001C007000001C00E000001C00E000001C00E000001C 00E000003800E000003801C000003801C000003801C000003801C000007001C000007003 8000007003800000700380000070038000006003800000E007000000E007000000E00700 0000E007000000C006000001C00E000001C00E000031860C0000798F180000F31E100000 620C6000003C07C000002429829F1C>11 D<00003FE00000E01000018038000380780003 007800070030000700000007000000070000000E0000000E0000000E000000FFFFE0000E 00E0001C01C0001C01C0001C01C0001C01C0001C03800038038000380380003803800038 070000380700007007000070071000700E2000700E2000700E2000E00E2000E0064000E0 038000E0000000C0000001C0000001C000003180000079800000F3000000620000003C00 00001D29829F1A>I<0E1F3F3F1D0102020404081020C0080E779F0E>39 D<000100020004000800100020006000C0018001800300070006000E000C001C00180038 00380030007000700060006000E000E000C000C000C000C000C000C000C000C000C000C0 00C000C000C0004000600060002000100010000800102E79A113>I<0010000008000004 000006000002000003000003000003000001000001800001800001800001800001800001 80000180000380000380000380000300000300000300000700000700000600000600000E 00000C00000C00001C0000180000380000300000700000600000E00000C0000180000100 000300000600000C0000180000300000600000800000112E80A113>I<7FF0FFE07FE00C 037D8A10>45 D<70F8F8F0E005057B840E>I<00000040000000C0000001800000018000 00030000000300000006000000060000000C000000180000001800000030000000300000 0060000000C0000000C0000001800000018000000300000003000000060000000C000000 0C0000001800000018000000300000003000000060000000C0000000C000000180000001 8000000300000003000000060000000C0000000C00000018000000180000003000000030 00000060000000C0000000C0000000800000001A2D7FA117>I<000F800030E000E07001 C0700380300380380700380F00780F00780E00781E00781E00703C00F03C00F03C00F03C 00F07801E07801E07801E07801C07003C0F003C0F00380F00780F00700700700700E0070 1C003038001870000FC000151F7C9D17>I<000200020006000E003C00DC031C001C0038 003800380038007000700070007000E000E000E000E001C001C001C001C0038003800380 03800780FFF80F1E7B9D17>I<0007C0001C200030200060E000C1E00181E00380C00700 000F00000E00001E00001E78001D84003E06003E07003C07007C07807807807807807807 80700F00700F00F00F00F00E00F01E00701C00601C0070380030700010C0000F8000131F 7B9D17>54 D<070F1F1F0E0000000000000000000070F8F8F0E008147B930E>58 D<00000200000006000000060000000E0000001E0000001E0000003F0000002F0000004F 0000004F0000008F0000010F0000010F0000020F0000020F0000040F00000C0F0000080F 0000100F0000100F0000200F80003FFF800040078000C007800080078001000780010007 800200078002000780060007801E000F80FF807FF81D207E9F22>65 D<01FFFFC0001E00F0001E0078001E0038001E003C003C003C003C003C003C003C003C00 3C0078007800780078007800F0007801E000F0078000FFFE0000F00F8000F003C001E001 C001E001E001E001E001E001E003C001E003C001E003C001E003C001C0078003C0078007 8007800F0007801E000F007800FFFFE0001E1F7D9E20>I<01FFFFFE001E001C001E000C 001E0004001E0004003C0004003C0004003C0004003C0004007808080078080000780800 0078180000F0300000FFF00000F0300000F0300001E0200001E0200001E0200001E00010 03C0002003C0002003C0004003C00040078000800780018007800100078007000F001F00 FFFFFE001F1F7D9E1F>69 D<01FFFFFC001E0038001E0018001E0008001E0008003C0008 003C0008003C0008003C00080078001000780800007808000078080000F0100000F03000 00FFF00000F0300001E0200001E0200001E0200001E0200003C0000003C0000003C00000 03C00000078000000780000007800000078000000F800000FFF800001E1F7D9E1E>I<01 FFFF00001E03C0001E00E0001E0070001E0078003C0078003C0078003C0078003C007800 7800F0007800F0007801E0007801C000F0070000F01E0000FFF00000F0380001E01C0001 E01E0001E00E0001E00F0003C01E0003C01E0003C01E0003C01E0007803C0007803C0807 803C0807803C100F801C10FFF00C20000007C01D207D9E21>82 D<0007E040001C18C000 3005800060038000C0038001C00180018001000380010003800100038001000380000003 C0000003C0000003F8000001FF800001FFE000007FF000001FF0000001F8000000780000 007800000038000000380020003800200038002000300060007000600060006000E00070 00C000E8038000C606000081F800001A217D9F1A>I<00F1800389C00707800E03801C03 803C0380380700780700780700780700F00E00F00E00F00E00F00E20F01C40F01C40703C 40705C40308C800F070013147C9317>97 D<007E0001C1000300800E07801E07801C0700 3C0200780000780000780000F00000F00000F00000F00000F00000700100700200300400 18380007C00011147C9315>99 D<0000780003F80000700000700000700000700000E000 00E00000E00000E00001C00001C000F1C00389C00707800E03801C03803C038038070078 0700780700780700F00E00F00E00F00E00F00E20F01C40F01C40703C40705C40308C800F 070015207C9F17>I<007C01C207010E011C013C013802780C7BF07C00F000F000F000F0 007000700170023804183807C010147C9315>I<00007800019C00033C00033C00071800 0700000700000E00000E00000E00000E00000E0001FFE0001C00001C00001C00001C0000 380000380000380000380000380000700000700000700000700000700000700000E00000 E00000E00000E00000C00001C00001C0000180003180007B0000F300006600003C000016 29829F0E>I<003C6000E27001C1E00380E00700E00F00E00E01C01E01C01E01C01E01C0 3C03803C03803C03803C03803C07003C07001C0F001C17000C2E0003CE00000E00000E00 001C00001C00301C00783800F0700060E0003F8000141D7E9315>I<01E0000FE00001C0 0001C00001C00001C000038000038000038000038000070000070000071E000763000E81 800F01C00E01C00E01C01C03801C03801C03801C0380380700380700380700380E10700E 20700C20701C20700C40E00CC060070014207D9F17>I<00C001E001E001C00000000000 0000000000000000000E003300230043804300470087000E000E000E001C001C001C0038 40388030807080310033001C000B1F7C9E0E>I<03C01FC0038003800380038007000700 070007000E000E000E000E001C001C001C001C0038003800380038007000700070007100 E200E200E200E200640038000A207C9F0C>108 D<1C0F80F0002630C318004740640C00 4780680E004700700E004700700E008E00E01C000E00E01C000E00E01C000E00E01C001C 01C038001C01C038001C01C038001C01C0708038038071003803806100380380E1003803 8062007007006600300300380021147C9325>I<1C0F802630C047406047806047007047 00708E00E00E00E00E00E00E00E01C01C01C01C01C01C01C038438038838030838070838 03107003303001C016147C931A>I<007C0001C3000301800E01C01E01C01C01E03C01E0 7801E07801E07801E0F003C0F003C0F003C0F00780F00700700F00700E00301800187000 07C00013147C9317>I<01C1E002621804741C04781C04701E04701E08E01E00E01E00E0 1E00E01E01C03C01C03C01C03C01C0380380780380700380E003C1C0072380071E000700 000700000E00000E00000E00000E00001C00001C0000FFC000171D809317>I<1C1E0026 61004783804787804707804703008E00000E00000E00000E00001C00001C00001C00001C 000038000038000038000038000070000030000011147C9313>114 D<00FC030206010C030C070C060C000F800FF007F803FC003E000E700EF00CF00CE00840 1020601F8010147D9313>I<018001C0038003800380038007000700FFF007000E000E00 0E000E001C001C001C001C003800380038003820704070407080708031001E000C1C7C9B 0F>I<0E00C03300E02301C04381C04301C04701C08703800E03800E03800E03801C0700 1C07001C07001C07101C0E20180E20180E201C1E200C264007C38014147C9318>I<0E03 803307802307C04383C04301C04700C08700800E00800E00800E00801C01001C01001C01 001C02001C02001C04001C04001C08000E300003C00012147C9315>I<0E00C1C03300E3 C02301C3E04381C1E04301C0E04701C060870380400E0380400E0380400E0380401C0700 801C0700801C0700801C0701001C0701001C0602001C0F02000C0F04000E13080003E1F0 001B147C931E>I E /Fj 58 122 df<007F07F001FF1FF003FF3FF00780780007007000 0F00F0000F00F0000F00F0000F00F0000F00F0000F00F0000F00F000FFF8FF80FFF8FF80 FFF8FF800F00F0000F00F0000F00F0000F00F0000F00F0000F00F0000F00F0000F00F000 0F00F0000F00F0000F00F0000F00F0000F00F0000F00F0000F00F0000F00F0000F00F000 1C20809F1B>11 D<00F8F001F8F003F8F00780F00700F00F00F00F00F00F00F00F00F00F 00F00F00F00F00F0FFF8F0FFF8F0FFF8F00F00F00F00F00F00F00F00F00F00F00F00F00F 00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F01420809F 18>13 D<007000E001C00380078007000E001E001E003C003C003C007800780078007800 7000F000F000F000F000F000F000F000F000F000F000F000F00070007800780078007800 3C003C003C001E001E000E0007000780038001C000E000700C2E7EA112>40 DI<0006000000060000 000600000006000000060000000600000006000000060000000600000006000000060000 00060000000600000006000000060000FFFFFFF0FFFFFFF0000600000006000000060000 000600000006000000060000000600000006000000060000000600000006000000060000 0006000000060000000600001C207D9A23>43 D<787878781830306060E0050A7D830D> III<00C001C00FC0FFC0FFC0F3C0 03C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C0 03C003C003C003C0FFFEFFFEFFFE0F1F7C9E17>49 D<07F0000FFC001FFE00383F00700F 00600780E00780E003C04003C04003C00003C00003C00003C0000780000780000F00000E 00001C00003C0000780000E00001C0000380000700000E00001C0000380000700000FFFF C0FFFFC0FFFFC0121F7E9E17>I61 D<001F0000001F0000003F8000003B8000003B8000007BC0000073C0000071C00000F1E0 0000E1E00000E0E00001E0F00001E0F00001C0F00003C0780003C078000380780007803C 0007803C0007003C000FFFFE000FFFFE000FFFFE001E000F001E000F003C000F803C0007 803C000780780007C0780003C0780003C0F00003E01B207F9F1E>65 DI<001FC000 FFF801FFFC03E03C07800C0F00001E00003E00003C00007C0000780000780000780000F0 0000F00000F00000F00000F00000F00000F00000F000007800007800007800007C00003C 00003E00001E00000F000207800E03E03E01FFFC00FFF0001FC017227DA01D>IIII<001FE000FFF801FFFE03E03E07800E0F0000 1E00003E00003C00007C0000780000780000780000F00000F00000F00000F00000F00000 F00000F003FEF003FE7803FE78001E78001E7C001E3C001E3E001E1E001E0F001E07801E 03E03E01FFFE00FFF8001FC017227DA01E>III75 DIII<003F000000FF C00003FFF00007E1F8000F807C001F003E001E001E003C000F003C000F00780007807800 078078000780F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F000 03C0F00003C0F80007C07800078078000780780007803C000F003C000F001E001E001F00 3E000F807C0007E1F80003FFF00000FFC000003F00001A227DA021>II82 D<01FC0007FF800FFFC01F03C03C00C03C 00007800007800007800007800007800007C00003C00003F00001FE0000FFC0007FE0001 FF00003F800007C00003C00003E00001E00001E00001E00001E00001E00001C0C003C0F0 07C0FC0F807FFF001FFE0003F80013227EA019>III II<780007807C000F003E001F00 1E001E000F003C000F807C000780780003C0F00003E1F00001F1E00000F3C000007FC000 007F8000003F0000001F0000001E0000003F0000007F8000007FC00000F3C00001F1E000 01E0F00003C0F80007C0780007803C000F003E001F001E001E000F003C000F807C0007C0 780003C0F00003E01B207F9F1E>II I<07E03FF87FFC701E401F000F000F000F003F07FF1FFF7E0FF80FF00FF00FF00FF83F7F FF3FEF1F8F10147E9316>97 DI<03F00FFC1FFE3E0E3C0278007800F000F000F000F000F000F0007800 78003C013E0F1FFF0FFE03F010147E9314>I<0007800007800007800007800007800007 8000078000078000078000078000078000078007C7800FF7801FFF803E1F807C07807807 80F80780F00780F00780F00780F00780F00780F00780F00780780780780F803E1F801FFF 800FF78007C78011207E9F17>I<03F0000FFC001FFE003E1F003C0700780700700380FF FF80FFFF80FFFF80F00000F00000F000007000007800003C01003E07001FFF0007FE0001 F80011147F9314>I<007E01FE03FE078007000F000F000F000F000F000F000F00FFF0FF F0FFF00F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F 000F20809F0E>I<03E0F00FFFF01FFFF03E3E003C1E00780F00780F00780F00780F0078 0F003C1E003E3E001FFC003FF80033E0003000003800003FFE003FFF801FFFC03FFFE078 03F0F000F0F000F0F000F0F801F07E07E03FFFC00FFF0003FC00141E7F9317>III107 DIII<01F80007FE00 1FFF803F0FC03C03C07801E07801E0F000F0F000F0F000F0F000F0F000F0F000F07801E0 7801E03C03C03F0FC01FFF8007FE0001F80014147F9317>II114 D<07F01FFC3FFC780C78007800 78007C003FC01FF00FF803F8007C003C003CC03CF07CFFF87FF00FC00E147F9311>I<1E 001E001E001E001E001E00FFF0FFF0FFF01E001E001E001E001E001E001E001E001E001E 001E001E001E001E201FF00FF007C00C1A7F9910>IIII<7801E07C03C03E07801E0F000F0F00079E0003FC0003F80001F80000F0 0001F00001F80003FC00079E000F0F000E0F001E07803C03C07801E0F801F01414809315 >II E /Fk 15 117 df<020408103020604040C0C0C0C0C0C0C0C0404060203010080402071A7F920C >40 D<8040201018080C0404060606060606060604040C081810204080071A7E920C>I< 1F00318060C04040C060C060C060C060C060C060C060C060404060C031801F000B107F8F 0F>48 D<0C003C00CC000C000C000C000C000C000C000C000C000C000C000C000C00FF80 09107E8F0F>I<1F00618040C08060C0600060006000C00180030006000C00102020207F C0FFC00B107F8F0F>I<1F00218060C060C000C0008001800F00008000400060C060C060 804060801F000B107F8F0F>I<0300030007000F000B001300330023004300C300FFE003 000300030003001FE00B107F8F0F>I<20803F002C002000200020002F00308020400060 00600060C06080C061801F000B107F8F0F>I<0780184030C060C06000C000CF00F080E0 40C060C060C060406060C030801F000B107F8F0F>I<40007FE07FC08080808001000200 040004000C0008000800180018001800180018000B117E900F>I<1F00318060C060C060 C071803F000F00338061C0C060C060C060404060801F000B107F8F0F>I<1F00318060C0 C040C060C060C06040E021E01E600060004060C0608043003E000B107F8F0F>I<03E000 0C1800100400200200600300400100C00180C00180C00180C00180C00180600300600300 3006003006000C180003E00011117E9017>79 D<1F8030C06000C000C000C000C000C000 604030801F000A0B7F8A0E>99 D<10103030FE3030303030323232321C070F7F8E0C> 116 D E /Fl 7 104 df0 D21 D<00000C00000C0000180000180000300000300000600000600000C00000C00001800001 80000180000300000300000600000600000C00000C000018000018000030000030000060 0000600000C00000C0000180000180000300000300000600000600000600000C00000C00 00180000180000300000300000600000600000C00000400000162C7AA000>54 D62 D<00040000000C0000000C0000000C0000000C0000000C00 00000C0000000C0000000C0000000C0000000C0000000C0000000C0000000C0000000C00 00000C0000000C0000000C0000000C0000000C0000000C0000000C0000000C0000000C00 00000C0000000C0000FFFFFFE0FFFFFFE01B1C7C9B23>I<000F0038006000E001C001C0 01C001C001C001C001C001C001C001C001C001C001C001C001C0038007001E00F8001E00 0700038001C001C001C001C001C001C001C001C001C001C001C001C001C001C001C000E0 00600038000F102D7DA117>102 DI E /Fm 77 126 df<70F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F870000000000070 F8F8F870051C779B18>33 D<4010E038F078E038E038E038E038E038E038E038E038E038 E03860300D0E7B9C18>I<01C00007E0000FF0000E70001C38001C38001C38001C38001C 73F01C73F01CE3F00FE3800FC7000F87000F07001F0E003F0E007B8E0073DC00E1DC00E0 F800E0F800E07070E0787070FC707FFFE03FCFE00F03C0141C7F9B18>38 D<007000F001E003C007800F001E001C00380038007000700070007000E000E000E000E0 00E000E000E000E0007000700070007000380038001C001E000F00078003C001F000F000 700C24799F18>40 D<6000F00078003C001E000F000780038001C001C000E000E000E000 E00070007000700070007000700070007000E000E000E000E001C001C0038007800F001E 003C007800F00060000C247C9F18>I<01C00001C00001C00001C000C1C180F1C780F9CF 807FFF001FFC0007F00007F0001FFC007FFF00F9CF80F1C780C1C18001C00001C00001C0 0001C00011147D9718>I<1C3E7E7F3F1F070E1E7CF860080C788518>44 D<7FFF00FFFF80FFFF807FFF0011047D8F18>I<3078FCFC78300606778518>I<00030000 0780000780000F80000F00001F00001E00001E00003E00003C00007C0000780000780000 F80000F00001F00001E00003E00003C00003C00007C0000780000F80000F00000F00001F 00001E00003E00003C00003C00007C0000780000F80000F00000F0000060000011247D9F 18>I<01F00007FC000FFE001F1F001C07003803807803C07001C07001C0E000E0E000E0 E000E0E000E0E000E0E000E0E000E0E000E0E000E0F001E07001C07001C07803C0380380 1C07001F1F000FFE0007FC0001F000131C7E9B18>I<01800380038007800F803F80FF80 FB8043800380038003800380038003800380038003800380038003800380038003800380 7FFCFFFE7FFC0F1C7B9B18>I<03F0000FFE003FFF007C0F807003C0E001C0F000E0F000 E06000E00000E00000E00001C00001C00003C0000780000F00001E00003C0000780000F0 0001E00007C0000F80001E00E03C00E07FFFE0FFFFE07FFFE0131C7E9B18>I<07F8001F FE003FFF007807807803C07801C03001C00001C00003C0000380000F0003FF0003FE0003 FF000007800003C00001C00000E00000E00000E0F000E0F000E0F001C0F003C07C07803F FF001FFE0003F800131C7E9B18>I<1FFF803FFF803FFF80380000380000380000380000 3800003800003800003800003BF8003FFE003FFF003C07801803C00001C00000E00000E0 6000E0F000E0F000E0E001C07003C07C0F803FFF001FFC0003F000131C7E9B18>53 D<007E0001FF0007FF800F83C01E03C01C03C0380180380000700000700000E1F800E7FE 00FFFF00FE0780F803C0F001C0F000E0E000E0F000E07000E07000E07000E03801C03C03 C01E07800FFF0007FE0001F800131C7E9B18>I<3078FCFC783000000000000000003078 FCFC78300614779318>58 D<183C7E7E3C180000000000000000183C7E7E3E1E0E1C3C78 F060071A789318>I<000300000780001F80003F00007E0001FC0003F00007E0001FC000 3F00007E0000FC0000FC00007E00003F00001FC00007E00003F00001FC00007E00003F00 001F8000078000030011187D9918>I<7FFFC0FFFFE0FFFFE0FFFFE00000000000000000 00000000FFFFE0FFFFE0FFFFE07FFFC0130C7E9318>I<600000F00000FC00007E00003F 00001FC00007E00003F00001FC00007E00003F00001F80001F80003F00007E0001FC0003 F00007E0001FC0003F00007E0000FC0000F0000060000011187D9918>I<0FF0003FFC00 7FFF00700F00F00380F00380600780000F00003E00007C0001F00001E00003C00003C000 03C00003C00003C00003800000000000000000000000000000000003800007C00007C000 07C000038000111C7D9B18>I<00700000F80000F80000D80000D80001DC0001DC0001DC 00018C00038E00038E00038E00038E000306000707000707000707000707000FFF800FFF 800FFF800E03800E03801C01C01C01C07F07F0FF8FF87F07F0151C7F9B18>65 DI<00F8E003FEE007FFE00F07E01E03E03C 01E03800E07000E07000E0700000E00000E00000E00000E00000E00000E00000E00000E0 00007000007000E07000E03800E03C00E01E01C00F07C007FF8003FE0000F800131C7E9B 18>I<7FF800FFFE007FFF001C0F801C03C01C03C01C01E01C00E01C00E01C00F01C0070 1C00701C00701C00701C00701C00701C00701C00701C00F01C00E01C00E01C01E01C01C0 1C03C01C0F807FFF00FFFE007FF800141C7F9B18>III<01F1C003FDC00FFFC01F0FC0 1C03C03803C03801C07001C07001C0700000E00000E00000E00000E00000E00000E00FF0 E01FF0E00FF07001C07001C07003C03803C03803C01C07C01F0FC00FFFC003FDC001F1C0 141C7E9B18>I<7F07F0FF8FF87F07F01C01C01C01C01C01C01C01C01C01C01C01C01C01 C01C01C01C01C01FFFC01FFFC01FFFC01C01C01C01C01C01C01C01C01C01C01C01C01C01 C01C01C01C01C01C01C07F07F0FF8FF87F07F0151C7F9B18>I<7FFF00FFFF807FFF0001 C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001 C00001C00001C00001C00001C00001C00001C00001C00001C00001C0007FFF00FFFF807F FF00111C7D9B18>I<7F07F0FF87F87F07F01C03C01C07801C07001C0E001C1E001C3C00 1C38001C70001CF0001DF0001DF0001FB8001FB8001F1C001E1C001C0E001C0E001C0700 1C07001C03801C03801C01C07F03F0FF87F87F03F0151C7F9B18>75 D<7FE000FFE0007FE0000E00000E00000E00000E00000E00000E00000E00000E00000E00 000E00000E00000E00000E00000E00000E00000E00000E00000E00700E00700E00700E00 700E00707FFFF0FFFFF07FFFF0141C7F9B18>II<7E07F0FF0FF87F07F01D81C01D81C01D81C01DC1C01CC1C01CC1C01CE1C01CE1C0 1CE1C01C61C01C71C01C71C01C31C01C39C01C39C01C39C01C19C01C19C01C1DC01C0DC0 1C0DC01C0DC07F07C0FF87C07F03C0151C7F9B18>I<0FF8003FFE007FFF00780F007007 00F00780E00380E00380E00380E00380E00380E00380E00380E00380E00380E00380E003 80E00380E00380E00380E00380E00380F00780700700780F007FFF003FFE000FF800111C 7D9B18>II<0FF8003FFE007FFF00780F00 700700F00780E00380E00380E00380E00380E00380E00380E00380E00380E00380E00380 E00380E00380E00380E00380E1E380E1E380F0E78070F700787F007FFF003FFE000FFC00 001C00001E00000E00000F0000070000070011227D9B18>I<7FF800FFFE007FFF001C0F 801C03801C03C01C01C01C01C01C01C01C03C01C03801C0F801FFF001FFE001FFE001C0F 001C07001C03801C03801C03801C03801C03801C039C1C039C1C039C7F01F8FF81F87F00 F0161C7F9B18>I<03F3801FFF803FFF807C0F80700780E00380E00380E00380E0000070 00007800003F00001FF00007FE0000FF00000F800003C00001C00000E00000E06000E0E0 00E0E001E0F001C0F80780FFFF80FFFE00E7F800131C7E9B18>I<7FFFF8FFFFF8FFFFF8 E07038E07038E07038E07038007000007000007000007000007000007000007000007000 00700000700000700000700000700000700000700000700000700000700007FF0007FF00 07FF00151C7F9B18>IIII<7F8FE07F9F E07F8FE00E07000F0700070E00078E00039C0003DC0001F80001F80000F00000F0000070 0000F00000F80001F80001DC00039E00038E00070F000707000E07800E03801E03C07F07 F0FF8FF87F07F0151C7F9B18>II<3FFFE0 7FFFE07FFFE07001C07003C0700780700700000F00001E00001C00003C00007800007000 00F00001E00001C00003C0000780000700000F00001E00E01C00E03C00E07800E07000E0 FFFFE0FFFFE0FFFFE0131C7E9B18>II93 D<7FFF00FFFF80FFFF807FFF0011047D7F18>95 D<1FE0003FF8007FFC00781E00300E00 00070000070000FF0007FF001FFF007F0700780700E00700E00700E00700F00F00781F00 3FFFF01FFBF007E1F014147D9318>97 D<7E0000FE00007E00000E00000E00000E00000E 00000E00000E3E000EFF800FFFC00FC1E00F80E00F00700E00700E00380E00380E00380E 00380E00380E00380F00700F00700F80E00FC1E00FFFC00EFF80063E00151C809B18>I< 01FE0007FF001FFF803E0780380300700000700000E00000E00000E00000E00000E00000 E000007000007001C03801C03E03C01FFF8007FF0001FC0012147D9318>I<001F80003F 80001F8000038000038000038000038000038003E3800FFB801FFF803C1F80380F807007 80700380E00380E00380E00380E00380E00380E00380700780700780380F803C1F801FFF F00FFBF803E3F0151C7E9B18>I<01F00007FC001FFE003E0F00380780700380700380E0 01C0E001C0FFFFC0FFFFC0FFFFC0E000007000007001C03801C03E03C01FFF8007FF0001 FC0012147D9318>I<001F80007FC000FFE000E1E001C0C001C00001C00001C0007FFFC0 FFFFC0FFFFC001C00001C00001C00001C00001C00001C00001C00001C00001C00001C000 01C00001C00001C00001C0007FFF007FFF007FFF00131C7F9B18>I<01E1F007FFF80FFF F81E1E301C0E003807003807003807003807003807001C0E001E1E001FFC001FF80039E0 003800001C00001FFE001FFFC03FFFE07801F0700070E00038E00038E00038E000387800 F07E03F01FFFC00FFF8001FC00151F7F9318>I<7E0000FE00007E00000E00000E00000E 00000E00000E00000E3E000EFF800FFFC00FC1C00F80E00F00E00E00E00E00E00E00E00E 00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E07FC3FCFFE7FE7FC3FC171C809B 18>I<03800007C00007C00007C0000380000000000000000000000000007FC000FFC000 7FC00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C000 01C00001C00001C000FFFF00FFFF80FFFF00111D7C9C18>I107 D<7FE000FFE0007FE00000E00000E00000E00000E00000E00000E0 0000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E0 0000E00000E00000E00000E0007FFFC0FFFFE07FFFC0131C7E9B18>I<7CE0E000FFFBF8 007FFFF8001F1F1C001E1E1C001E1E1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C 001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C007F1F1F00FFBFBF807F1F1F 001914819318>I<7E3E00FEFF807FFFC00FC1C00F80E00F00E00E00E00E00E00E00E00E 00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E07FC3FCFFE7FE7FC3FC17148093 18>I<01F0000FFE001FFF003E0F803803807001C07001C0E000E0E000E0E000E0E000E0 E000E0F001E07001C07803C03C07803E0F801FFF000FFE0001F00013147E9318>I<7E3E 00FEFF807FFFC00FC1E00F80E00F00700E00700E00380E00380E00380E00380E00380E00 380F00700F00700F80E00FC1E00FFFC00EFF800E3E000E00000E00000E00000E00000E00 000E00000E00007FC000FFE0007FC000151E809318>I<01E38007FB801FFF803E1F8038 0F80700780700780E00380E00380E00380E00380E00380E00380700780700780380F803C 1F801FFF800FFB8003E380000380000380000380000380000380000380000380003FF800 3FF8003FF8151E7E9318>I<7F87E0FF9FF07FBFF803F87803F03003E00003C00003C000 0380000380000380000380000380000380000380000380000380007FFE00FFFF007FFE00 15147F9318>I<07F7003FFF007FFF00780F00E00700E00700E007007C00007FE0001FFC 0003FE00001F00600780E00380E00380F00380F80F00FFFF00FFFC00E7F00011147D9318 >I<0180000380000380000380000380007FFFC0FFFFC0FFFFC003800003800003800003 80000380000380000380000380000380000380400380E00380E00380E001C1C001FFC000 FF80003E0013197F9818>I<7E07E0FE0FE07E07E00E00E00E00E00E00E00E00E00E00E0 0E00E00E00E00E00E00E00E00E00E00E00E00E00E00E01E00F03E007FFFC03FFFE01FCFC 1714809318>I<7F8FF0FF8FF87F8FF01E03C00E03800E03800E03800707000707000707 00038E00038E00038E00038E0001DC0001DC0001DC0000F80000F80000700015147F9318 >II<7F8FF0 7F9FF07F8FF0070700078E00039E0001DC0001F80000F80000700000F00000F80001DC00 039E00038E000707000F07807F8FF0FF8FF87F8FF015147F9318>I<7F8FF0FF8FF87F8F F00E01C00E03800E0380070380070700070700038700038600038E0001CE0001CE0000CC 0000CC0000DC0000780000780000780000700000700000700000F00000E00079E0007BC0 007F80003F00001E0000151E7F9318>I<3FFFF07FFFF07FFFF07001E07003C070078000 0F00001E00003C0000F80001F00003C0000780000F00701E00703C0070780070FFFFF0FF FFF0FFFFF014147F9318>I<0007E0001FE0007FE000780000E00000E00000E00000E000 00E00000E00000E00000E00000E00000E00000E00001E0007FC000FF8000FF80007FC000 01E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E000 007800007FE0001FE00007E013247E9F18>I<7C0000FF0000FFC00003C00000E00000E0 0000E00000E00000E00000E00000E00000E00000E00000E00000E00000F000007FC0003F E0003FE0007FC000F00000E00000E00000E00000E00000E00000E00000E00000E00000E0 0000E00000E00003C000FFC000FF00007C000013247E9F18>125 D E /Fn 48 124 df<004000800100020006000C000C0018001800300030007000600060 006000E000E000E000E000E000E000E000E000E000E000E000E000600060006000700030 003000180018000C000C00060002000100008000400A2A7D9E10>40 D<800040002000100018000C000C000600060003000300038001800180018001C001C001 C001C001C001C001C001C001C001C001C001C0018001800180038003000300060006000C 000C00180010002000400080000A2A7E9E10>I<00060000000600000006000000060000 000600000006000000060000000600000006000000060000000600000006000000060000 FFFFFFE0FFFFFFE000060000000600000006000000060000000600000006000000060000 0006000000060000000600000006000000060000000600001B1C7E9720>43 D<60F0F0701010101020204080040C7C830C>II<60F0F0600404 7C830C>I<07C01830201C400C400EF00FF80FF807F8077007000F000E000E001C001C00 380070006000C00180030006010C01180110023FFE7FFEFFFE101C7E9B15>50 D<0FE03038401CE00EF00EF00EF00E000C001C0030006000C00080018001000100010001 00010001000000000000000000000003000780078003000F1D7E9C14>63 D66 D<001F808000E0618001801980070007 800E0003801C0003801C00018038000180780000807800008070000080F0000000F00000 00F0000000F0000000F0000000F0000000F0000000F00000007000008078000080780000 80380000801C0001001C0001000E000200070004000180080000E03000001FC000191E7E 9C1E>III73 D77 DI<003F800000E0E0000380380007001C000E000E001C000700 3C00078038000380780003C0780003C0700001C0F00001E0F00001E0F00001E0F00001E0 F00001E0F00001E0F00001E0F00001E0700001C0780003C0780003C0380003803C000780 1C0007000E000E0007001C000380380000E0E000003F80001B1E7E9C20>II<07E0801C1980300580700380600180E00180E00080E0 0080E00080F00000F800007C00007FC0003FF8001FFE0007FF0000FF80000F800007C000 03C00001C08001C08001C08001C0C00180C00180E00300D00200CC0C0083F800121E7E9C 17>83 D<7FFFFFC0700F01C0600F00C0400F0040400F0040C00F0020800F0020800F0020 800F0020000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000 000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000 001F800003FFFC001B1C7F9B1E>IIII<1FC000307000783800781C00301C00001C0000 1C0001FC000F1C00381C00701C00601C00E01C40E01C40E01C40603C40304E801F870012 127E9115>97 DI<07E00C30187830 7870306000E000E000E000E000E000E00060007004300418080C3007C00E127E9112>I< 003F0000070000070000070000070000070000070000070000070000070000070003E700 0C1700180F00300700700700600700E00700E00700E00700E00700E00700E00700600700 700700300700180F000C370007C7E0131D7E9C17>I<03E00C301818300C700E6006E006 FFFEE000E000E000E00060007002300218040C1803E00F127F9112>I<00F8018C071E06 1E0E0C0E000E000E000E000E000E00FFE00E000E000E000E000E000E000E000E000E000E 000E000E000E000E000E000E007FE00F1D809C0D>I<00038003C4C00C38C01C38801818 00381C00381C00381C00381C001818001C38000C300013C0001000003000001800001FF8 001FFF001FFF803003806001C0C000C0C000C0C000C06001803003001C0E0007F800121C 7F9215>II<18003C003C00180000 00000000000000000000000000FC001C001C001C001C001C001C001C001C001C001C001C 001C001C001C001C001C00FF80091D7F9C0C>I<00C001E001E000C00000000000000000 0000000000000FE000E000E000E000E000E000E000E000E000E000E000E000E000E000E0 00E000E000E000E000E000E060E0F0C0F1C061803E000B25839C0D>IIIII<03F0000E1C001806003003007003 80600180E001C0E001C0E001C0E001C0E001C0E001C06001807003803003001806000E1C 0003F00012127F9115>II114 D<1F9030704030C010C010E010F8007F803FE00FF000F880388018C018C018E010D0608F C00D127F9110>I<04000400040004000C000C001C003C00FFE01C001C001C001C001C00 1C001C001C001C001C101C101C101C101C100C100E2003C00C1A7F9910>IIII<7F8FF00F03800F030007020003840001C80001D8 0000F00000700000780000F800009C00010E00020E000607000403801E07C0FF0FF81512 809116>II<7FFC70386038407040F040E041C003C00380 07000F040E041C043C0C380870087038FFF80E127F9112>II E /Fo 16 118 df<78FCFCFCFC7800000000000078FCFCFCFC7806127D910D>58 D<001FE02000FFF8E003F80FE007C003E00F8001E01F0000E03E0000E03E0000607E0000 607C000060FC000000FC000000FC000000FC000000FC000000FC000000FC000000FC0000 007C0000607E0000603E0000603E0000C01F0000C00F80018007C0030003F80E0000FFFC 00001FE0001B1C7D9B22>67 DI77 D<07F8201FFEE03C07E07801E07000E0F000E0F00060F00060F80000FE0000FFE0007FFE 003FFF003FFF800FFFC007FFE0007FE00003F00001F00000F0C000F0C000F0C000E0E000 E0F001C0FC03C0EFFF0083FC00141C7D9B1B>83 D<0FF8001C1E003E0F803E07803E07C0 1C07C00007C0007FC007E7C01F07C03C07C07C07C0F807C0F807C0F807C0780BC03E13F8 0FE1F815127F9117>97 D<03FC000E0E001C1F003C1F00781F00780E00F80000F80000F8 0000F80000F80000F800007800007801803C01801C03000E0E0003F80011127E9115>99 D<01FC000F07001C03803C01C07801C07801E0F801E0F801E0FFFFE0F80000F80000F800 007800007C00603C00601E00C00F038001FC0013127F9116>101 D<03F8F00E0F381E0F381C07303C07803C07803C07803C07801C07001E0F000E0E001BF8 001000001800001800001FFF001FFFC00FFFE01FFFF07801F8F00078F00078F000787000 707800F01E03C007FF00151B7F9118>103 D<1E003F003F003F003F001E000000000000 00000000000000FF00FF001F001F001F001F001F001F001F001F001F001F001F001F001F 001F00FFE0FFE00B1E7F9D0E>105 D110 D<01FC000F07801C01C03C01E07800F07800F0F800F8F800F8F800F8F800 F8F800F8F800F87800F07800F03C01E01E03C00F078001FC0015127F9118>I114 D<1FD830786018E018E018F000FF 807FE07FF01FF807FC007CC01CC01CE01CE018F830CFC00E127E9113>I<030003000300 0300070007000F000F003FFCFFFC1F001F001F001F001F001F001F001F001F001F0C1F0C 1F0C1F0C0F08079803F00E1A7F9913>II E /Fp 35 122 df46 D<007F000001FFC00007 FFF0000FFFF8000FC1F8001F007C003F007E003E003E003C001E007C001F007C001F007C 001F0078000F00F8000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8 000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8000F8078000F007C 001F007C001F007C001F003E003E003E003E003F007E001F80FC000FC1F8000FFFF80007 FFF00001FFC000007F000019297EA71E>48 D<00180000380000F80007F800FFF800FFF8 00F8F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F8 0000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F8 0000F80000F80000F80000F80000F80000F80000F8007FFFF07FFFF07FFFF014287BA71E >I<00FE0003FFC007FFE00FFFF01F03F83C00FC38007E78003E70003EF0001FF0001F60 001F20001F00001F00001F00001F00003E00003E00007C00007C0000F80001F00001E000 03C0000780000F00001E00003C0000780000F00001E00003C0000780000F00001E00003C 00007FFFFF7FFFFF7FFFFF7FFFFF18287EA71E>I<007F000001FFC00007FFF0000FFFF8 001FC1F8003E007C003C003E0078003E0038003E0010003E0000003E0000003E0000003C 0000007C000000FC000001F8000007F00000FFE00000FFC00000FFE00000FFF0000001FC 0000007C0000003E0000001F0000001F0000000F8000000F8000000F8000000F8000000F 8040000F8060001F00F0001F00F8003F007E007E003F81FC001FFFF8000FFFF00003FFE0 00007F000019297EA71E>I<0003F0000007F0000005F000000DF000000DF000001DF000 0039F0000039F0000079F0000079F00000F1F00000F1F00001E1F00003E1F00003E1F000 07C1F00007C1F0000F81F0000F81F0001F01F0001F01F0003E01F0007C01F0007C01F000 F801F000FFFFFF80FFFFFF80FFFFFF80FFFFFF800001F0000001F0000001F0000001F000 0001F0000001F0000001F0000001F0000001F0000001F00019277EA61E>I<3FFFFC3FFF FC3FFFFC3FFFFC3E00003E00003E00003E00003E00003E00003E00003E00003E00003E00 003E3F003EFFC03FFFE03FFFF03FE1F83F807C3F003E3E003E00003E00001F00001F0000 1F00001F00001F00001F00001F20001F60003E70003EF8007C7C00FC3F03F81FFFF00FFF E007FF8000FE0018287EA61E>I<0001FF00000FFFE0003FFFF8007FFFF800FE01F801F8 003003F0001007C000000F8000001F8000001F0000003E0000003E0000007E0000007C00 00007C0000007C000000F8000000F8000000F8000000F8000000F8000000F8000000F800 0000F8000000F8000000F80000007C0000007C0000007C0000007E0000003E0000003E00 00001F0000001F8000000F80000007C0000003F0000401F8001C00FE00FC007FFFFC003F FFF8000FFFE00001FF001E2C7CAA26>67 DII73 D76 DII80 D<007FC00001FFF80007FFFE 000FFFFF001FC07F003F000F007E0006007C0000007C000000F8000000F8000000F80000 00F8000000F8000000FC0000007E0000007F0000003F8000001FF800000FFF800007FFE0 0003FFF80000FFFC00000FFE000000FF0000003F0000001F8000000F8000000FC0000007 C0000007C0000007C0000007C0000007C0000007C000000F8060000F80F0001F00FC003F 00FF80FE007FFFFC001FFFF80007FFE00000FF80001A2C7DAA21>83 D<01FE000FFF803FFFC03FFFE03C03F03001F00001F80000F80000F80000F80000F80000 F8007FF807FFF81FFFF83FE0F87F00F8FC00F8F800F8F800F8F800F8FC01F87E07F87FFF F83FFFF81FFCF80FE0F8151B7E9A1D>97 DI<007FC001FFF007FFFC0FFFFC1FC07C1F00083E00007C00007C00007C00 00F80000F80000F80000F80000F80000F80000F800007C00007C00007E00003E00001F00 0C1FC07C0FFFFC07FFFC01FFF0007F80161B7E9A1B>I<00003E00003E00003E00003E00 003E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E00FC3E03 FF3E07FFFE0FFFFE1FC1FE3F007E3E003E7C003E7C003EFC003EF8003EF8003EF8003EF8 003EF8003EF8003EF8003EFC003E7C003E7C003E3E007E3F00FE1FC1FE0FFFFE07FFBE03 FF3E00FC3E172A7EA91F>I<007E0003FF8007FFC00FFFE01F83F03F00F03E00787C0078 7C003878003CFFFFFCFFFFFCFFFFFCFFFFFCF80000F80000F800007800007C00007C0000 3E00003F000C1FC07C0FFFFC07FFFC01FFF0007F80161B7E9A1B>I<001FC0007FC000FF C001FFC003F00003E00007C00007C00007C00007C00007C00007C00007C00007C00007C0 00FFFE00FFFE00FFFE0007C00007C00007C00007C00007C00007C00007C00007C00007C0 0007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C0 0007C00007C00007C000122A7FA912>I<00F8078003FE7FC00FFFFFC01FFFFFC01F07C0 003E03E0003E03E0007C01F0007C01F0007C01F0007C01F0007C01F0007C01F0003E03E0 003E03E0001F07C0001FFFC0003FFF80003BFE000038F8000078000000780000003C0000 003FFFC0003FFFF8001FFFFC001FFFFE003FFFFF007C007F00F8001F80F8000F80F8000F 80F8000F80FC001F807E003F003F80FE003FFFFE000FFFF80007FFF00000FF80001A287E 9A1E>I105 D108 DII<007F000001FFC00007FFF0000FFFF8001FC1FC003F 007E003E003E007C001F007C001F0078000F00F8000F80F8000F80F8000F80F8000F80F8 000F80F8000F80F8000F807C001F007C001F007E003F003E003E003F007E001FC1FC000F FFF80007FFF00001FFC000007F0000191B7E9A1E>II114 D<03FC001FFF803FFFC07FFFC07C07C0F80080F80000F80000F80000FC00007F80007FF8 003FFE001FFF0007FF8000FFC0000FE00007E00003E00003E04003E0E007E0FC0FC0FFFF C07FFF801FFE0003F800131B7E9A17>I<07C00007C00007C00007C00007C00007C00007 C000FFFFC0FFFFC0FFFFC007C00007C00007C00007C00007C00007C00007C00007C00007 C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C04007E1C003 FFE003FFE001FF8000FC0013227FA116>II119 D121 D E /Fq 9 122 df77 D<0007FFFC000000007FFFFFC0000001FFFFFFF8000003FFFFFFFE000007FE001FFF0000 07FF0003FFC0000FFF8001FFE0000FFF8000FFF0000FFF80007FF0000FFF80007FF8000F FF80007FF80007FF00003FFC0007FF00003FFC0003FE00003FFC0000F800003FFC000000 00003FFC00000000003FFC00000000003FFC00000000003FFC00000007FFFFFC000000FF FFFFFC000007FFFFFFFC00003FFFE03FFC0000FFFE003FFC0003FFF0003FFC0007FFC000 3FFC000FFF00003FFC001FFE00003FFC003FFC00003FFC007FF800003FFC007FF800003F FC00FFF000003FFC00FFF000003FFC00FFF000003FFC00FFF000003FFC00FFF000003FFC 00FFF000007FFC007FF80000FFFC007FF80001EFFC003FFC0003EFFC003FFF0007CFFF00 0FFFC03F8FFFF807FFFFFF07FFFC01FFFFFC03FFFC007FFFF001FFFC0003FF80007FF836 2E7DAD3A>97 D<00001FFFC0000000FFFFF8000007FFFFFE00001FFFFFFF80007FFC00FF C000FFE001FFC001FFC003FFE003FF8003FFE007FF0003FFE00FFE0003FFE00FFE0003FF E01FFC0001FFC01FFC0001FFC03FFC0000FF803FFC00003E007FF8000000007FF8000000 007FF800000000FFF800000000FFF800000000FFF800000000FFF800000000FFF8000000 00FFF800000000FFF800000000FFF800000000FFF800000000FFF8000000007FF8000000 007FF8000000007FFC000000003FFC000000003FFC000000001FFC000000F81FFE000000 F80FFE000000F80FFF000001F007FF800003F003FFC00007E001FFE0000FC000FFF0001F 80007FFE00FF00001FFFFFFE000007FFFFF8000000FFFFE00000001FFE00002D2E7CAD35 >99 D<00001FFE00000001FFFFE0000007FFFFF800001FFFFFFE00007FFC07FF0000FFE0 01FF8001FFC0007FC003FF80003FE007FF00003FF00FFE00001FF01FFE00000FF81FFC00 000FF83FFC00000FFC3FFC000007FC7FFC000007FC7FF8000007FC7FF8000007FE7FF800 0007FEFFF8000007FEFFF8000007FEFFFFFFFFFFFEFFFFFFFFFFFEFFFFFFFFFFFEFFFFFF FFFFFCFFF800000000FFF800000000FFF800000000FFF8000000007FF8000000007FF800 0000007FFC000000003FFC000000003FFC000000003FFC0000001C1FFE0000003E0FFE00 00003E07FF0000007E07FF000000FC03FF800001F801FFC00003F0007FF0001FE0003FFE 00FFC0001FFFFFFF800007FFFFFE000000FFFFF80000000FFF80002F2E7DAD36>101 D<00FC0001FE0003FF0007FF800FFFC01FFFE01FFFE01FFFE01FFFE01FFFE01FFFE00FFF C007FF8003FF0001FE0000FC000000000000000000000000000000000000000000000000 00000000000000000000007FC0FFFFC0FFFFC0FFFFC0FFFFC0FFFFC003FFC001FFC001FF C001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FF C001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FF C001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC0FFFFFFFFFFFFFFFFFFFFFF FFFFFFFF18497CC820>105 D<007FC000FFFFC000FFFFC000FFFFC000FFFFC000FFFFC0 0003FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC000FFFFFF80FFFFFF 80FFFFFF80FFFFFF80FFFFFF8019487CC720>108 D<007FC001FFC00000FFFFC00FFFF8 0000FFFFC03FFFFE0000FFFFC0FFFFFF0000FFFFC1FC07FF8000FFFFC3E003FFC00003FF C7C001FFC00001FFCF0001FFE00001FFDE0000FFE00001FFDC0000FFE00001FFFC0000FF F00001FFF80000FFF00001FFF00000FFF00001FFF00000FFF00001FFF00000FFF00001FF E00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FF F00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FF E00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FF F00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FF E00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FF F00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF000FFFFFFC07FFFFFE0FFFF FFC07FFFFFE0FFFFFFC07FFFFFE0FFFFFFC07FFFFFE0FFFFFFC07FFFFFE03B2E7CAD42> 110 D<000FFF00E0007FFFF3E001FFFFFFE007FFFFFFE00FF800FFE01FC0001FE03F8000 0FE03F000007E07F000003E07F000003E0FF000003E0FF000003E0FF800003E0FFC00000 00FFF0000000FFFE000000FFFFF800007FFFFFC0007FFFFFF0003FFFFFFC001FFFFFFF00 0FFFFFFF8007FFFFFFC003FFFFFFE000FFFFFFF0003FFFFFF00003FFFFF800001FFFF800 0000FFFC0000001FFC7800000FFCF8000007FCF8000003FCFC000003FCFC000003FCFE00 0003F8FE000003F8FF000003F8FF800007F0FFC0000FF0FFF0001FE0FFFC00FFC0FFFFFF FF80FC7FFFFE00F81FFFF800E003FF8000262E7CAD2F>115 D<7FFFFFC000FFFF807FFF FFC000FFFF807FFFFFC000FFFF807FFFFFC000FFFF807FFFFFC000FFFF8000FFF000000F E00000FFF800000FC00000FFF800000FC000007FFC00000F8000007FFC00001F8000003F FC00001F0000003FFE00003F0000001FFE00003E0000001FFF00007E0000000FFF00007C 0000000FFF8000FC00000007FF8000F800000007FFC001F800000003FFC001F000000003 FFE003F000000003FFE003F000000001FFF003E000000001FFF007E000000000FFF007C0 00000000FFF80FC0000000007FF80F80000000007FFC1F80000000003FFC1F0000000000 3FFE3F00000000001FFE3E00000000001FFF7E00000000000FFF7C00000000000FFFFC00 000000000FFFFC000000000007FFF8000000000007FFF8000000000003FFF00000000000 03FFF0000000000001FFE0000000000001FFE0000000000000FFC0000000000000FFC000 00000000007F800000000000007F800000000000003F000000000000003F000000000000 003F000000000000003E000000000000007E000000000000007C00000000000000FC0000 00001F8000F8000000003FC001F8000000007FE001F000000000FFF003F000000000FFF0 03E000000000FFF007E000000000FFF00FC000000000FFF01F8000000000FFF03F800000 00007FE07F00000000007F43FE00000000003FFFF800000000001FFFF0000000000007FF C0000000000001FE00000000000039427EAD3F>121 D E /Fr 9 117 df<0001FF8000000FFFF000003FFFFC0000FF81FF0001FE007F8003FC003FC007F8 001FE00FF8001FF00FF0000FF01FF0000FF81FF0000FF83FF0000FFC3FE00007FC3FE000 07FC7FE00007FE7FE00007FE7FE00007FE7FE00007FE7FE00007FEFFE00007FFFFE00007 FFFFE00007FFFFE00007FFFFE00007FFFFE00007FFFFE00007FFFFE00007FFFFE00007FF FFE00007FFFFE00007FFFFE00007FFFFE00007FFFFE00007FFFFE00007FFFFE00007FFFF E00007FFFFE00007FFFFE00007FF7FE00007FE7FE00007FE7FE00007FE7FE00007FE7FE0 0007FE3FE00007FC3FF0000FFC3FF0000FFC1FF0000FF81FF0000FF80FF0000FF00FF800 1FF007F8001FE003FC003FC001FE007F8000FF81FF00003FFFFC00000FFFF0000001FF80 0028397CB731>48 D<00001E000000003E00000000FE00000003FE0000003FFE0000FFFF FE0000FFFFFE0000FFFFFE0000FFCFFE0000000FFE0000000FFE0000000FFE0000000FFE 0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE00 00000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000 000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE000000 0FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000F FE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE 0000000FFE0000000FFE0000000FFE00007FFFFFFFC07FFFFFFFC07FFFFFFFC07FFFFFFF C0223879B731>I<0000001FFF000030000001FFFFE000F000000FFFFFFC01F000007FFF FFFE03F00001FFFE007F87F00003FFE0000FCFF0000FFF000003FFF0001FFC000001FFF0 003FF80000007FF0007FF00000003FF000FFC00000003FF001FFC00000001FF003FF8000 00000FF007FF000000000FF00FFF0000000007F00FFE0000000007F01FFE0000000003F0 1FFE0000000003F03FFC0000000003F03FFC0000000001F03FFC0000000001F07FFC0000 000001F07FF80000000001F07FF80000000000007FF8000000000000FFF8000000000000 FFF8000000000000FFF8000000000000FFF8000000000000FFF8000000000000FFF80000 00000000FFF8000000000000FFF8000000000000FFF8000000000000FFF8000000000000 FFF80000000000007FF80000000000007FF80000000000007FF80000000000007FFC0000 000000F03FFC0000000000F03FFC0000000000F03FFC0000000000F01FFE0000000000F0 1FFE0000000001E00FFE0000000001E00FFF0000000001E007FF0000000003C003FF8000 000003C001FFC0000000078000FFE00000000F00007FF00000001F00003FF80000003E00 001FFC0000007C00000FFF000001F8000003FFE00007F0000001FFFE003FC00000007FFF FFFF000000000FFFFFFC0000000001FFFFF000000000001FFF0000003C3D7BBB47>67 D<001FFF00000001FFFFF0000003FFFFFC000007F007FE00000FF801FF00001FFC00FF80 001FFC007FC0001FFC007FE0001FFC003FE0000FF8003FF0000FF8003FF00007F0003FF0 0001C0003FF0000000003FF0000000003FF0000000003FF0000000FFFFF000000FFFFFF0 00007FF83FF00001FF803FF00007FE003FF0000FF8003FF0001FF0003FF0003FE0003FF0 007FE0003FF0007FE0003FF000FFC0003FF000FFC0003FF000FFC0003FF000FFC0003FF0 00FFC0007FF0007FE0007FF0007FE000DFF0003FF0039FF8001FFC0F0FFFF007FFFE0FFF F001FFFC07FFF0003FE000FFF02C267DA530>97 D<0001FFC000000FFFF800003FFFFE00 00FF80FF0001FE003F8007FC001FC00FF8000FE00FF8000FF01FF00007F03FF00007F83F F00007F87FE00007F87FE00003FC7FE00003FC7FE00003FCFFE00003FCFFFFFFFFFCFFFF FFFFFCFFFFFFFFFCFFE0000000FFE0000000FFE0000000FFE00000007FE00000007FE000 00007FE00000003FE00000003FF000003C1FF000003C1FF000003C0FF800007807FC0000 F803FE0001F001FF0007E000FFC03FC0003FFFFF000007FFFC000000FFE00026267DA52D >101 D<00FF00000000FFFF00000000FFFF00000000FFFF00000000FFFF0000000007FF 0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF 0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF 0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF007FC00003FF 01FFF80003FF07FFFC0003FF0F03FE0003FF1C01FF0003FF3001FF8003FF6000FF8003FF E000FFC003FFC000FFC003FF8000FFC003FF8000FFC003FF8000FFC003FF0000FFC003FF 0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF 0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF 0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF 0000FFC003FF0000FFC003FF0000FFC0FFFFFC3FFFFFFFFFFC3FFFFFFFFFFC3FFFFFFFFF FC3FFFFF303C7CBB37>104 D<00FF01FF8000FFFF0FFFF000FFFF3FFFFC00FFFFFE03FF 00FFFFF000FF8003FFC0007FC003FF80003FE003FF00003FF003FF00001FF803FF00001F FC03FF00000FFC03FF00000FFE03FF00000FFE03FF000007FE03FF000007FF03FF000007 FF03FF000007FF03FF000007FF03FF000007FF03FF000007FF03FF000007FF03FF000007 FF03FF000007FF03FF000007FE03FF000007FE03FF00000FFE03FF00000FFC03FF00000F FC03FF00001FF803FF00001FF803FF00003FF003FF80003FE003FFC0007FC003FFF001FF 8003FFFC07FF0003FF3FFFFC0003FF0FFFF00003FF01FF000003FF0000000003FF000000 0003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF000000 0003FF0000000003FF0000000003FF0000000003FF0000000003FF00000000FFFFFC0000 00FFFFFC000000FFFFFC000000FFFFFC00000030377DA537>112 D<00FE03F000FFFE0FFE00FFFE1FFF00FFFE3C3F80FFFE707FC007FE60FFE003FEE0FFE0 03FEC0FFE003FFC0FFE003FF807FC003FF807FC003FF803F8003FF800E0003FF00000003 FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF 00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00 000003FF00000003FF00000003FF00000003FF00000003FF000000FFFFFE0000FFFFFE00 00FFFFFE0000FFFFFE000023267DA529>114 D<00078000000780000007800000078000 00078000000F8000000F8000000F8000000F8000001F8000001F8000003F8000003F8000 007F800000FF800001FF800007FF80001FFFFFF0FFFFFFF0FFFFFFF0FFFFFFF001FF8000 01FF800001FF800001FF800001FF800001FF800001FF800001FF800001FF800001FF8000 01FF800001FF800001FF800001FF800001FF800001FF800001FF800001FF800001FF8000 01FF803C01FF803C01FF803C01FF803C01FF803C01FF803C01FF803C01FF803C00FF8078 00FFC078007FC070003FE0E0001FFFC00007FF800001FF001E377EB626>116 D E /Fs 73 123 df<001F83E000F06E3001C078780380F8780300F03007007000070070 000700700007007000070070000700700007007000FFFFFF800700700007007000070070 000700700007007000070070000700700007007000070070000700700007007000070070 000700700007007000070070000700700007007000070070007FE3FF001D20809F1B>11 D<003F0000E0C001C0C00381E00701E00701E00700000700000700000700000700000700 00FFFFE00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700 E00700E00700E00700E00700E00700E00700E00700E07FC3FE1720809F19>I<003FE000 E0E001C1E00381E00700E00700E00700E00700E00700E00700E00700E00700E0FFFFE007 00E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E007 00E00700E00700E00700E00700E00700E07FE7FE1720809F19>I<001F81F80000F04F04 0001C07C06000380F80F000300F00F000700F00F00070070000007007000000700700000 070070000007007000000700700000FFFFFFFF0007007007000700700700070070070007 007007000700700700070070070007007007000700700700070070070007007007000700 700700070070070007007007000700700700070070070007007007000700700700070070 07007FE3FE3FF02420809F26>I<7038F87CFC7EFC7E743A040204020402080408041008 1008201040200F0E7E9F17>34 D<0020004000800100020006000C000C00180018003000 300030007000600060006000E000E000E000E000E000E000E000E000E000E000E000E000 6000600060007000300030003000180018000C000C000600020001000080004000200B2E 7DA112>40 D<800040002000100008000C00060006000300030001800180018001C000C0 00C000C000E000E000E000E000E000E000E000E000E000E000E000E000C000C000C001C0 01800180018003000300060006000C00080010002000400080000B2E7DA112>I<000600 000006000000060000000600000006000000060000000600000006000000060000000600 000006000000060000000600000006000000060000FFFFFFF0FFFFFFF000060000000600 000006000000060000000600000006000000060000000600000006000000060000000600 00000600000006000000060000000600001C207D9A23>43 D<70F8FCFC74040404080810 102040060E7C840D>II<70F8F8F87005057C840D>I<00010003 0003000600060006000C000C000C00180018001800300030003000600060006000C000C0 00C00180018001800300030003000600060006000C000C000C0018001800180030003000 3000600060006000C000C000C000102D7DA117>I<03F0000E1C001C0E00180600380700 700380700380700380700380F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0 F003C0F003C0F003C0F003C0F003C07003807003807003807807803807001806001C0E00 0E1C0003F000121F7E9D17>I<018003800F80F380038003800380038003800380038003 80038003800380038003800380038003800380038003800380038003800380038007C0FF FE0F1E7C9D17>I<03F0000C1C00100E00200700400780800780F007C0F803C0F803C0F8 03C02007C00007C0000780000780000F00000E00001C0000380000700000600000C00001 80000300000600400C00401800401000803FFF807FFF80FFFF80121E7E9D17>I<03F000 0C1C00100E00200F00780F80780780780780380F80000F80000F00000F00000E00001C00 00380003F000003C00000E00000F000007800007800007C02007C0F807C0F807C0F807C0 F00780400780400F00200E001C3C0003F000121F7E9D17>I<000600000600000E00000E 00001E00002E00002E00004E00008E00008E00010E00020E00020E00040E00080E00080E 00100E00200E00200E00400E00C00E00FFFFF0000E00000E00000E00000E00000E00000E 00000E0000FFE0141E7F9D17>I<1803001FFE001FFC001FF8001FE00010000010000010 000010000010000010000011F000161C00180E001007001007800003800003800003C000 03C00003C07003C0F003C0F003C0E00380400380400700200600100E000C380003E00012 1F7E9D17>I<007C000182000701000E03800C07801C0780380300380000780000700000 700000F1F000F21C00F40600F80700F80380F80380F003C0F003C0F003C0F003C0F003C0 7003C07003C07003803803803807001807000C0E00061C0001F000121F7E9D17>I<4000 007FFFC07FFF807FFF804001008002008002008004000008000008000010000020000020 0000400000400000C00000C00001C0000180000380000380000380000380000780000780 00078000078000078000078000078000030000121F7D9D17>I<03F0000C0C0010060030 03002001806001806001806001807001807803003E03003F06001FC8000FF00003F80007 FC000C7E00103F00300F806003804001C0C001C0C000C0C000C0C000C0C0008060018020 01001002000C0C0003F000121F7E9D17>I<03F0000E18001C0C00380600380700700700 700380F00380F00380F003C0F003C0F003C0F003C0F003C07007C07007C03807C0180BC0 0E13C003E3C0000380000380000380000700300700780600780E00700C00201800107000 0FC000121F7E9D17>I<70F8F8F8700000000000000000000070F8F8F87005147C930D>I< 70F8F8F8700000000000000000000070F0F8F878080808101010202040051D7C930D>I< 7FFFFFE0FFFFFFF000000000000000000000000000000000000000000000000000000000 00000000FFFFFFF07FFFFFE01C0C7D9023>61 D<0FC0307040384038E03CF03CF03C603C 0038007000E000C001800180010003000200020002000200020002000000000000000000 000007000F800F800F8007000E207D9F15>63 D<00010000000380000003800000038000 0007C0000007C0000007C0000009E0000009E0000009E0000010F0000010F0000010F000 00207800002078000020780000403C0000403C0000403C0000801E0000801E0000FFFE00 01000F0001000F0001000F00020007800200078002000780040003C00E0003C01F0007E0 FFC03FFE1F207F9F22>65 DI<00 0FC040007030C001C009C0038005C0070003C00E0001C01E0000C01C0000C03C0000C07C 0000407C00004078000040F8000000F8000000F8000000F8000000F8000000F8000000F8 000000F8000000F8000000780000007C0000407C0000403C0000401C0000401E0000800E 000080070001000380020001C0040000703800000FC0001A217D9F21>IIII<000FE0200078186000 E004E0038002E0070001E00F0000E01E0000601E0000603C0000603C0000207C00002078 000020F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8007FFCF8 0003E0780001E07C0001E03C0001E03C0001E01E0001E01E0001E00F0001E0070001E003 8002E000E0046000781820000FE0001E217D9F24>III76 DII<001F800000F0F00001C0380007801E000F000F000E0007001E0007803C00 03C03C0003C07C0003E0780001E0780001E0F80001F0F80001F0F80001F0F80001F0F800 01F0F80001F0F80001F0F80001F0F80001F0780001E07C0003E07C0003E03C0003C03C00 03C01E0007800E0007000F000F0007801E0001C0380000F0F000001F80001C217D9F23> II82 D<07E0800C1980100780300380600180600180E001 80E00080E00080E00080F00000F000007800007F00003FF0001FFC000FFE0003FF00001F 800007800003C00003C00001C08001C08001C08001C08001C0C00180C00380E00300F006 00CE0C0081F80012217D9F19>I<7FFFFFE0780F01E0600F0060400F0020400F0020C00F 0030800F0010800F0010800F0010800F0010000F0000000F0000000F0000000F0000000F 0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F 0000000F0000000F0000000F0000000F0000000F0000001F800007FFFE001C1F7E9E21> II87 D<7FF83FF80FE00FC007C0070003C0020001E0040001F00C00 00F0080000781000007C1000003C2000003E4000001E4000000F8000000F800000078000 0003C0000007E0000005E0000009F0000018F8000010780000207C0000603C0000401E00 00801F0001800F0001000780020007C0070003C01F8007E0FFE01FFE1F1F7F9E22>I<08 0410082010201040204020804080408040B85CFC7EFC7E7C3E381C0F0E7B9F17>92 D<1FE000303000781800781C00300E00000E00000E00000E0000FE00078E001E0E00380E 00780E00F00E10F00E10F00E10F01E10781E103867200F83C014147E9317>97 D<0E0000FE00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00 000E3E000EC3800F01C00F00E00E00E00E00700E00700E00780E00780E00780E00780E00 780E00780E00700E00700E00E00F00E00D01C00CC300083E0015207F9F19>I<03F80E0C 1C1E381E380C70007000F000F000F000F000F000F00070007000380138011C020E0C03F0 10147E9314>I<000380003F800003800003800003800003800003800003800003800003 8000038000038003E380061B801C0780380380380380700380700380F00380F00380F003 80F00380F00380F003807003807003803803803807801C07800E1B8003E3F815207E9F19 >I<03F0000E1C001C0E00380700380700700700700380F00380F00380FFFF80F00000F0 0000F000007000007000003800801800800C010007060001F80011147F9314>I<007C00 C6018F038F07060700070007000700070007000700FFF007000700070007000700070007 00070007000700070007000700070007000700070007007FF01020809F0E>I<0000E003 E3300E3C301C1C30380E00780F00780F00780F00780F00780F00380E001C1C001E380033 E0002000002000003000003000003FFE001FFF800FFFC03001E0600070C00030C00030C0 0030C000306000603000C01C038003FC00141F7F9417>I<0E0000FE00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E3E000E43000E81800F01C0 0F01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0 0E01C00E01C00E01C0FFE7FC16207F9F19>I<1C001E003E001E001C0000000000000000 00000000000E007E000E000E000E000E000E000E000E000E000E000E000E000E000E000E 000E000E000E00FFC00A1F809E0C>I<00E001F001F001F000E000000000000000000000 0000007007F000F000700070007000700070007000700070007000700070007000700070 00700070007000700070007000706070F060F0C061803F000C28829E0E>I<0E0000FE00 000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E0FF00E03 C00E03000E02000E04000E08000E10000E30000E70000EF8000F38000E1C000E1E000E0E 000E07000E07800E03800E03C00E03E0FFCFF815207F9F18>I<0E00FE000E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00 0E000E000E000E000E000E000E000E00FFE00B20809F0C>I<0E1F01F000FE618618000E 81C81C000F00F00E000F00F00E000E00E00E000E00E00E000E00E00E000E00E00E000E00 E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E0 0E000E00E00E000E00E00E00FFE7FE7FE023147F9326>I<0E3E00FE43000E81800F01C0 0F01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0 0E01C00E01C00E01C0FFE7FC16147F9319>I<01F800070E001C03803801C03801C07000 E07000E0F000F0F000F0F000F0F000F0F000F0F000F07000E07000E03801C03801C01C03 80070E0001F80014147F9317>I<0E3E00FEC3800F01C00F00E00E00E00E00F00E00700E 00780E00780E00780E00780E00780E00780E00700E00F00E00E00F01E00F01C00EC3000E 3E000E00000E00000E00000E00000E00000E00000E00000E0000FFE000151D7F9319>I< 03E0800619801C05803C0780380380780380700380F00380F00380F00380F00380F00380 F003807003807803803803803807801C0B800E138003E380000380000380000380000380 000380000380000380000380003FF8151D7E9318>I<0E78FE8C0F1E0F1E0F0C0E000E00 0E000E000E000E000E000E000E000E000E000E000E000E00FFE00F147F9312>I<1F9030 704030C010C010C010E00078007F803FE00FF00070803880188018C018C018E030D0608F 800D147E9312>I<020002000200060006000E000E003E00FFF80E000E000E000E000E00 0E000E000E000E000E000E000E080E080E080E080E080610031001E00D1C7F9B12>I<0E 01C0FE1FC00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E03C00603C0030DC001F1FC16147F9319>III<7FC3FC0F01E00701C007018003810001C20000E40000EC00007800003800003C 00007C00004E000087000107000303800201C00601E01E01E0FF07FE1714809318>II<3FFF380E200E201C40384078407000 E001E001C00380078007010E011E011C0338027006700EFFFE10147F9314>I E /Ft 46 122 df<00200040008001000300060004000C000C0018001800300030003000 7000600060006000E000E000E000E000E000E000E000E000E000E000E000E000E000E000 6000600060007000300030003000180018000C000C000400060003000100008000400020 0B327CA413>40 D<800040002000100018000C0004000600060003000300018001800180 01C000C000C000C000E000E000E000E000E000E000E000E000E000E000E000E000E000E0 00C000C000C001C0018001800180030003000600060004000C0018001000200040008000 0B327DA413>I<70F8FCFC7404040404080810102040060F7C840E>44 DI<70F8F8F87005057C840E>I<008003800F80F3800380038003 800380038003800380038003800380038003800380038003800380038003800380038003 8003800380038003800380038007C0FFFE0F217CA018>49 D<03F0000C1C001007002007 804003C04003C08003E0F003E0F801E0F801E0F801E02003E00003E00003C00003C00007 80000700000E00001C0000180000300000600000C0000180000100000200200400200800 201800603000403FFFC07FFFC0FFFFC013217EA018>I<000200000600000E00000E0000 1E00001E00002E00004E00004E00008E00008E00010E00020E00020E00040E00040E0008 0E00100E00100E00200E00200E00400E00800E00FFFFF8000E00000E00000E00000E0000 0E00000E00000E00001F0001FFF015217FA018>52 D<1000801E07001FFF001FFE001FF8 0013E00010000010000010000010000010000010000010F800130E001407001803801003 800001C00001C00001E00001E00001E00001E07001E0F001E0F001E0E001C08001C04003 C04003802007001006000C1C0003F00013227EA018>I<007E0001C1000300800601C00E 03C01C03C0180180380000380000780000700000700000F0F800F30C00F40600F40300F8 0380F801C0F001C0F001E0F001E0F001E0F001E0F001E07001E07001E07001E03801C038 01C01803801C03000C0600070C0001F00013227EA018>I<01F800060E00080300100180 2001802000C06000C06000C06000C07000C07801803E01003F02001FC4000FF80003F800 03FC00067F00083F80100F803007C06001C06000E0C000E0C00060C00060C00060C00060 6000406000C03000801803000E0E0003F00013227EA018>56 D<01F000060C000C060018 0700380380700380700380F001C0F001C0F001C0F001E0F001E0F001E0F001E0F001E070 01E07003E03803E01805E00C05E00619E003E1E00001C00001C00001C000038000038030 0300780700780600700C002018001030000FC00013227EA018>I<000180000001800000 0180000003C0000003C0000003C0000005E0000005E000000DF0000008F0000008F00000 10F800001078000010780000203C0000203C0000203C0000401E0000401E0000401E0000 800F0000800F0000FFFF000100078001000780030007C0020003C0020003C0040003E004 0001E0040001E00C0000F00C0000F03E0001F8FF800FFF20237EA225>65 D<0007E0100038183000E0063001C00170038000F0070000F00E0000701E0000701C0000 303C0000303C0000307C0000107800001078000010F8000000F8000000F8000000F80000 00F8000000F8000000F8000000F800000078000000780000107C0000103C0000103C0000 101C0000201E0000200E000040070000400380008001C0010000E0020000381C000007E0 001C247DA223>67 DIII72 DI77 DI80 D82 D<03F0200C0C601802603001E07000E0600060E00060E00060E00020 E00020E00020F00000F000007800007F00003FF0001FFE000FFF0003FF80003FC00007E0 0001E00000F00000F0000070800070800070800070800070C00060C00060E000C0F000C0 C80180C6070081FC0014247DA21B>I<7FFFFFF878078078600780184007800840078008 40078008C007800C80078004800780048007800480078004000780000007800000078000 000780000007800000078000000780000007800000078000000780000007800000078000 000780000007800000078000000780000007800000078000000780000007800000078000 000FC00003FFFF001E227EA123>I<0FE0001838003C0C003C0E00180700000700000700 00070000FF0007C7001E07003C0700780700700700F00708F00708F00708F00F08781708 3C23900FC1E015157E9418>97 D<0E0000FE00001E00000E00000E00000E00000E00000E 00000E00000E00000E00000E00000E00000E00000E1F000E61C00E80600F00300E00380E 003C0E001C0E001E0E001E0E001E0E001E0E001E0E001E0E001E0E001C0E003C0E00380F 00700C80600C41C0083F0017237FA21B>I<01FE000703000C07801C0780380300780000 700000F00000F00000F00000F00000F00000F00000F000007000007800403800401C0080 0C010007060001F80012157E9416>I<0000E0000FE00001E00000E00000E00000E00000 E00000E00000E00000E00000E00000E00000E00000E001F8E00704E00C02E01C01E03800 E07800E07000E0F000E0F000E0F000E0F000E0F000E0F000E0F000E07000E07800E03800 E01801E00C02E0070CF001F0FE17237EA21B>I<01FC000707000C03801C01C03801C078 01E07000E0F000E0FFFFE0F00000F00000F00000F00000F000007000007800203800201C 00400E008007030000FC0013157F9416>I<003C00C6018F038F030F0700070007000700 07000700070007000700FFF8070007000700070007000700070007000700070007000700 07000700070007000700070007807FF8102380A20F>I<00007001F198071E180E0E181C 07001C07003C07803C07803C07803C07801C07001C07000E0E000F1C0019F00010000010 00001800001800001FFE000FFFC00FFFE03800F0600030400018C00018C00018C0001860 00306000303800E00E038003FE0015217F9518>I<0E0000FE00001E00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E00000E1F800E60C00E80E0 0F00700F00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E0070 0E00700E00700E00700E00700E0070FFE7FF18237FA21B>I<1C001E003E001E001C0000 0000000000000000000000000000000E00FE001E000E000E000E000E000E000E000E000E 000E000E000E000E000E000E000E000E000E00FFC00A227FA10E>I<01C003E003E003E0 01C00000000000000000000000000000000001E00FE001E000E000E000E000E000E000E0 00E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E060E0 F0C0F18061803E000B2C82A10F>I<0E0000FE00001E00000E00000E00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E03FC0E01F00E01C00E01800E0200 0E04000E08000E10000E38000EF8000F1C000E1E000E0E000E07000E07800E03C00E01C0 0E01E00E00F00E00F8FFE3FE17237FA21A>I<0E1FC07F00FE60E183801E807201C00F00 3C00E00F003C00E00E003800E00E003800E00E003800E00E003800E00E003800E00E0038 00E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800 E00E003800E00E003800E0FFE3FF8FFE27157F942A>109 D<0E1F80FE60C01E80E00F00 700F00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00 700E00700E00700E00700E0070FFE7FF18157F941B>I<01FC000707000C01801800C038 00E0700070700070F00078F00078F00078F00078F00078F00078F000787000707800F038 00E01C01C00E038007070001FC0015157F9418>I<0E1F00FE61C00E80600F00700E0038 0E003C0E001C0E001E0E001E0E001E0E001E0E001E0E001E0E001E0E003C0E003C0E0038 0F00700E80E00E41C00E3F000E00000E00000E00000E00000E00000E00000E00000E0000 0E0000FFE000171F7F941B>I<0E3CFE461E8F0F0F0F060F000E000E000E000E000E000E 000E000E000E000E000E000E000E000F00FFF010157F9413>114 D<0F8830786018C018C008C008E008F0007F803FE00FF001F8003C801C800C800CC00CC0 08E018D0308FC00E157E9413>I<02000200020002000600060006000E001E003E00FFF8 0E000E000E000E000E000E000E000E000E000E000E000E040E040E040E040E040E040708 030801F00E1F7F9E13>I<0E0070FE07F01E00F00E00700E00700E00700E00700E00700E 00700E00700E00700E00700E00700E00700E00700E00700E00F00E00F006017003827800 FC7F18157F941B>I119 D121 D E /Fu 20 118 df45 D68 D73 D77 D80 D<007F802001FFE02007C078600F001C601E0006E03C0003E0380001E078 0000E0700000E070000060F0000060F0000060F0000020F0000020F0000020F8000020F8 0000007C0000007E0000003F0000003FC000001FF800000FFF800007FFF80003FFFC0000 FFFF00000FFF800000FFC000001FE0000007E0000003F0000001F0000000F0000000F800 0000F88000007880000078800000788000007880000078C0000078C0000070E00000F0E0 0000E0F00000E0F80001C0EC000380C7000700C1F01E00807FFC00800FF0001D337CB125 >83 D<00FE00000303C0000C00E00010007000100038003C003C003E001C003E001E003E 001E0008001E0000001E0000001E0000001E00000FFE0000FC1E0003E01E000F801E001F 001E003E001E003C001E007C001E00F8001E04F8001E04F8001E04F8003E04F8003E0478 003E047C005E043E008F080F0307F003FC03E01E1F7D9E21>97 D<003F8000E060038018 0700040F00041E001E1C003E3C003E7C003E7C0008780000F80000F80000F80000F80000 F80000F80000F80000F80000F800007800007C00007C00003C00011E00011E00020F0002 07000403801800E060003F80181F7D9E1D>99 D<000001E000003FE000003FE0000003E0 000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0 000001E0000001E0000001E0000001E0000001E0000001E0001F81E000F061E001C019E0 078005E00F0003E00E0003E01E0001E03C0001E03C0001E07C0001E0780001E0F80001E0 F80001E0F80001E0F80001E0F80001E0F80001E0F80001E0F80001E0F80001E0780001E0 780001E03C0001E03C0001E01C0001E01E0003E00E0005E0070009E0038011F000E061FF 003F81FF20327DB125>I<003F800000E0E0000380380007003C000E001E001E001E001C 000F003C000F007C000F0078000F8078000780F8000780F8000780FFFFFF80F8000000F8 000000F8000000F8000000F8000000F8000000780000007C0000003C0000003C0000801E 0000800E0001000F0002000780020001C00C0000F03000001FC000191F7E9E1D>I<0007 E0001C1000383800707C00E07C01E07C01C03803C00003C00003C00003C00003C00003C0 0003C00003C00003C00003C00003C00003C000FFFFC0FFFFC003C00003C00003C00003C0 0003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C0 0003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00007E0007FFF 007FFF0016327FB114>I<000000F0007F030801C1C41C0380E81C070070080F0078001E 003C001E003C003E003E003E003E003E003E003E003E003E003E003E003E001E003C001E 003C000F007800070070000780E00009C1C000087F000018000000180000001800000018 000000180000001C0000000E0000000FFFF80007FFFF0003FFFF800E000FC0180001E030 0000F070000070E0000038E0000038E0000038E0000038E0000038700000707000007038 0000E01C0001C00700070001C01C00003FE0001E2F7E9F21>I<07000F801F801F800F80 0700000000000000000000000000000000000000000000000780FF80FF800F8007800780 078007800780078007800780078007800780078007800780078007800780078007800780 07800780078007800FC0FFF8FFF80D307EAF12>105 D<0780FE001FC000FF83078060F0 00FF8C03C18078000F9001E2003C0007A001E4003C0007A000F4001E0007C000F8001E00 07C000F8001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E0007 8000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E000780 00F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000 F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0 001E000FC001F8003F00FFFC1FFF83FFF0FFFC1FFF83FFF0341F7E9E38>109 D<0780FE0000FF83078000FF8C03C0000F9001E00007A001E00007A000F00007C000F000 07C000F000078000F000078000F000078000F000078000F000078000F000078000F00007 8000F000078000F000078000F000078000F000078000F000078000F000078000F0000780 00F000078000F000078000F000078000F000078000F000078000F000078000F0000FC001 F800FFFC1FFF80FFFC1FFF80211F7E9E25>I<001FC00000F0780001C01C00070007000F 0007801E0003C01C0001C03C0001E03C0001E0780000F0780000F0780000F0F80000F8F8 0000F8F80000F8F80000F8F80000F8F80000F8F80000F8F80000F8780000F07C0001F03C 0001E03C0001E01E0003C01E0003C00F00078007800F0001C01C0000F07800001FC0001D 1F7E9E21>I<0783E0FF8C18FF907C0F907C07A07C07C03807C00007C00007C000078000 078000078000078000078000078000078000078000078000078000078000078000078000 0780000780000780000780000780000780000FC000FFFE00FFFE00161F7E9E19>114 D<01FC100E03301800F0300070600030E00030E00010E00010E00010F00010F800007E00 003FF0001FFF000FFFC003FFE0003FF00001F80000F880003C80003C80001CC0001CC000 1CE0001CE00018F00038F00030CC0060C301C080FE00161F7E9E1A>I<00400000400000 400000400000400000C00000C00000C00001C00001C00003C00007C0000FC0001FFFE0FF FFE003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003 C00003C00003C00003C00003C00003C01003C01003C01003C01003C01003C01003C01003 C01001C02001E02000E0400078C0001F00142C7FAB19>I<078000F000FF801FF000FF80 1FF0000F8001F000078000F000078000F000078000F000078000F000078000F000078000 F000078000F000078000F000078000F000078000F000078000F000078000F000078000F0 00078000F000078000F000078000F000078000F000078000F000078000F000078001F000 078001F000078001F000038002F00003C004F00001C008F800007030FF80001FC0FF8021 1F7E9E25>I E /Fv 5 85 df<00000000C00000000000E00000000001E00000000003E0 0000000003E00000000007E00000000007E0000000000FE0000000000FE0000000001FE0 000000001FE00000000037E00000000067E00000000067E000000000C7E000000000C7F0 0000000183F00000000183F00000000303F00000000703F00000000603F00000000C03F0 0000000C03F00000001803F00000001803F00000003003F00000003003F00000006003F0 000000C003F0000000C003F00000018003F00000018003F8000003FFFFF8000003FFFFF8 0000060001F800000E0001F800000C0001F80000180001F80000180001F80000300001F8 0000300001F80000600001F80000E00001F80000C00001F80001C00001F80001C00001F8 0007C00001FC001FC00003FC00FFF8007FFFE0FFF8007FFFE02B327BB135>65 D<000FFFFFFE0000000FFFFFFF800000007F000FE00000007E0003F00000007E0000F800 00007E0000FC0000007E00007C000000FC00003E000000FC00003E000000FC00003F0000 00FC00001F000001F800001F000001F800001F800001F800001F800001F800001F800003 F000001F800003F000001F800003F000001F800003F000001F800007E000003F800007E0 00003F800007E000003F800007E000003F80000FC000003F00000FC000007F00000FC000 007F00000FC000007F00001F8000007E00001F800000FE00001F800000FE00001F800000 FC00003F000001FC00003F000001F800003F000001F800003F000003F000007E000003E0 00007E000007E000007E00000FC000007E00000F800000FC00001F800000FC00003F0000 00FC00007E000000FC0000FC000001F80001F0000001F80003E0000001F8000FC0000003 F8007F000000FFFFFFFC000000FFFFFFE000000031317BB036>68 D<000FFFFFFFFC000FFFFFFFFC00007F0001FC00007E00007C00007E00003C00007E0000 3C00007E0000180000FC0000180000FC0000180000FC0000180000FC0000180001F80000 180001F80000180001F80000180001F80000180003F00080100003F00180000003F00180 000003F00180000007E00300000007E00300000007E00700000007E01F0000000FFFFE00 00000FFFFE0000000FC01E0000000FC00E0000001F800C0000001F800C0000001F800C00 00001F800C0000003F00180000003F00080000003F00000000003F00000000007E000000 00007E00000000007E00000000007E0000000000FC0000000000FC0000000000FC000000 0000FC0000000001F80000000001F80000000001F80000000003F800000000FFFFF00000 00FFFFF00000002E317BB02F>70 D<000FFFFFF000000FFFFFFE0000007F003F8000007E 000FC000007E0007E000007E0003F000007E0001F80000FC0001F80000FC0001F80000FC 0001F80000FC0001F80001F80003F80001F80003F80001F80003F80001F80003F00003F0 0007F00003F00007E00003F0000FC00003F0000FC00007E0001F000007E0007E000007E0 00FC000007E007F000000FFFFFC000000FFFFF0000000FC00F8000000FC003C000001F80 03E000001F8001F000001F8001F000001F8001F800003F0001F800003F0001F800003F00 01F800003F0001F800007E0003F800007E0003F800007E0003F000007E0003F00000FC00 07F00000FC0007F00000FC0007F00800FC0007F00C01F80007F01801F80007F01801F800 03F03003F80003F030FFFFE001F0E0FFFFE000FFC0000000003F002E327BB034>82 D<07FFFFFFFFF00FFFFFFFFFF00FC00FE003F01E000FC000F01C000FC000E018000FC000 E038000FC0006030001F8000E030001F8000E060001F8000C060001F8000C060003F0000 C0C0003F0000C0C0003F0000C0C0003F0000C080007E00008000007E00000000007E0000 0000007E0000000000FC0000000000FC0000000000FC0000000000FC0000000001F80000 000001F80000000001F80000000001F80000000003F00000000003F00000000003F00000 000003F00000000007E00000000007E00000000007E00000000007E0000000000FC00000 00000FC0000000000FC0000000000FC0000000001F80000000001F80000000001F800000 00001F80000000003F00000000003F00000000003F0000000000FF00000000FFFFFF0000 00FFFFFF0000002C3173B033>84 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300dpi TeXDict begin %%EndSetup %%Page: 0 1 0 0 bop 795 947 a Fv(D)26 b(R)g(A)f(F)h(T)225 1038 y Fu(Do)r(cumen)n(t)20 b(for)i(a)f(Standard)g(Message-P)n(assing)f(In)n (terface)621 1232 y Ft(Message)c(P)o(assing)h(In)o(terface)e(F)l(orum) 766 1359 y(Septem)o(b)q(er)g(19,)h(1996)190 1417 y(This)h(w)o(ork)f(w)o (as)h(supp)q(orted)g(in)f(part)g(b)o(y)g(NSF)g(and)h(ARP)l(A)e(under)h (NSF)g(con)o(tract)283 1475 y(CD)o(A-9115428)j(and)e(Esprit)f(under)h (pro)s(ject)e(HPC)i(Standards)g(\(21111\).)p eop %%Page: 1 2 1 1 bop 166 45 a Fs(This)20 b(is)h(the)f(result)g(of)f(a)h(LaT)l(eX)g (run)g(of)g(a)f(draft)g(of)h(a)f(single)j(c)o(hapter)d(of)h(the)g(MPIF) f(Final)75 102 y(Rep)q(ort)d(do)q(cumen)o(t.)969 2828 y(i)p eop %%Page: 1 3 1 2 bop 75 356 a Fr(Chapter)34 b(10)75 564 y Fq(Miscellan)m(y)75 786 y Fs(There)12 b(are)g(a)g(n)o(um)o(b)q(er)g(of)g(topics)g(that)f (do)h(not)g(\014t)g(con)o(v)o(enien)o(tly)h(in)o(to)f(the)g(other)g(c)o (hapters.)19 b(W)l(e)12 b(collect)75 843 y(them)j(here.)75 986 y Fp(10.1)59 b(P)n(o)n(rtable)20 b(MPI)g(Pro)r(cess)f(Sta)n(rtup)75 1135 y Fo(Curren)o(t)14 b(Status:)j Fn(P)o(assed)e(once.)166 1239 y Fs(A)f(n)o(um)o(b)q(er)g(of)f(implemen)o(tations)i(of)e(MPI-1)g (pro)o(vide)i(a)e(startup)g(command)g(for)h(MPI)f(programs)75 1295 y(that)h(is)i(of)f(the)g(form)170 1389 y Fm(mpirun)23 b()g()g()75 1483 y Fs(Separating)16 b(the)g(command)g(to)g(start)f(the)h(program)f (from)g(the)h(program)f(itself)i(pro)o(vides)g(\015exibilit)o(y)l(,)75 1539 y(particularly)23 b(for)e(net)o(w)o(ork)g(and)h(heterogeneous)g (implemen)o(tations.)41 b(F)l(or)21 b(example,)j(the)e(startup)75 1596 y(script)16 b(need)g(not)f(run)g(on)g(one)g(of)g(the)g(mac)o (hines)h(that)f(will)i(b)q(e)f(executing)g(the)f(MPI)g(program)f (itself.)166 1652 y(Ha)o(ving)i(a)f(standard)h(startup)f(mec)o(hanism)i (also)e(extends)i(the)f(p)q(ortabilit)o(y)h(of)e(F)l(ortran,)f(C,)i (and)75 1708 y(C++)f(programs)f(one)g(step)h(further,)f(to)g(the)h (command)g(lines)h(and)f(scripts)g(that)f(manage)g(them.)20 b(F)l(or)75 1765 y(example,)c(a)f(v)m(alidation)i(suite)f(script)g (that)e(runs)i(h)o(undreds)g(of)f(programs)f(can)i(b)q(e)g(a)f(p)q (ortable)h(script)75 1821 y(if)f(it)f(is)g(written)g(using)h Fm(mpirun)p Fs(.)k(In)c(order)e(that)h(the)g(\\statndard")f(command)h (not)f(b)q(e)i(confused)g(with)p Fl(>)h Fk(\(Oct\))75 1878 y Fs(existing)h(practice,)f(whic)o(h)h(is)f(not)g(standard)f(and)h (not)g(p)q(ortable)g(among)f(implemen)o(tations,)i(instead)75 1934 y(of)e Fm(mpirun)f Fs(w)o(e)h(sp)q(ecify)i Fm(mpiexec)p Fs(.)1191 b Fl(?)16 b Fk(\(Oct\))166 1991 y Fs(It)f(is)h(prop)q(osed)f (that)170 2085 y Fm(mpiexec)23 b(-n)h()e()75 2178 y Fs(b)q(e)c(at)f(least)g(one)g(w)o(a)o(y)g(to)f(start)g Fm()g Fs(with)i(an)f(initial)j Fj(MPI)p 1275 2178 14 2 v 15 w(COMM)p 1432 2178 V 17 w(W)o(ORLD)d Fs(whose)g(group)75 2235 y(con)o(tains)22 b Fm()f Fs(pro)q(cesses.)42 b(Other)22 b(argumen)o(ts)g(to)g Fm(mpiexec)f Fs(ma)o(y)h(b)q(e)h (implemen)o(tation-)75 2291 y(dep)q(enden)o(t.)166 2348 y(Curren)o(tly)14 b(this)g(is)g(prop)q(osed)g(as)f(advice)h(to)f (implemen)o(tors,)i(rather)e(than)g(as)g(a)h(required)g(part)f(of)75 2404 y(MPI-2.)20 b(It)15 b(is)h(not)e(suggested)i(that)e(this)i(b)q(e)g (the)f(only)h(w)o(a)o(y)e(to)g(start)g(MPI)h(programs.)189 2510 y Fi(A)n(dvic)n(e)c(to)j(implementors.)37 b Fs(Implemen)o(tors,)12 b(if)g(they)g(do)f(pro)o(vide)h(a)f(sp)q(ecial)i(startup)e(command)189 2567 y(for)j(MPI)i(programs,)e(are)h(advised)h(to)f(giv)o(e)g(it)h(the) g(follo)o(wing)g(form.)j(The)d(syn)o(tax)f(is)g(c)o(hosen)h(in)189 2623 y(order)g(that)g Fm(mpiexec)g Fs(b)q(e)i(able)f(to)f(b)q(e)i(view) o(ed)f(as)g(a)f(command-line)j(v)o(ersion)e(of)f Fj(MPI)p 1711 2623 V 16 w(SP)l(A)-5 b(WN)189 2680 y Fs(\(See)15 b(Section)h Fh(??)p Fs(\).)964 2828 y(1)p eop %%Page: 2 4 2 3 bop 75 -100 a Fs(2)1133 b Fg(CHAPTER)15 b(10.)35 b(MISCELLANY)284 45 y Fm(mpiexec)23 b(-n)95 b()451 102 y(-soft)47 b(<)191 b(>)451 158 y(-host)47 b(<)191 b(>)451 214 y(-arch)47 b(<)191 b(>)451 271 y(-wdir)47 b(<)191 b(>)451 327 y(-path)47 b(<)191 b(>)451 384 y(-file)47 b(<)191 b(>)475 440 y(...)451 497 y()189 585 y Fs(for)9 b(the)i(case)f(where)g(a)g(single)i(command)e(line)i (for)e(the)g(application)i(program)d(and)h(its)h(argumen)o(ts)189 641 y(will)23 b(su\016ce.)41 b(F)l(or)22 b(the)g(case)g(corresp)q (onding)h(to)f Fj(MPI)p 1180 641 14 2 v 16 w(SP)l(A)-5 b(WN)p 1346 641 V 17 w(MUL)l(TIPLE)21 b Fs(there)h(are)g(t)o(w)o(o)189 698 y(p)q(ossible)17 b(formats:)189 769 y(F)l(orm)d(A:)284 861 y Fm(mpiexec)23 b({)h()f(})i(:)g({)f(...)h(})g (:)f({)h(...)g(})f(:)h(...)f(:)h({)g(...)f(})189 953 y Fs(As)17 b(with)g Fj(MPI)p 448 953 V 16 w(SP)l(A)-5 b(WN)p Fs(,)18 b(all)g(the)f(argumen)o(ts)f(are)h(optional.)26 b(\(Ev)o(en)17 b(the)h Fm(-n)23 b(x)41 b Fs(argumen)o(t)16 b(is)189 1009 y(optional;)h(the)f(default)h(is)g(implemen)o(tation)h (dep)q(enden)o(t.)25 b(It)17 b(migh)o(t)f(b)q(e)h Fm(1)p Fs(,)f(it)h(migh)o(t)f(b)q(e)h(tak)o(en)189 1066 y(from)11 b(an)i(en)o(vironmen)o(t)g(v)m(ariable,)h(or)e(it)g(migh)o(t)h(b)q(e)g (sp)q(eci\014ed)h(at)e(compile)i(time\).)19 b(The)13 b(names)f(of)189 1122 y(the)k(argumen)o(ts)e(are)i(tak)o(en)f(from)g (the)h(k)o(eys)g(in)g(the)g Fm(info)f Fs(argumen)o(t)g(to)h Fj(MPI)p 1567 1122 V 15 w(SP)l(A)-5 b(WN)p Fs(.)17 b(There)189 1179 y(ma)o(y)d(b)q(e)i(other,)e(implemen)o(tation-dep)q(end)q(en)o(t)k (argumen)o(ts)c(as)h(w)o(ell.)189 1250 y(Note)k(that)g(F)l(orm)f(A,)h (though)h(con)o(v)o(enien)o(t)g(to)f(t)o(yp)q(e,)h(prev)o(en)o (tscolons)g(from)e(b)q(eing)j(program)189 1306 y(argumen)o(ts.)e (Therefore)c(and)g(alternate,)g(\014le-based)i(form)d(is)i(allo)o(w)o (ed:)189 1377 y(F)l(orm)e(B:)284 1470 y Fm(mpiexec)23 b(-runfile)g()189 1562 y Fs(where)15 b(the)g(lines)i(of)e Ff(<)p Fs(\014lename)p Ff(>)i Fs(are)e(of)g(the)g(form)f(separated)h(b) o(y)g(the)h(colons)f(in)h(F)l(orm)f(A.)189 1633 y(\()p Fi(End)g(of)i(advic)n(e)f(to)g(implementors.)p Fs(\))75 1710 y Fh(Example)i(10.1)23 b Fi(Start)16 b(16)h(instanc)n(es)d(of)i Fm(myprog)g Fi(on)g(the)g(curr)n(ent)h(or)f(default)h(machine:)170 1784 y Fm(mpiexec)23 b(-n)h(16)g(myprog)75 1861 y Fh(Example)18 b(10.2)23 b Fi(Start)16 b(10)h(pr)n(o)n(c)n(esses)d(on)i(the)h(machine) f(c)n(al)r(le)n(d)f Fm(ferrari)p Fi(:)170 1935 y Fm(mpiexec)23 b(-n)h(10)g(-host)f(ferrari)g(myprog)75 2012 y Fh(Example)18 b(10.3)23 b Fi(Start)11 b(the)h Fm(ocean)f Fi(pr)n(o)n(gr)n(am)h(on)f (\014ve)g(suns)g(and)h(the)f Fm(atmos)g Fi(pr)n(o)n(gr)n(am)h(on)g(10)g (RS/6000's:)170 2086 y Fm(mpiexec)23 b(-n)h(5)g(-arch)f(sun)g(ocean)g (:)h(-n)g(10)f(-arch)h(rs6000)f(atmos)75 2163 y Fh(Example)18 b(10.4)23 b Fi(Start)17 b(thr)n(e)n(e)h(c)n(opies)f(of)h(the)g(same)f (pr)n(o)n(gr)n(am)h(with)g(di\013er)n(ent)f(c)n(ommand-line)g(ar)n(gu-) 75 2220 y(ments:)170 2293 y Fm(mpiexec)23 b(myprog)g(infile1)g(:)h (myprog)f(infile2)g(:)h(myprog)f(infile3)75 2371 y Fh(Example)18 b(10.5)23 b Fi(Start)11 b(the)h Fm(ocean)f Fi(pr)n(o)n(gr)n(am)h(on)f (\014ve)g(suns)g(and)h(the)f Fm(atmos)g Fi(pr)n(o)n(gr)n(am)h(on)g(10)g (RS/6000's)75 2427 y(\(F)m(orm)k(B\):)170 2500 y Fm(mpiexec)23 b(-runfile)g(myfile)75 2574 y Fi(wher)n(e)16 b Fm(myfile)g Fi(c)n(ontains)170 2647 y Fm(-n)24 b(5)48 b(-arch)23 b(sun)95 b(ocean)170 2704 y(-n)24 b(10)g(-arch)f(rs6000)g(atmos)1967 46 y Fk(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 3 5 3 4 bop 75 -100 a Fg(10.2.)34 b(LANGUA)o(GE)15 b(INTER)o(OPERABILITY) 898 b Fs(3)75 45 y Fp(10.2)59 b(Language)19 b(Interop)r(erabilit)n(y)75 148 y Fe(10.2.1)49 b(Intro)q(duction)75 281 y Fo(Curren)o(t)14 b(Status:)j Fn(P)o(assed)e(once.)166 385 y Fs(It)h(is)h(not)f(uncommon) h(for)e(library)i(dev)o(elop)q(ers)h(to)e(use)g(one)h(language)f(to)g (dev)o(elop)h(an)f(applica-)75 441 y(tions)g(library)g(to)f(b)q(e)h (used)h(b)o(y)e(applications)i(dev)o(elop)q(ers)g(using)f(another)g (language.)21 b(MPI)15 b(curren)o(tly)75 498 y(supp)q(orts)h(ANSI)g(C,) f(ANSI)h(C++,)g(and)g(F)l(ortran)e(77)h(bindings.)23 b(It)15 b(should)i(b)q(e)f(p)q(ossible)i(for)c(applica-)75 554 y(tions)h(in)h(an)o(y)f(of)g(the)g(supp)q(orted)h(languages)f(to)g (call)h(MPI-related)g(functions)g(in)g(another)f(language.)166 611 y(MPI)h(allo)o(ws)h(the)f(dev)o(elopmen)o(t)h(of)f(clien)o(t-serv)o (er)i(co)q(de,)f(with)g(MPI)f(comm)o(unication)h(used)g(b)q(e-)75 667 y(t)o(w)o(een)d(a)h(parallel)h(clien)o(t)g(and)f(a)g(parallel)h (serv)o(er.)k(It)14 b(should)i(b)q(e)g(p)q(ossible)g(to)e(co)q(de)i (the)f(serv)o(er)f(in)i(one)75 724 y(language,)j(and)g(the)f(clien)o (ts)i(in)f(another)f(language.)30 b(T)l(o)18 b(do)h(so,)f(in)o(tercomm) o(unications)i(should)f(b)q(e)75 780 y(p)q(ossible)e(b)q(et)o(w)o(een)e (applications)i(written)e(in)h(an)o(y)f(language.)166 837 y(There)g(are)g(sev)o(eral)h(issues)g(that)e(need)i(to)f(b)q(e)h (addressed,)f(to)f(ac)o(hiev)o(e)i(in)o(terop)q(erabilit)o(y)l(.)75 930 y Fh(Initializat)q(ion)26 b Fs(W)l(e)12 b(need)h(to)f(sp)q(ecify)i (ho)o(w)d(is)i(the)f(MPI)g(en)o(vironmen)o(t)h(initialized)i(for)d(all) h(languages.)75 1024 y Fh(Comm)o(unication)18 b(of)g(MPI)e(opaque)i(ob) s(jects)23 b Fs(W)l(e)13 b(need)h(to)f(sp)q(ecify)i(ho)o(w)e(are)g(MPI) g(ob)s(ject)g(han-)189 1081 y(dles)j(passed)g(b)q(et)o(w)o(een)g (languages.)21 b(W)l(e)16 b(also)f(need)i(to)e(sp)q(ecify)i(what)e (happ)q(ens)h(when)g(an)g(MPI)189 1137 y(ob)s(ject)k(is)i(accessed)g (in)g(one)f(language,)i(to)d(retriev)o(e)i(information)f(set)g(with)h (this)f(ob)s(ject)g(in)189 1194 y(another)14 b(language.)75 1287 y Fh(In)o(terlanguage)k(comm)o(unication)24 b Fs(W)l(e)18 b(need)g(to)e(sp)q(ecify)i(ho)o(w)f(messages)g(sen)o(t)g(in)g(one)h (language)189 1344 y(can)d(b)q(e)h(receiv)o(ed)g(in)g(another)f (language.)166 1438 y(It)f(is)h(highly)h(desirable)g(that)d(the)i (solution)g(for)f(in)o(terlanguage)h(in)o(terop)q(erabilit)o(y)h(b)q(e) f(extendable)75 1494 y(to)g(new)g(languages,)g(should)h(MPI)f(bindings) i(b)q(e)f(de\014ned)h(for)d(suc)o(h)i(languages.)75 1616 y Fe(10.2.2)49 b(Assumptions)1875 1655 y Fl(>)16 b Fk(\(Oct\))75 1702 y Fs(W)l(e)g(assume)g(that)f(con)o(v)o(en)o(tions)h(exist)h(for)e (MPI)h(programs)f(to)g(call)i(functions)g(in)g(MPI)f(programs)e(of)75 1758 y(another)e(language.)19 b(These)13 b(con)o(v)o(en)o(tions)f(sp)q (ecify)i(ho)o(w)e(to)f(link)j(routines)f(in)g(di\013eren)o(t)g (langauges)f(in)o(to)75 1815 y(one)j(program,)e(ho)o(w)g(to)h(call)i (functions)f(in)g(a)f(di\013eren)o(t)h(language,)f(ho)o(w)g(to)g(pass)g (argumen)o(ts)g(b)q(et)o(w)o(een)75 1871 y(languages,)23 b(and)f(the)f(corresp)q(ondence)i(b)q(et)o(w)o(een)f(basic)h(data)d(t)o (yp)q(es)i(of)f(di\013eren)o(t)h(languages.)39 b(In)75 1928 y(general,)18 b(these)f(con)o(v)o(en)o(tions)g(will)i(b)q(e)e (implemen)o(tation)i(dep)q(enden)o(t)f(\(ho)o(w)o(ev)o(er,)e(this)h(is) h(an)f(ongoing)75 1984 y(e\013ort)11 b(to)g(standardize)i(a)e(C)h (calling)i(in)o(terface)e(in)g(F)l(ortran\).)18 b(F)l(urthernote,)12 b(not)f(ev)o(ery)h(basic)g(datat)o(yp)q(e)75 2040 y(ma)o(y)h(ha)o(v)o (e)h(a)g(matc)o(hing)g(t)o(yp)q(e)g(in)h(other)e(languages.)20 b(F)l(or)14 b(example,)g(C/C++)g(c)o(haracter)g(strings)g(ma)o(y)75 2097 y(not)i(b)q(e)h(compatible)g(with)g(F)l(ortran)e Fm(CHARACTER)f Fs(v)m(ariables.)25 b(Ho)o(w)o(ev)o(er,)15 b(w)o(e)h(assume)g(that)f(a)h(F)l(ortran)75 2153 y Fm(INTEGER)c Fs(can)g(b)q(e)h(passed)g(to)f(a)g(C)h(or)f(C++)h(program.)18 b(W)l(e)12 b(also)h(assume)f(that)g(F)l(ortran,)g(C,)g(and)g(C++)75 2210 y(ha)o(v)o(e)j(address)g(sized)h(in)o(tegers.)1256 b Fl(?)16 b Fk(\(Oct\))75 2332 y Fe(10.2.3)49 b(Initializatio)q(n)75 2465 y Fo(Curren)o(t)14 b(Status:)j Fn(P)o(assed)e(once.)166 2568 y Fs(A)k(call)h(to)f Fm(MPI)p 441 2568 15 2 v 16 w(INIT\(\))p Fs(,)g(from)f(either)i(language,)g(initializes)i(MPI)d (for)f(execution)i(in)g(all)g(lan-)75 2625 y(guages.)1655 b Fl(>)16 b Fk(\(Oct\))-32 46 y(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 4 6 4 5 bop 75 -100 a Fs(4)1138 b Fg(CHAPTER)15 b(10.)30 b(MISCELLANY)189 45 y Fi(A)n(dvic)n(e)17 b(to)i(users.)56 b Fs(Certain)17 b(implemen)o(tations)i(use)g(the)e(\(inout\))h(argc,)g (argv)e(argumen)o(ts)h(of)189 102 y(the)f(C/C++)g(v)o(ersion)g(of)g Fj(MPI)p 736 102 14 2 v 16 w(INIT)f Fs(in)i(order)f(to)f(propagate)h (the)g(righ)o(t)g(v)m(alue)h(for)e(argc,)h(argv)189 158 y(to)g(all)i(executing)h(pro)q(cesses.)26 b(Use)18 b(of)f(the)g(F)l (ortran)f(v)o(ersion)i(of)f Fj(MPI)p 1424 158 V 15 w(INIT)g Fs(to)g(initialize)j(MPI)189 214 y(ma)o(y)14 b(result)i(in)g(a)f(loss)g (of)g(this)g(abilit)o(y)l(.)22 b(\()p Fi(End)15 b(of)i(advic)n(e)f(to)g (users.)p Fs(\))-117 276 y Fl(?)f Fk(\(Oct\))166 321 y Fs(The)g(function)h Fm(MPI)p 512 321 15 2 v 17 w(INITIALIZED)e Fs(returns)h(the)g(same)g(answ)o(er)g(in)h(an)o(y)f(language.)166 377 y(The)g(function)h Fm(MPI)p 512 377 V 17 w(FINALIZE)e Fs(\014nalizes)j(the)e(MPI)h(en)o(vironmen)o(ts)f(for)g(all)h (languages.)166 434 y(The)h(function)g Fm(MPI)p 515 434 V 17 w(ABORT)e Fs(kill)k(pro)q(cesses,)e(irresp)q(ectiv)o(e)h(of)e(the) g(language)h(used)g(b)o(y)f(the)h(caller)75 490 y(or)e(b)o(y)g(the)g (pro)q(cesses)h(killed.)166 547 y(The)e(MPI)g(en)o(vironmen)o(t)g(is)h (initialized)i(in)e(the)f(same)f(manner)h(for)f(all)i(languages)f(b)o (y)g Fj(MPI)p 1761 547 14 2 v 16 w(INIT)p Fs(.)75 603 y(I.e.,)d Fd(MPI)p 238 603 13 2 v 14 w(COMM)p 382 603 V 15 w(W)o(ORLD)f Fs(carries)g(the)g(same)g(information)g(regardless)h (of)f(language:)17 b(same)10 b(pro)q(cesses,)75 659 y(same)15 b(en)o(vironmen)o(tal)h(attributes,)e(same)h(error)g(handlers.)189 766 y Fi(A)n(dvic)n(e)h(to)h(users.)44 b Fs(The)16 b(use)g(of)g(sev)o (eral)g(languages)g(in)g(one)g(MPI)g(program)f(ma)o(y)g(require)i(the) 189 822 y(use)e(of)g(sp)q(ecial)i(options)e(at)g(compile)h(and/or)f (link)i(time.)j(\()p Fi(End)c(of)g(advic)n(e)g(to)h(users.)p Fs(\))189 928 y Fi(A)n(dvic)n(e)k(to)h(implementors.)79 b Fs(Implemen)o(tations)23 b(ma)o(y)e(selectiv)o(ely)i(link)g(language) f(sp)q(eci\014c)189 985 y(MPI)15 b(libraries)i(only)f(to)f(co)q(des)h (that)f(need)h(them,)g(so)f(as)g(not)g(to)g(increase)h(the)g(size)g(of) g(binaries)189 1041 y(for)21 b(co)q(des)i(that)e(use)i(only)f(one)g (language.)41 b(The)22 b(MPI)g(initializati)q(on)j(co)q(de)d(need)h(p)q (erform)189 1098 y(initialization)18 b(for)c(a)h(language)g(only)g(if)h (that)e(language)h(library)h(is)g(loaded.)k(\()p Fi(End)c(of)g(advic)n (e)g(to)189 1154 y(implementors.)p Fs(\))75 1276 y Fe(10.2.4)49 b(T)l(ransfer)16 b(of)g(handles)75 1409 y Fo(Curren)o(t)e(Status:)j Fn(P)o(assed)e(once.)-117 1456 y Fl(>)g Fk(\(Oct\))166 1513 y Fs(Handles)e(are)f(passed)h(across)f(languages)g(b)o(y)h(using)g (an)f(explicit)j(C)d(wrapp)q(er)h(to)e(con)o(v)o(ert)h(handles.)75 1569 y(The)j(con)o(v)o(ersion)h(is)f(automatic)g(b)q(et)o(w)o(een)h(C)f (and)g(C++.)166 1626 y(The)d(data)f(t)o(yp)q(e)h Fd(MPI)p 532 1626 V 14 w(Fint)g Fs(is)g(pro)o(vided)g(in)h(C)e(for)g(an)h(in)o (teger)g(of)f(the)h(size)h(that)d(matc)o(hes)i(a)f(F)l(ortran)75 1682 y Fd(INTEGER)p Fs(;)j(usually)l(,)i Fd(MPI)p 517 1682 V 15 w(Fint)f Fs(will)i(b)q(e)f(equiv)m(alen)o(t)h(to)d Fd(int)p Fs(.)166 1739 y(The)f(follo)o(wing)h(t)o(w)o(o)e(functions)i (are)f(pro)o(vided)h(in)g(C)f(to)g(con)o(v)o(ert)f(from)h(a)g(F)l (ortran)f(handle)i(\(whic)o(h)75 1795 y(is)i(an)f(in)o(teger\))g(to)f (a)h(C)g(handle,)h(and)g(vice)g(v)o(ersa.)75 1851 y Fm(MPI)p 150 1851 15 2 v 17 w(Handle)23 b(MPI)p 406 1851 V 17 w(Int2handle\()f(MPI)p 781 1851 V 17 w(Fint)h(f)p 941 1851 V 17 w(handle,)g(MPI)p 1221 1851 V 17 w(Handle)p 1382 1851 V 16 w(type)g(handle)p 1661 1851 V 17 w(kind\))75 1938 y(MPI)p 150 1938 V 17 w(Fint)g(MPI)p 358 1938 V 17 w(Handle2int\()f(MPI)p 733 1938 V 17 w(Handle)h(c)p 941 1938 V 17 w(handle,)g(MPI)p 1221 1938 V 17 w(Handle)p 1382 1938 V 16 w(type)g(handle)p 1661 1938 V 17 w(kind\))166 2024 y Fs(If)f Fj(f)p 235 2024 14 2 v 16 w(handle)g Fs(is)g(a)f(v)m (alid)i(F)l(ortran)e(handle)h(to)f(an)g(opaque)h(ob)s(ject,)g(and)f Fj(handle)p 1598 2024 V 18 w(kind)h Fs(sp)q(eci\014es)75 2081 y(the)17 b(t)o(yp)q(e)f(of)h(ob)s(ject,)f(then)h Fj(MPI)p 654 2081 V 16 w(INT2HANDLE)f Fs(returns)h(a)f(v)m(alid)i(C)f (handle)h(to)e(that)g(same)g(ob)s(ject;)75 2137 y(if)21 b Fj(f)p 139 2137 V 16 w(handle)h Fs(is)f(a)g(n)o(ull)h(F)l(ortran)d (handle,)k(then)e Fj(MPI)p 1016 2137 V 16 w(INT2HANDLE)f Fs(returns)h(a)f(n)o(ull)i(C)e(handle;)k(if)75 2194 y Fj(f)p 92 2194 V 16 w(handle)17 b Fs(is)e(an)g(in)o(v)m(alid)j(F)l (ortran)c(handle,)i(then)f Fj(MPI)p 1011 2194 V 16 w(INT2HANDLE)g Fs(returns)g(an)g(in)o(v)m(alid)i(C)e(handle;)75 2250 y(and)g(similarly)l(,)i(for)e Fj(MPI)p 513 2250 V 16 w(HANDLE2INT)p Fs(.)75 2356 y Fh(Example)j(10.6)23 b Fs(The)10 b(example)h(b)q(elo)o(w)g(illustrates)g(ho)o(w)e(the)i(F)l (ortran)e(MPI)h(function)h Fj(MPI)p 1677 2356 V 16 w(TYPE)p 1810 2356 V 16 w(COMMIT)75 2413 y Fs(can)k(b)q(e)h(implemen)o(ted)h(b)o (y)d(wrapping)i(the)f(C)g(MPI)g(function)h Fj(MPI)p 1244 2413 V 15 w(T)l(yp)q(e)p 1351 2413 V 18 w(commit)d Fs(with)i(a)g(C)g (wrapp)q(er)75 2469 y(to)g(do)g(handle)h(con)o(v)o(ersions.)75 2576 y Fm(!)24 b(FORTRAN)f(PROCEDURE)75 2632 y(SUBROUTINE)f (MPI_TYPE_COMMIT\()g(DATATYPE,)h(IERR\))75 2689 y(INTEGER)g(DATATYPE,)g (IERR)1967 46 y Fk(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 5 7 5 6 bop 75 -100 a Fg(10.2.)34 b(LANGUA)o(GE)15 b(INTER)o(OPERABILITY) 898 b Fs(5)75 45 y Fm(CALL)23 b(MPI_X_TYPE_COMMIT\(DATATYPE,)d(IERR\)) 75 102 y(RETURN)75 158 y(END)75 271 y(!)k(C)f(wrapper)75 384 y(void)g(MPI_X_TYPE_COMMIT\()f(MPI_Fint)g(*f_handle,)h(*ierr\))75 440 y({)75 497 y(MPI_Datatype)f(datatype;)75 610 y(datatype)h(=)g (\(MPI_Datatype\)MPI_int2handle\()d(*f_handle,)j (MPI_DATATYPE_HANDLE\);)75 666 y(*ierr)g(=)h (\(MPI_Fint\)MPI_Type_commit)o(\()d(&datatype\);)75 723 y(*f_handle)i(=)g(MPI_handle2int\()f(\(MPI_Handle\)datatype,)75 779 y(MPI_DATATYPE_HANDLE\);)75 835 y(return\(\);)75 892 y(})166 994 y Fs(The)10 b(same)g(approac)o(h)g(can)g(b)q(e)h(used)g (for)e(all)i(other)f(MPI)g(functions.)19 b(The)11 b(call)g(to)e Fj(MPI)p 1637 994 14 2 v 16 w(INT2HANDLE)75 1050 y Fs(\(resp.)k Fj(MPI)p 285 1050 V 16 w(HANDLE2INT)p Fs(\))g(can)i(b)q(e)f(omitted)g (when)g(the)g(handle)i(is)e(an)g Fn(OUT)g Fs(\(resp.)f Fn(IN)p Fs(\))h(argumen)o(t,)75 1107 y(rather)h(than)g Fn(INOUT)p Fs(.)166 1256 y Fo(Discussion:)g Fn(Do)f(w)o(e)g(also)f(w)o (an)o(t)h(C++)g(bindings)f(for)h(these)h(functions?)189 1406 y Fi(R)n(ationale.)62 b Fs(The)19 b(design)h(here)g(pro)o(vides)f (a)g(con)o(v)o(enien)o(t)g(solution)h(for)e(the)h(prev)m(alen)o(t)h (case,)189 1462 y(where)f(a)f(C)h(wrapp)q(er)g(is)h(used)f(to)f(allo)o (w)i(F)l(ortran)d(co)q(de)j(to)e(call)i(a)f(C)f(library)l(,)j(or)d(C)h (co)q(de)h(to)189 1518 y(call)g(a)f(F)l(ortran)f(library)l(.)32 b(The)19 b(use)h(of)f(C)f(wrapp)q(ers)i(is)f(m)o(uc)o(h)g(more)g(lik)o (ely)i(than)e(the)g(use)g(of)189 1575 y(F)l(ortran)12 b(wrapp)q(ers,)i(b)q(ecause)h(it)f(is)g(m)o(uc)o(h)g(more)f(lik)o(ely)j (that)d(a)g(v)m(ariable)j(of)d(t)o(yp)q(e)h Fd(INTEGER)f Fs(can)189 1631 y(b)q(e)j(passed)f(to)g(C,)f(than)h(a)g(C)g(handle)h (can)g(b)q(e)g(passed)f(to)g(F)l(ortran.)189 1706 y(The)21 b(use)h(of)f(con)o(v)o(ersion)g(functions,)j(rather)c(than)i(pro)q (cedures,)h(allo)o(w)f(us)f(to)g(use)g(inlinin)q(g,)189 1762 y(esp)q(ecially)c(in)f(the)f(case)g(where)g(these)g(functions)h (are)e(the)h(iden)o(tit)o(y)l(.)21 b(The)15 b(con)o(v)o(ersion)g (function)189 1819 y(in)f(the)g(wrapp)q(er)g(do)q(es)g(not)f(catc)o(h)h (an)f(in)o(v)m(alid)j(handle)f(argumen)o(t.)k(Instead,)14 b(an)g(in)o(v)m(alid)i(handle)189 1875 y(is)11 b(passed)h(b)q(elo)o(w)g (to)f(the)g(library)h(function,)h(whic)o(h,)f(presumably)l(,)h(c)o(hec) o(ks)e(its)h(input)g(argumen)o(ts.)189 1932 y(\()p Fi(End)j(of)i(r)n (ationale.)p Fs(\))1875 1989 y Fl(>)f Fk(\(Oct\))189 2079 y Fi(A)n(dvic)n(e)d(to)h(users.)38 b Fs(The)13 b(user)g(needs)g (to)g(co)q(erce)g(the)g(result)g(returned)g(b)o(y)f Fj(MPI)p 1560 2079 V 16 w(int2handle)j Fs(from)189 2136 y Fd(MPI)p 266 2136 13 2 v 14 w(Handle)i Fs(to)d(a)h(sp)q(eci\014c)i(handle)g(t)o (yp)q(e.)j(\()p Fi(End)15 b(of)i(advic)n(e)f(to)g(users.)p Fs(\))1875 2193 y Fl(?)g Fk(\(Oct\))75 2301 y Fj(C)f(and)g(C++.)46 b Fs(The)15 b(C++)h(language)f(in)o(terface)g(pro)o(vides)g(the)g (functions)g(listed)h(b)q(elo)o(w)g(for)e(C/C++)75 2358 y(in)o(terop)q(erabilit)o(y)k(\(in)e(particular,)h(to)e(allo)o(w)i(syn) o(tactically)g(pleasing)g(calls)g(of)f(C)g(from)f(C++)i(and/or)75 2414 y(to)h(build)i(C/F)l(ortran)d(in)o(terfaces)i(for)f(C++\).)30 b(The)19 b(tok)o(en)f Fm(CLASS)g Fs(is)h(used)g(b)q(elo)o(w)g(to)f (indicate)i(an)o(y)75 2470 y(v)m(alid)d Fj(MPI)d Fs(opaque)i(handle)g (name)f(\(e.g.)f Fm(Comm)p Fs(\).)166 2527 y(The)f(follo)o(wing)g Fm(operator=\(\))e Fs(function)i(allo)o(ws)g(assignmen)o(t)f(from)g(a)g (C)h Fj(MPI)f Fs(handle)h(to)f(a)g(C++)75 2583 y Fj(MPI)j Fs(handle.)75 2663 y Fm(MPI::CLASS&)22 b(MPI::CLASS::operator=\(const)f (MPI)p 1105 2663 15 2 v 16 w(CLASS&)i(data\))1875 2704 y Fl(>)16 b Fk(\(Oct\))-32 46 y(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 6 8 6 7 bop 75 -100 a Fs(6)1138 b Fg(CHAPTER)15 b(10.)30 b(MISCELLANY)166 45 y Fs(The)17 b(constructor)f(b)q(elo)o(w)h(creates)g (a)f(C++)i Fj(MPI)e Fs(ob)s(ject)g(from)g(a)h(C)f Fj(MPI)g Fs(handle.)26 b(This)18 b(allo)o(ws)75 102 y(the)d(automatic)g (promotion)g(of)g(a)f(C)h Fj(MPI)g Fs(handle)h(to)f(a)g(C++)h Fj(MPI)e Fs(handle.)75 176 y Fm(MPI::CLASS\(const)22 b(MPI)p 556 176 15 2 v 16 w(CLASS&)h(data\))75 290 y Fh(Example)18 b(10.7)23 b Fs(In)d(order)f(for)h(a)f(C)h(program)e(to)h (use)i(a)e(C++)h(library)l(,)i(the)e(C++)g(library)h(m)o(ust)75 347 y(exp)q(ort)16 b(a)h(C)f(in)o(terface)h(that)f(pro)o(vides)h (appropriate)g(con)o(v)o(ersions)f(b)q(efore)h(in)o(v)o(oking)g(the)g (underlying)75 403 y(C++)g(library)h(call.)26 b(This)17 b(example)h(sho)o(ws)e(a)g(C)h(in)o(terface)g(function)h(that)e(in)o(v) o(ok)o(es)g(a)h(C++)g(library)75 459 y(call)c(with)g(a)f(C)g(comm)o (unicator;)g(the)h(comm)o(unicator)e(is)i(automatically)g(promoted)f (to)f(a)h(C++)h(handle)75 516 y(when)j(the)f(underlying)i(C++)f (function)g(is)g(in)o(v)o(ok)o(ed.)75 600 y Fm(void)23 b(cpp_lib_call\(MPI::Comm)e(cpp_comm\);)75 713 y(extern)i("C")g({)75 769 y(void)g(c_interface\(MPI_Comm)e(c_comm\);)75 825 y(})75 938 y(void)i(c_interface\(MPI_Comm)e(c_comm\))75 995 y({)75 1051 y(cpp_lib_call\(c_comm\);)75 1108 y(})166 1191 y Fs(The)15 b(follo)o(wing)g(function)h(allo)o(ws)f(con)o(v)o (ersion)g(from)f(C++)h(ob)s(jects)f(to)g(C)h Fj(MPI)f Fs(handles.)21 b(In)16 b(this)75 1248 y(case,)f(the)g(casting)g(op)q (erator)g(is)g(o)o(v)o(erloaded)h(to)e(pro)o(vide)i(the)f(functionalit) o(y)l(.)75 1323 y Fm(MPI::CLASS::operator)21 b(MPI)p 651 1323 V 17 w(CLASS\(\))i(const)75 1436 y Fh(Example)18 b(10.8)23 b Fs(A)15 b(C)f(library)i(routine)g(is)f(called)i(from)d(a)h (C++)g(program.)k(The)c(C)g(library)g(routine)75 1493 y(is)h(protot)o(yp)q(ed)e(to)h(tak)o(e)f(an)i Fm(MPI)p 647 1493 V 16 w(Comm)f Fs(as)g(an)g(argumen)o(t.)75 1577 y Fm(extern)23 b("C")g({)75 1633 y(void)g(c_lib_call\(MPI_Comm)f (c_comm\);)75 1689 y(})75 1802 y(void)h(cpp_function\(\))75 1859 y({)75 1915 y(MPI::Comm)g(cpp_comm;)75 1972 y (cpp_comm.Dup\(MPI::COMM_WOR)o(LD\);)75 2028 y(c_lib_call\(cpp_comm\);) 75 2085 y(})189 2168 y Fi(R)n(ationale.)60 b Fs(Pro)o(viding)19 b(con)o(v)o(ersion)f(from)g(C)h(to)e(C++)i(via)g(constructors)f(and)h (from)e(C++)189 2225 y(to)e(C)h(via)g(casting)g(allo)o(ws)h(the)f (compiler)h(to)f(mak)o(e)f(automatic)h(con)o(v)o(ersions.)22 b(Calling)c(C)e(from)189 2281 y(C++)k(b)q(ecomes)g(trivial,)h(as)f(do)q (es)g(the)f(pro)o(vision)i(of)e(a)g(C)h(or)f(F)l(ortran)f(in)o(terface) i(to)f(a)g(C++)189 2338 y(library)l(.)i(\()p Fi(End)15 b(of)h(r)n(ationale.)p Fs(\))189 2421 y Fi(A)n(dvic)n(e)f(to)i (implementors.)43 b Fs(While)17 b(the)f(pro)o(vided)g(casting)g(and)g (promotion)f(functions)i(allo)o(w)189 2478 y(the)11 b(seamless)g(con)o (v)o(ersions)g(b)q(et)o(w)o(een)g(C)f(and)h(C++)h Fj(MPI)e Fs(handles,)i(there)f(are)g(sev)o(eral)g(situations)189 2534 y(in)16 b(the)f Fj(MPI)g Fs(C)g(bindings)i(\(and)e(third)h(part)o (y)e(C)h(library)h(calls\))g(where)g(a)f Fi(p)n(ointer)g Fs(to)f(an)i Fj(MPI)e Fs(C)189 2591 y(handle)g(is)h(required)f(\(e.g.,) f Fm(MPI)p 742 2591 V 16 w(CANCEL)p Fs(\).)f(Curren)o(tly)l(,)i(a)f (C++)h(user)g(w)o(ould)g(ha)o(v)o(e)f(to)g(explicitly)189 2647 y(cast)k(the)g(C++)h(handle)h(to)e(C)g(v)m(ariable,)i(and)f(then)f (use)h(the)g Fm(&)f Fs(op)q(erator)g(to)g(get)g(its)g(address.)189 2704 y(F)l(or)d(example,)i(the)f(C++)h(syn)o(tax)e(to)h(call)h(the)f(C) g Fm(MPI)p 1143 2704 V 17 w(Cancel\(\))f Fs(function)i(is)g(as)f(follo) o(ws:)1967 46 y Fk(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 7 9 7 8 bop 75 -100 a Fg(10.2.)29 b(LANGUA)o(GE)15 b(INTER)o(OPERABILITY) 903 b Fs(7)189 45 y Fm(MPI::Request)22 b(cpp_request;)189 102 y(MPI_Request)g(c_request;)189 158 y(...)189 214 y(c_request)g(=)i(cpp_request;)189 271 y(MPI_Cancel\(&c_request\);)189 396 y Fs(A)12 b(cleaner)h(syn)o(tax)e(can)i(b)q(e)g(supp)q(orted)f (with)h(the)f(follo)o(wing)h(functions:)19 b(a)12 b(promotion)g(op)q (erator)189 452 y(from)i(a)h(C)g Fj(MPI)g Fs(handle)h(p)q(oin)o(ter,)f (and)h(a)f(casting)g(op)q(erator)f(to)h(a)g(C)g Fj(MPI)f Fs(handle)j(p)q(oin)o(ter.)189 527 y Fm(MPI::CLASS\(const)k(MPI)p 669 527 15 2 v 17 w(CLASS*)i(data\))189 633 y(MPI::CLASS::operator)e (MPI)p 765 633 V 17 w(CLASS*\(\))189 738 y Fs(If)g(these)g(functions)h (are)e(supp)q(orted)i(b)o(y)f(the)g(implemen)o(tation,)i(a)e(C++)g (program)f(can)h(call)189 794 y Fm(MPI)p 264 794 V 16 w(Cancel\(\))14 b Fs(as)h(follo)o(ws:)189 919 y Fm(MPI::Request)22 b(cpp_request;)189 976 y(...)189 1032 y(MPI_Cancel\(cpp_request\))o(;) 189 1157 y Fs(\()p Fi(End)15 b(of)i(advic)n(e)f(to)g(implementors.)p Fs(\))1875 1218 y Fl(?)g Fk(\(Oct\))75 1329 y Fe(10.2.5)49 b(MPI)17 b(Opaque)g(objects)75 1462 y Fo(Curren)o(t)d(Status:)j Fn(P)o(assed)e(once.)166 1565 y Fs(In)h(general,)f(opaque)g(ob)s(jects) f(are)h(\\the)g(same")f(in)i(all)g(languages:)k(they)15 b(carry)g(the)g(same)g(infor-)75 1622 y(mation,)j(and)f(ha)o(v)o(e)h (the)f(same)g(meaning)i(in)f(b)q(oth)g(languages.)27 b(The)18 b(mec)o(hanism)g(describ)q(ed)h(in)g(the)75 1678 y(previous)d(section)g(can)f(b)q(e)h(used)g(to)e(pass)h (references)h(to)f(MPI)g(ob)s(jects)g(from)f(language)i(to)e(language.) 75 1735 y(An)h(ob)s(ject)g(created)g(in)h(one)g(language)f(can)g(b)q(e) h(accessed,)g(mo)q(di\014ed)g(or)f(freed)h(in)g(another)e(language.)166 1791 y(W)l(e)h(examine)h(b)q(elo)o(w)g(in)g(more)f(detail,)h(issues)g (that)e(arise)i(for)e(eac)o(h)h(t)o(yp)q(e)h(of)f(MPI)g(ob)s(ject.)75 1913 y Fe(10.2.6)49 b(Datat)o(yp)q(es)75 2046 y Fo(Curren)o(t)14 b(Status:)j Fn(P)o(assed)e(once.)166 2150 y Fs(Datat)o(yp)q(es)d(enco)q (de)j(the)f(same)f(information)h(in)h(all)f(languages.)20 b(E.g.,)12 b(a)h(datat)o(yp)q(e)g(accessor)h(lik)o(e)75 2206 y Fj(MPI)p 160 2206 14 2 v 16 w(TYPE)p 293 2206 V 17 w(EXTENT)h Fs(will)h(return)f(the)g(same)f(information)h(in)h(all) f(three)g(languages.)20 b(If)15 b(a)f(datat)o(yp)q(e)75 2262 y(de\014ned)22 b(in)f(one)f(language)h(is)f(used)h(for)f(a)g(comm) o(unication)h(call)g(in)g(another)f(language,)h(then)g(the)75 2319 y(message)c(sen)o(t)h(will)h(b)q(e)f(iden)o(tical)i(to)d(the)g (message)h(that)f(w)o(ould)h(b)q(e)g(sen)o(t)f(from)g(the)h(\014rst)f (language:)75 2375 y(the)e(same)g(comm)o(unication)h(bu\013er)f(is)g (accessed,)g(and)h(the)f(same)g(represen)o(tation)g(con)o(v)o(ersion)g (is)g(p)q(er-)75 2432 y(formed,)c(if)h(needed.)20 b(All)13 b(basic)f(datat)o(yp)q(es)f(can)g(b)q(e)h(used)g(in)h(datat)o(yp)q(e)d (constructors)h(in)h(an)o(y)f(language.)75 2488 y(If)k(a)g(datat)o(yp)q (e)g(is)h(committed,)f(it)g(can)g(b)q(e)h(used)g(for)f(comm)o (unication)h(in)g(an)o(y)e(languages.)166 2545 y(The)19 b(function)h Fj(MPI)p 530 2545 V 16 w(ADDRESS)g Fs(returns)f(the)h (same)f(v)m(alue)h(in)g(all)g(languages.)32 b(On)20 b(the)f(other)75 2601 y(hand,)h(w)o(e)e(do)h(not)g(require)g(that)f(the)h(constan)o(t)f Fd(MPI)p 1032 2601 13 2 v 15 w(BOTTOM)g Fs(has)h(the)g(same)f(v)m(alue) i(in)g(all)g(three)75 2658 y(languages.)-32 46 y Fk(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 8 10 8 9 bop 75 -100 a Fs(8)1133 b Fg(CHAPTER)15 b(10.)35 b(MISCELLANY)75 45 y Fh(Example)18 b(10.9)75 134 y Fm(!)24 b(FORTRAN)f(CODE)75 190 y(REAL)g(R\(5\))75 247 y(INTEGER)g(TYPE,)g (ADDR,)g(IERR)75 360 y(!)h(create)f(an)g(absolute)g(datatype)g(for)g (array)h(R)75 416 y(CALL)f(MPI_ADDRESS\()g(R,)g(ADDR,)g(IERR\))75 473 y(CALL)g(MPI_TYPE_STRUCT\(1,)f(5,)h(ADDR,)g(MPI_REAL,)g(TYPE,)g (IERR\))75 529 y(CALL)g(C_ROUTINE\(TYPE\))75 642 y(/*)h(C)f(code)h(*/) 75 755 y(void)f(C_ROUTINE\(MPI_Fint)f(*ftype\))75 811 y({)75 868 y(int)h(count)h(=)f(5;)75 924 y(void)g(*handle;)75 981 y(int)g(lens[2])g(=)h({1,1};)75 1037 y(MPI_Aint)f(displs[2];)75 1094 y(MPI_Datatype)f(types[2],)h(newtype;)75 1207 y(/*)h(create)f(an)g (absolute)g(datatype)g(for)g(buffer)g(that)h(consists)70 b(*/)75 1263 y(/*)47 b(of)24 b(count,)f(followed)g(by)g(R\(5\))668 b(*/)75 1376 y(MPI_Addr\(&count,)22 b(displs[0]\);)75 1432 y(displs[1])h(=)g(0;)75 1489 y(types[0])g(=)g(MPI_INT;)75 1545 y(types[1])g(=)g(\(MPI_Datatype\)MPI_Int2handle\(f)o(type\);)75 1602 y(MPI_Type_struct\(2,)e(lens,)j(displs,)e(types,)i(&newtype\);)75 1658 y(MPI_Type_commit\(&newtype\);)75 1771 y(MPI_Send\(MPI_BOTTOM,)d (1,)j(newtype,)e(1,)i(0,)g(MPI_COMM_WORLD\);)75 1828 y(/*)g(the)f(message)g(sent)g(contains)g(an)h(int)f(count)g(of)h(5,)f (followed)47 b(*/)75 1884 y(/*)24 b(by)f(the)h(5)f(REAL)h(entries)f(of) g(the)h(Fortran)f(array)g(R.)238 b(*/)75 1940 y(})189 2040 y Fi(R)n(ationale.)58 b Fs(The)18 b(curren)o(t)g(MPI)g(standard)g (sp)q(eci\014es)i(that)d Fd(MPI)p 1363 2040 13 2 v 15 w(ADDRESS)g Fs(can)h(b)q(e)h(used)g(in)189 2096 y(initialization)i (expressions)f(in)g(C,)e(but)h(not)f(in)i(F)l(ortran.)29 b(Since)21 b(F)l(ortran)c(do)q(es)i(not)g(supp)q(ort)189 2153 y(normally)11 b(call)h(b)o(y)f(v)m(alue,)h(then)f Fd(MPI)p 815 2153 V 15 w(ADDRESS)f Fs(m)o(ust)g(b)q(e)i(in)f(F)l (ortran)f(the)h(name)f(of)h(a)f(prede\014ned)189 2209 y(static)19 b(v)m(ariable,)k(e.g.,)d(a)f(v)m(ariable)j(in)f(an)f(MPI)g (declared)h Fd(COMMON)f Fs(blo)q(c)o(k.)35 b(On)20 b(the)g(other)189 2266 y(hand,)e(in)h(C,)e(it)h(is)g(natural)f(to)g(tak)o(e)g Fd(MPI)p 930 2266 V 15 w(BOTTOM)f(=)g(0)h Fs(\(Ca)o(v)o(eat:)23 b(De\014ning)c Fd(MPI)p 1681 2266 V 14 w(BOTTOM)189 2322 y(=)e(0)h Fs(implies)i(that)e Fd(NULL)i Fs(p)q(oin)o(ter)e(cannot)h(b)q (e)g(distinguished)i(from)d Fd(MPI)p 1502 2322 V 14 w(BOTTOM)p Fs(;)g(ma)o(y)f(b)q(e)189 2378 y Fd(MPI)p 266 2378 V 14 w(BOTTOM)12 b(=)i(1)f Fs(is)i(b)q(etter)f Ff(:)8 b(:)g(:)e Fs(\))13 b(Requiring)j(that)e(the)g(F)l(ortran)f(and)h(C)g(v)m(alues)h (b)q(e)g(the)f(same)189 2435 y(will)i(complicate)h(the)e (initialization)j(pro)q(cess.)i(\()p Fi(End)c(of)g(r)n(ationale.)p Fs(\))189 2534 y Fi(A)n(dvic)n(e)e(to)h(implementors.)39 b Fs(The)14 b(follo)o(wing)g(implemen)o(tation)h(can)f(b)q(e)h(used:)k (MPI)14 b(addresses,)189 2591 y(as)e(returned)h(b)o(y)f Fj(MPI)p 569 2591 14 2 v 16 w(ADDRESS)p Fs(,)h(will)h(ha)o(v)o(e)f(the) f(same)h(v)m(alue)g(in)h(all)f(languages.)19 b(One)14 b(ob)o(vious)189 2647 y(c)o(hoice)e(is)g(that)g(MPI)f(addresses)h(b)q (e)g(iden)o(tical)i(to)d(regular)h(addresses.)19 b(The)12 b(address)g(is)g(stored)f(in)189 2704 y(the)k(datat)o(yp)q(e,)f(when)h (datat)o(yp)q(es)f(with)h(absolute)g(addresses)g(are)g(constructed.)20 b(When)15 b(a)f(send)1967 46 y Fk(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 9 11 9 10 bop 75 -100 a Fg(10.2.)34 b(LANGUA)o(GE)15 b(INTER)o(OPERABILITY) 898 b Fs(9)189 45 y(or)13 b(receiv)o(e)i(op)q(eration)f(is)h(p)q (erformed,)f(then)g(addresses)g(stored)g(in)h(a)e(datat)o(yp)q(e)h(are) f(in)o(terpreted)189 102 y(as)h(displacemen)o(ts)j(that)d(are)g(all)i (augmen)o(ted)f(b)o(y)g(a)f(base)h(address.)20 b(This)c(base)f(address) g(is)g(\(the)189 158 y(address)f(of)t(\))e Fj(buf)p Fs(,)j(or)e(zero,)h (if)h Fj(buf)f(=)h(MPI)p 921 158 14 2 v 15 w(BOTTOM)p Fs(.)g(Th)o(us,)e(if)i Fd(MPI)p 1403 158 13 2 v 14 w(BOTTOM)f Fs(is)g(zero)g(then)g(a)189 214 y(send)g(or)f(receiv)o(e)i(call)g(with) f Fj(buf)g(=)g(MPI)p 885 214 14 2 v 16 w(BOTTOM)h Fs(is)f(implemen)o (ted)h(exactly)g(as)e(a)g(call)i(with)f(a)189 271 y(regular)h(bu\013er) g(argumen)o(t:)k(in)d(b)q(oth)f(cases)g(the)h(base)f(address)g(is)h Fj(buf)p Fs(.)21 b(On)15 b(the)g(other)g(hand,)g(if)189 327 y Fd(MPI)p 266 327 13 2 v 14 w(BOTTOM)e Fs(is)h(not)g(zero,)f(then) h(the)g(implemen)o(tation)h(has)e(to)g(b)q(e)h(sligh)o(tly)h (di\013eren)o(t.)20 b(A)14 b(test)189 384 y(is)h(p)q(erformed)f(to)g(c) o(hec)o(k)h(whether)f Fj(buf)i(=)f(MPI)p 1011 384 14 2 v 15 w(BOTTOM)p Fs(.)g(If)g(true,)f(then)h(the)f(base)h(address)f(is) 189 440 y(zero,)g(otherwise)i(it)f(is)h Fj(buf)p Fs(.)21 b(In)16 b(particular,)f(if)h Fd(MPI)p 1083 440 13 2 v 14 w(BOTTOM)f Fs(do)q(es)h(not)f(ha)o(v)o(e)f(the)i(same)f(v)m(alue)189 497 y(in)i(F)l(ortran)f(and)h(C/C++,)g(then)g(an)f(additional)i(test)f (for)f Fj(bu\013)h(=)g(MPI)p 1463 497 14 2 v 16 w(BOTTOM)g Fs(is)h(needed)189 553 y(in)e(at)e(least)i(one)f(of)g(the)g(languages.) 189 628 y(It)f(ma)o(y)g(b)q(e)i(desirable)g(to)e(use)h(a)g(v)m(alue)h (other)e(than)g(zero)h(for)f Fd(MPI)p 1339 628 13 2 v 14 w(BOTTOM)h Fs(ev)o(en)g(in)g(C/C++,)189 685 y(so)h(as)h(to)f (distinguish)j(it)e(from)f(a)h(NULL)g(p)q(oin)o(ter.)26 b(If)17 b Fd(MPI)p 1234 685 V 14 w(BOTTOM)e(=)h(c)h Fs(then)g(one)h (can)f(still)189 741 y(a)o(v)o(oid)f(the)g(test)g Fj(bu\013)h(=)f(MPI)p 705 741 14 2 v 16 w(BOTTOM)p Fs(,)h(b)o(y)f(using)h(the)g(displacemen)o (t)h(from)d Fd(MPI)p 1668 741 13 2 v 15 w(BOTTOM)p Fs(,)189 798 y(i.e.,)d(the)g(regular)f(address)h(-)g(c,)g(as)g(the)g(MPI)f (address)h(returned)g(b)o(y)g Fj(MPI)p 1441 798 14 2 v 16 w(ADDRESS)g Fs(and)g(stored)189 854 y(in)k(absolute)f(datat)o(yp)q (es.)k(\()p Fi(End)d(of)g(advic)n(e)h(to)f(implementors.)p Fs(\))75 976 y Fe(10.2.7)49 b(Addresses)75 1109 y Fo(Curren)o(t)14 b(Status:)j Fn(No)d(v)o(otes.)166 1213 y Fs(Some)i(of)g(the)h(datat)o (yp)q(e)e(accessors)h(and)h(constructors)e(ha)o(v)o(e)h(argumen)o(ts)g (of)f(t)o(yp)q(e)i Fd(MPI)p 1717 1213 13 2 v 14 w(Aint)f Fs(\(in)75 1269 y(C\))e(or)g Fd(MPI::Aint)f Fs(in)j(C++,)f(to)e(hold)j (addresses.)k(The)15 b(corresp)q(onding)g(argumen)o(ts,)f(in)h(F)l (ortran,)e(ha)o(v)o(e)75 1326 y(t)o(yp)q(e)20 b Fd(INTEGER)p Fs(.)g(This)h(causes)f(F)l(ortran)f(and)i(C/C++)f(to)g(b)q(e)h (incompatible,)i(in)e(an)g(en)o(vironmen)o(t)75 1382 y(where)15 b(addresses)h(ha)o(v)o(e)f(64)f(bits,)h(but)h(F)l(ortran)e Fd(INTEGER)p Fs(s)g(ha)o(v)o(e)h(32)g(bits.)166 1438 y(This)k(is)h(a)e(problem,)i(irresp)q(ectiv)o(e)h(of)d(in)o (terlanguage)h(issues.)32 b(Supp)q(ose)20 b(that)e(a)g(F)l(ortran)g (pro-)75 1495 y(cess)h(has)g(an)g(address)g(space)g(of)g Fl(\025)g Fs(4)g(GB.)g(What)f(should)i(b)q(e)g(the)f(v)m(alue)h (returned)f(in)h(F)l(ortran)e(b)o(y)75 1551 y Fj(MPI)p 160 1551 14 2 v 16 w(ADDRESS)p Fs(,)i(for)e(a)h(v)m(ariable)i(with)e (an)h(address)f(ab)q(o)o(v)o(e)g(2)1195 1535 y Fc(32)1232 1551 y Fs(?)33 b(The)19 b(prop)q(osed)h(design)g(aims)g(to)75 1608 y(solv)o(e)15 b(this)h(issue,)g(while)g(main)o(taining)h (compatibilit)o(y)f(with)g(curren)o(t)f(F)l(ortran)f(co)q(des.)166 1664 y(The)h(constan)o(t)g Fd(MPI)p 520 1664 13 2 v 14 w(ADDRESS)p 720 1664 V 14 w(KIND)g Fs(is)h(de\014ned)g(so)f(that,)f(in) i(F)l(ortran)e(90,)75 1721 y Fd(INTEGER\(KIND=MPI)p 474 1721 V 14 w(ADDRESS)p 674 1721 V 14 w(KIND\))22 b Fs(is)g(an)g(address) g(sized)h(in)o(teger)f(t)o(yp)q(e)g(\(t)o(ypically)l(,)i(but)e(not)75 1777 y(necessarily)l(,)f Fd(MPI)p 392 1777 V 15 w(ADDRESS)p 593 1777 V 14 w(KIND)e Fs(is)g(4)g(on)g(32)g(bit)g(address)h(mac)o (hines)g(and)f(8)g(on)g(64)f(bit)i(address)75 1834 y(mac)o(hines\).)g (Similarly)l(,)d(the)f(constan)o(t)75 1890 y Fd(MPI)p 152 1890 V 14 w(INTEGER)p 340 1890 V 15 w(KIND)g Fs(is)h(de\014ned)h (so)f(that)f Fd(INTEGER\(KIND=MPI)p 1231 1890 V 13 w(INTEGER)p 1418 1890 V 15 w(KIND\))g Fs(is)h(a)g(default)g(size)75 1947 y Fd(INTEGER)p Fs(.)166 2003 y(There)e(are)g(sev)o(en)h(functions) g(that)e(ha)o(v)o(e)h(address)g(argumen)o(ts:)k Fj(MPI)p 1380 2003 14 2 v 16 w(TYPE)p 1513 2003 V 17 w(HVECTOR)p Fs(,)75 2059 y Fj(MPI)p 160 2059 V 16 w(TYPE)p 293 2059 V 17 w(HINDEXED)p Fs(,)c Fj(MPI)p 647 2059 V 15 w(TYPE)p 779 2059 V 17 w(STRUCT)p Fs(,)h Fj(MPI)p 1083 2059 V 16 w(ADDRESS)p Fs(,)f Fj(MPI)p 1411 2059 V 16 w(TYPE)p 1544 2059 V 17 w(EXTENT)75 2116 y(MPI)p 160 2116 V 16 w(TYPE)p 293 2116 V 17 w(LB)g Fs(and)g Fj(MPI)p 550 2116 V 16 w(TYPE)p 683 2116 V 17 w(UB)p Fs(.)1104 b Fl(>)16 b Fk(\(Oct\))166 2172 y Fs(F)l(our)k(new)i(functions)f(are)g(pro)o (vided)h(to)e(supplemen)o(t)i(the)f(\014rst)g(four)g(functions)g(in)h (this)g(list.)75 2229 y(These)15 b(functions)h(are)f(describ)q(ed)i(in) e(Section)h(10.3,)d(page)i(13)g(.)k(The)d(remaining)g(three)f (functions)g(are)75 2285 y(supplemen)o(ted)g(b)o(y)e(the)g(new)g (function)h Fj(MPI)p 849 2285 V 15 w(GET)p 952 2285 V 17 w(EXTENT)p Fs(,)f(describ)q(ed)i(in)f(that)e(same)h(section.)20 b(The)75 2342 y(new)g(functions)g(are)f(synonimous)h(with)g(the)g(old)g (functions)g(in)g(C/C++,)h(or)e(on)g(F)l(ortran)f(systems)75 2398 y(where)23 b(default)g Fd(INTEGER)p Fs(s)e(are)i(address)f(sized.) 43 b(In)23 b(F)l(ortran,)g(they)f(accept)h(argumen)o(ts)f(of)g(t)o(yp)q (e)75 2455 y Fd(INTEGER\(KIND=MPI)p 474 2455 13 2 v 14 w(ADDRESS)p 674 2455 V 14 w(SIZE\))p Fs(,)d(wherev)o(er)g(argumen)o(ts) f(of)h(t)o(yp)q(e)g Fd(MPI)p 1479 2455 V 15 w(Aint)g Fs(are)g(used)h(in)g(C.)75 2511 y(On)j(F)l(ortran)e(77)h(systems)g (that)f(do)h(not)g(supp)q(ort)h(the)f(F)l(ortran)f(90)h Fd(KIND)g Fs(notation,)h(and)g(where)75 2568 y(addresses)e(are)g(64)f (bits)i(whereas)f(default)h Fd(INTEGER)p Fs(s)e(are)g(32)h(bits,)h (these)g(argumen)o(ts)e(will)j(b)q(e)e(of)75 2624 y(t)o(yp)q(e)16 b Fd(INTEGER*8)p Fs(.)k(The)c(old)h(functions)f(will)i(con)o(tin)o(ue)e (to)f(b)q(e)i(pro)o(vided,)f(for)f(p)q(ortabilit)o(y)l(.)23 b(Ho)o(w)o(ev)o(er,)75 2680 y(users)14 b(are)g(encouraged)g(to)f(switc) o(h)h(to)f(the)h(new)h(functions,)f(in)h(F)l(ortran,)e(so)g(as)h(to)f (a)o(v)o(oid)g(problems)i(on)-32 46 y Fk(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 10 12 10 11 bop 75 -100 a Fs(10)1110 b Fg(CHAPTER)15 b(10.)35 b(MISCELLANY)75 45 y Fs(systems)15 b(with)g(an)g(address)h(range)e Ff(>)f Fs(2)767 29 y Fc(32)805 45 y Fs(,)h(and)i(to)e(pro)o(vide)i (compatibilit)o(y)h(across)d(languages.)75 165 y Fe(10.2.8)49 b(Groups)75 298 y Fo(Curren)o(t)14 b(Status:)j Fn(P)o(assed)e(once.)166 401 y Fs(A)j(group)g(ob)s(ject)g(represen)o(t)h(the)f(same)g(group)g (in)h(all)h(languages:)26 b(the)18 b(same)g(pro)q(cesses,)i(with)75 458 y(the)15 b(same)g(ranks.)20 b(Group)15 b(accessors)f(suc)o(h)i(as)f Fj(MPI)p 986 458 14 2 v 15 w(GROUP)p 1153 458 V 18 w(SIZE)g Fs(return)g(the)h(same)f(v)m(alue.)75 577 y Fe(10.2.9)49 b(Communicato)o(rs)75 710 y Fo(Curren)o(t)14 b(Status:)j Fn(P)o(assed)e(once.)166 814 y Fs(A)k(comm)o(unicator)g(represen)o(ts)h (the)f(same)g(group)g(and)h(same)f(comm)o(unication)h(domain)g(in)g (all)75 870 y(languages.)25 b(In)17 b(particular,)h(a)e(message)g(sen)o (t)h(from)f(one)h(language)g(using)g(a)g(comm)o(unicator)f(can)h(b)q(e) 75 927 y(receiv)o(ed)e(from)e(another)g(language,)h(using)h(the)f(matc) o(hing)g(comm)o(unicator)f(on)h(another)f(pro)q(cess;)h(col-)75 983 y(lectiv)o(e)j(comm)o(unication)f(can)g(in)o(v)o(olv)o(e)g(pro)q (cesses)g(executing)h(co)q(de)f(written)g(in)g(di\013eren)o(t)g (languages,)75 1040 y(using)f(matc)o(hing)g(comm)o(unicators;)f (accessors)g(suc)o(h)h(as)f Fj(MPI)p 1152 1040 V 16 w(COMM)p 1310 1040 V 16 w(SIZE)h Fs(return)f(the)h(same)f(v)m(alue;)75 1096 y(etc.)166 1153 y(T)l(op)q(ology)e(information)h(asso)q(ciated)f (to)g(a)g(comm)o(unicator)g(can)g(b)q(e)h(accessed)g(in)h(an)o(y)e(of)f (the)i(three)75 1209 y(languages.)75 1329 y Fe(10.2.10)50 b(A)o(ttributes)75 1462 y Fo(Curren)o(t)14 b(Status:)j Fn(No)d(v)o(otes.)166 1565 y Fs(A)o(ttribute)k(k)o(eys)g(can)h(b)q(e)g (allo)q(cated)g(in)g(one)f(language)h(and)f(freed)h(in)g(another.)28 b(Similarly)l(,)21 b(at-)75 1622 y(tributes)13 b(v)m(alues)i(can)e(b)q (e)h(set)e(in)i(one)g(langage)f(and)g(accessed)g(in)h(another.)19 b(T)l(o)13 b(ac)o(hiev)o(e)h(this,)f(attribute)75 1678 y(k)o(eys)j(will)i(b)q(e)f(allo)q(cated)g(in)g(an)g(in)o(teger)f(range) g(that)f(is)i(v)m(alid)h(all)f(languages.)24 b(The)16 b(same)g(holds)h(true)75 1735 y(for)e(system)f(de\014ned)j(attribute)e (v)m(alues)h(\(suc)o(h)f(as)g Fd(MPI)p 1032 1735 13 2 v 15 w(T)m(A)o(G)p 1127 1735 V 14 w(UB,)d(MPI)p 1297 1735 V 14 w(IO)p Fs(,)i(etc.\))166 1791 y(A)o(ttribute)d(k)o(eys)g (declared)h(in)g(one)f(language)g(are)g(asso)q(ciated)h(with)f(cop)o(y) g(and)g(delete)h(functions)g(in)75 1848 y(that)e(language)g(\(the)h (functions)g(pro)o(vided)g(b)o(y)f(the)h Fj(MPI)p 1035 1848 14 2 v 16 w(Keyval)p 1178 1848 V 15 w(create)g Fs(call\).)19 b(When)11 b(a)f(comm)o(unicator)75 1904 y(is)16 b(duplicated,)i(for)d (eac)o(h)h(attribute,)g(the)g(corresp)q(onding)h(cop)o(y)e(function)i (is)f(called,)i(using)e(the)g(righ)o(t)75 1960 y(calling)j(con)o(v)o (en)o(tion)e(for)g(the)h(language)f(of)g(that)g(function;)h(and)g (similarly)l(,)h(for)e(the)g(delete)i(callbac)o(k)75 2017 y(function.)189 2106 y Fi(A)n(dvic)n(e)i(to)h(implementors.)77 b Fs(This)22 b(requires)g(that)f(attributes)g(b)q(e)h(tagged)f(either)h (as)f(\\C",)189 2163 y(\\C++")15 b(or)g(\\F)l(ortran",)e(and)j(that)e (the)i(language)f(tag)g(b)q(e)h(c)o(hec)o(k)o(ed)g(in)g(order)f(to)g (use)h(the)f(righ)o(t)189 2219 y(calling)i(con)o(v)o(en)o(tion)e(for)f (the)i(callbac)o(k)g(function.)21 b(\()p Fi(End)15 b(of)i(advic)n(e)f (to)g(implementors.)p Fs(\))166 2308 y(The)21 b(attribute)g (manipulation)h(functions)g(describ)q(ed)g(in)g(section)f(5.7)f(of)h (the)g(MPI)f(standard)75 2365 y(de\014ne)d(attributes)e(argumen)o(ts)g (to)f(b)q(e)j(of)e(t)o(yp)q(e)g Fd(void*)h Fs(in)g(C,)f(and)h(of)f(t)o (yp)q(e)g Fd(INTEGER)p Fs(,)f(in)j(F)l(ortran.)i(On)75 2421 y(some)14 b(systems,)f Fd(INTEGER)p Fs(s)h(will)i(ha)o(v)o(e)e(32) f(bits,)i(while)g(C/C++)g(p)q(oin)o(ters)f(will)i(ha)o(v)o(e)e(64)g (bits.)20 b(This)14 b(is)75 2478 y(a)j(problem)g(if)g(comm)o(unicator)g (attributes)f(are)h(used)g(to)g(mo)o(v)o(e)f(information)h(from)f(a)g (F)l(ortran)g(caller)75 2534 y(to)f(a)f(C/C++)i(callee,)g(or)f(vice-v)o (ersa.)-858 b Fl(>)15 b Fk(\(Oct\))166 2591 y Fs(MPI)j(will)h(store,)e (in)o(ternally)l(,)j(address)d(sized)i(attributes.)27 b(If)18 b(F)l(ortran)e Fd(INTEGER)p Fs(s)h(are)h(smaller,)75 2647 y(then)g(the)f(F)l(ortran)g(function)h Fj(MPI)p 694 2647 V 16 w(A)l(TTR)p 827 2647 V 17 w(GET)g Fs(will)h(return)e(the) h(least)f(signi\014can)o(t)i(part)e(of)g(the)g(at-)75 2704 y(tribute)e(w)o(ord;)f(the)g(F)l(ortran)g(function)h Fj(MPI)p 855 2704 V 16 w(A)l(TTR)p 988 2704 V 17 w(PUT)g Fs(will)h(set)f(the)f(least)h(signi\014can)o(t)h(part)e(of)g(the)1967 46 y Fk(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 11 13 11 12 bop 75 -100 a Fg(10.2.)34 b(LANGUA)o(GE)15 b(INTER)o(OPERABILITY) 876 b Fs(11)75 45 y(attribute)16 b(w)o(ord,)f(whic)o(h)i(will)g(b)q(e)g (sign)f(extended)h(to)f(the)g(en)o(tire)g(w)o(ord.)21 b(\(These)16 b(t)o(w)o(o)f(functions)i(ma)o(y)75 102 y(b)q(e)f(in)o(v)o(ok)o(ed)f(explicitly)j(b)o(y)d(user)h(co)q(de,)f(or) g(implicitly)l(,)j(b)o(y)d(attribute)g(cop)o(ying)h(callbac)o(k)g (functions.\))189 208 y Fi(A)n(dvic)n(e)f(to)i(users.)40 b Fs(Users)15 b(are)g(encouraged)g(to)g(use)g(the)g(t)o(w)o(o)f(new)i (functions)189 264 y Fj(MPI)p 274 264 14 2 v 15 w(HANDLE)p 468 264 V 17 w(A)l(TTR)p 602 264 V 17 w(GET)24 b Fs(and)f Fj(MPI)p 909 264 V 16 w(HANDLE)p 1104 264 V 17 w(A)l(TTR)p 1238 264 V 17 w(GET)p Fs(,)g(describ)q(ed)i(in)f(Section)g Fh(??)p Fs(,)189 321 y(page)17 b Fh(??)p Fs(.)26 b(Their)18 b(F)l(ortran)e(binding)k(sp)q(eci\014es)f(address)f(sized)g(in)o(teger) g(argumen)o(ts,)f(th)o(us)g(pro-)189 377 y(viding)f(language)g(in)o (terop)q(erabilit)o(y)l(.)22 b(\()p Fi(End)15 b(of)h(advic)n(e)h(to)f (users.)p Fs(\))1875 438 y Fl(?)g Fk(\(Oct\))75 549 y Fe(10.2.11)50 b(Requests)75 682 y Fo(Curren)o(t)14 b(Status:)j Fn(P)o(assed)e(once.)166 785 y Fs(A)g(v)m(alid)i(request)e(created)g (in)h(on)f(language)g(can)g(b)q(e)h(accessed,)f(up)q(dated,)h(freed,)f (or)f(canceled)j(in)75 842 y(another)e(language.)75 964 y Fe(10.2.12)50 b(Erro)o(r)15 b(handlers)75 1097 y Fo(Curren)o(t)f (Status:)j Fn(P)o(assed)e(once.)166 1200 y Fs(Error)c(handles)i(can)g (b)q(e)f(allo)q(cated)i(and)e(deallo)q(cated)h(in)g(an)o(y)f(com)o (bination)h(of)f(languages.)19 b(When)75 1257 y(an)c(MPI)h(exception)h (o)q(ccurs,)e(then)h(the)g(last)f(error)g(handler)i(asso)q(ciated)e (with)h(the)g(relev)m(an)o(t)g(comm)o(u-)75 1313 y(nicator)h(is)g(in)o (v)o(ok)o(ed,)g(irresp)q(ectiv)o(e)h(of)f(the)f(language)h(en)o (vironmen)o(t)g(where)g(the)g(error)f(o)q(ccurred)i(and)75 1370 y(the)h(en)o(vironmen)o(t)f(where)h(the)g(handler)h(w)o(as)d(asso) q(ciated)i(with)g(the)g(comm)o(unicator.)29 b(The)19 b(handler)75 1426 y(will)e(b)q(e)f(in)o(v)o(ok)o(ed)f(with)h(the)f (righ)o(t)g(argumen)o(t)f(passing)i(con)o(v)o(en)o(tion.)189 1532 y Fi(A)n(dvic)n(e)j(to)h(implementors.)65 b Fs(Error)18 b(handler)j(ob)s(jects)d(need)j(to)d(ha)o(v)o(e)h(a)g(language)h(tag,)f (lik)o(e)189 1589 y(attribute)e(k)o(eys,)h(so)f(as)h(to)f(use)h(the)g (righ)o(t)f(calling)j(con)o(v)o(en)o(tion.)27 b(These)18 b(handlers)h(ha)o(v)o(e,)f(in)g(C)189 1645 y(and)f(C++,)h(a)f(\\)p Fm(stdargs)p Fs(")e(argumen)o(t)i(list.)26 b(It)18 b(migh)o(t)f(b)q(e)g (useful)i(to)d(pro)o(vide)i(to)f(the)g(handler)189 1702 y(information)c(on)h(the)f(language)h(en)o(vironmen)o(t)g(where)f(the)h (error)e(o)q(ccurred.)20 b(\()p Fi(End)14 b(of)h(advic)n(e)g(to)189 1758 y(implementors.)p Fs(\))75 1880 y Fe(10.2.13)50 b(Reduce)16 b(op)q(erations)75 2013 y Fo(Curren)o(t)e(Status:)j Fn(P)o(assed)e(once.)166 2117 y Fs(Reduce)g(op)q(erations)e(de\014ned)i (in)f(one)f(language)h(\(b)o(y)e(a)h(call)i(to)d Fj(MPI)p 1352 2117 V 16 w(OP)p 1430 2117 V 17 w(CREA)l(TE)p Fs(\))i(can)f(b)q(e) h(used)75 2173 y(or)h(deallo)q(cated)h(in)g(an)o(y)f(other)g(language.) 189 2280 y Fi(A)n(dvic)n(e)h(to)h(users.)47 b Fs(Reduce)18 b(op)q(erations)e(receiv)o(e,)h(as)f(one)g(of)g(their)h(argumen)o(t,)e (the)h(datat)o(yp)q(e)189 2336 y(of)d(the)g(op)q(erands.)20 b(Th)o(us,)14 b(one)f(can)h(de\014ne)h(\\p)q(olymorphic")f(reduce)g(op) q(erations)g(that)f(w)o(ork)g(for)189 2392 y(C,)h(C++,)i(and)f(F)l (ortran)f(datat)o(yp)q(es.)19 b(\()p Fi(End)d(of)g(advic)n(e)g(to)h (users.)p Fs(\))189 2499 y Fi(A)n(dvic)n(e)c(to)i(implementors.)39 b Fs(Reduce)15 b(op)q(eration)f(ob)s(jects)f(need)h(to)f(carry)g(a)h (language)f(tag,)g(lik)o(e)189 2555 y(attribute)h(k)o(eys,)g(so)h(as)f (to)g(use)h(the)g(righ)o(t)f(calling)j(con)o(v)o(en)o(tion.)i(\()p Fi(End)d(of)g(advic)n(e)f(to)h(implemen-)189 2612 y(tors.)p Fs(\))-32 46 y Fk(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 12 14 12 13 bop 75 -100 a Fs(12)1115 b Fg(CHAPTER)15 b(10.)30 b(MISCELLANY)75 45 y Fe(10.2.14)50 b(Status)75 178 y Fo(Curren)o(t)14 b(Status:)j Fn(V)m(ote??)166 282 y Fs(MPI)i(do)q(es)g (not)g(pro)o(vide)g(a)g(mec)o(hanism)g(for)g(passing)g(status)f(ob)s (jects)g(from)g(one)h(language)h(to)75 338 y(another.)189 442 y Fi(R)n(ationale.)40 b Fs(The)15 b(MPI)g(status)g(ob)s(ject)f(is)i (not)f(really)h(an)f(opaque)g(ob)s(ject,)f(since)j(its)e(structure)189 499 y(is)k(partially)g(visible)i(to)c(the)i(user,)g(and)g(the)f(ob)s (ject)g(is)h(allo)q(cated)g(b)o(y)g(the)f(user.)30 b(A)19 b(C)f(status)189 555 y(ob)s(ject)12 b(is)h(a)f(structure,)g(a)h(C++)g (status)e(ob)s(ject)h(is)h(a)g(real)f(ob)s(ject,)h(and)f(a)g(F)l (ortran)g(status)g(ob)s(ject)189 612 y(is)j(an)f(arra)o(y)l(.)19 b(Instead)c(of)f(passing)h(status)f(ob)s(jects,)g(one)h(can)f(pass)h (requests,)f(and)h(use)g(them)g(to)189 668 y(buld)h(status)f(ob)s (jects.)k(\()p Fi(End)c(of)i(r)n(ationale.)p Fs(\))75 790 y Fe(10.2.15)50 b(Constants)75 923 y Fo(Curren)o(t)14 b(Status:)j Fn(P)o(assed)e(once.)166 1026 y Fs(MPI)i(constan)o(ts)f(ha) o(v)o(e)g(the)h(same)g(v)m(alue)h(in)g(all)g(languages.)25 b(This)18 b(includes,)h(error)d(co)q(des,)i(con-)75 1083 y(stan)o(ts)12 b(suc)o(h)h(as)g Fm(MPI)p 435 1083 15 2 v 17 w(PROC)p 548 1083 V 16 w(NULL)g Fs(or)f Fm(MPI)p 798 1083 V 17 w(ANY)p 887 1083 V 17 w(TAG)p Fs(,)g(etc.)19 b(This)14 b(do)q(es)f(not)f(apply)i(to)f(constan)o(t)f(handles)75 1139 y(\()p Fd(MPI)p 170 1139 13 2 v 14 w(INT,)17 b(MPI)p 357 1139 V 15 w(COMM)p 502 1139 V 14 w(W)o(ORLD,)h(MPI)p 769 1139 V 14 w(ERRORS)p 943 1139 V 14 w(RETURN,)f(MPI)p 1226 1139 V 14 w(SUM)p Fs(,)i(etc.\))33 b(These)19 b(handles)i(need)75 1196 y(to)15 b(b)q(e)h(con)o(v)o(erted,)e(as)h(explained)j(in)e (Section)g(10.2.4.)i Fd(MPI)p 1109 1196 V 15 w(BOTTOM)d Fs(ma)o(y)f(ha)o(v)o(e)h(di\013eren)o(t)h(v)m(alues)g(in)75 1252 y(F)l(ortran,)e(C,)g(and)i(C++.)166 1356 y Fo(Missing:)h Fn(Need)d(to)g(add)g(the)g(new)h(MPI2)f(sp)q(ecial)g(addresses)75 1525 y Fe(10.2.16)50 b(Interlanguage)17 b(communication)75 1658 y Fo(Curren)o(t)d(Status:)j Fn(P)o(assed)e(once.)166 1761 y Fs(The)i(t)o(yp)q(e)h(matc)o(hing)f(rules)h(for)e(comm)o (unications)i(in)g(MPI)f(are)g(not)g(c)o(hanged:)24 b(The)17 b(datat)o(yp)q(e)75 1818 y(sp)q(eci\014cation)g(for)e(eac)o(h)h(item)f (sen)o(t)h(should)g(matc)o(h,)f(textually)l(,)h(the)f(datat)o(yp)q(e)g (sp)q(eci\014cation)j(used)e(to)75 1874 y(receiv)o(e)g(this)g(item)f (\(unless)h(one)g(of)e(the)i(t)o(yp)q(es)f(is)h Fd(MPI)p 1025 1874 V 14 w(P)m(A)o(CKED)p Fs(\).)c(And)k(the)f(t)o(yp)q(e)g(of)g (a)g(message)g(item)75 1931 y(should)21 b(matc)o(h)e(the)h(t)o(yp)q(e)f (declaration)i(for)e(the)h(corresp)q(onding)g(comm)o(unication)h (bu\013er)f(lo)q(cation,)75 1987 y(unless)c(the)g(t)o(yp)q(e)f(is)h Fd(MPI)p 513 1987 V 14 w(BYTE)e Fs(or)h Fd(MPI)p 781 1987 V 14 w(P)m(A)o(CKED)p Fs(.)e(In)o(terlanguage)i(comm)o(unication)h (is)g(allo)o(w)o(ed,)f(if)h(it)75 2044 y(complies)h(with)e(these)h (rules.)75 2148 y Fh(Example)i(10.10)23 b Fs(In)16 b(the)g(example)g(b) q(elo)o(w,)g(a)f(F)l(ortran)g(arra)o(y)f(is)i(sen)o(t)g(from)f(F)l (ortran)f(and)i(receiv)o(ed)75 2204 y(in)g(C.)75 2308 y Fm(!)24 b(FORTRAN)f(CODE)75 2365 y(REAL)g(R\(5\))75 2421 y(INTEGER)g(TYPE,)g(ADDR,)g(IERR,)g(MYRANK)75 2534 y(!)h(create)f(an)g(absolute)g(datatype)g(for)g(array)h(R)75 2591 y(CALL)f(MPI_ADDRESS\()g(R,)g(ADDR\))75 2647 y(CALL)g (MPI_TYPE_STRUCT\(1,)f(5,)h(ADDR,)g(MPI_REAL,)g(TYPE,)g(IERR\))75 2704 y(CALL)g(MPI_TYPE_COMMIT\()f(TYPE\))1967 46 y Fk(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 13 15 13 14 bop 75 -100 a Fg(10.3.)34 b(NEW)15 b(D)o(A)l(T)l(A)l(TYPE)g (MANIPULA)l(TION)i(FUNCTIONS)574 b Fs(13)75 102 y Fm(CALL)23 b(MPI_COMM_RANK\()f(MPI_COMM_WORLD,)g(MYRANK,)h(IERR\))75 158 y(IF)h(\(MYRANK.EQ.0\))e(THEN)75 214 y(CALL)h(MPI_SEND\()g (MPI_BOTTOM,)f(1,)i(TYPE,)f(1,)h(0,)f(MPI_COMM_WORLD,)f(IERR\))75 271 y(ELSE)75 327 y(CALL)h(MPI_HANDLE2INT\(TYPE,)e(CODE\))75 384 y(CALL)i(C_ROUTINE\(CODE\))75 440 y(END)g(IF)75 610 y(/*)h(C)f(code)h(*/)75 723 y(void)f(C_ROUTINE\(MPI_Fint)f(*fhandle\)) 75 779 y({)75 835 y(void)h(*handle;)75 892 y(MPI_Datatype)f(type;)75 948 y(MPI_Status)g(status;)75 1061 y(type)h(=)h (\(MPI_Datatype\)MPI_Int2hand)o(le\(fhand)o(le\);)75 1174 y(MPI_Recv\()f(MPI_BOTTOM,)f(1,)i(type,)f(0,)g(0,)h (MPI_COMM_WORLD,)e(status\);)75 1231 y(})166 1337 y Fs(MPI)17 b(implemen)o(tors)h(ma)o(y)f(w)o(eak)o(en)g(these)h(t)o(yp)q(e)f(matc)o (hing)h(rules,)g(and)g(allo)o(w)g(messages)e(to)h(b)q(e)75 1393 y(sen)o(t)c(with)g(F)l(ortran)f(t)o(yp)q(es)h(and)g(receiv)o(ed)h (with)f(C)g(t)o(yp)q(es,)g(and)g(vice)h(v)o(ersa,)e(when)h(those)g(t)o (yp)q(es)g(matc)o(h.)75 1450 y(I.e.,)g(if)h(the)f(F)l(ortran)f(t)o(yp)q (e)h Fd(INTEGER)f Fs(is)i(iden)o(tical)h(to)d(the)i(C)e(t)o(yp)q(e)i Fd(int)p Fs(,)f(then)h(an)f(MPI)g(implemen)o(tation)75 1506 y(ma)o(y)22 b(allo)o(w)g(data)g(to)f(b)q(e)i(sen)o(t)g(with)f (datat)o(yp)q(e)g Fd(MPI)p 1035 1506 13 2 v 14 w(INTEGER)g Fs(and)g(b)q(e)h(receiv)o(ed)h(with)e(datat)o(yp)q(e)75 1563 y Fd(MPI)p 152 1563 V 14 w(INT)p Fs(.)15 b(Ho)o(w)o(ev)o(er,)f (suc)o(h)i(co)q(de)f(is)h(not)f(p)q(ortable.)906 b Fl(>)16 b Fk(\(Oct\))75 1706 y Fp(10.3)59 b(New)20 b(datat)n(yp)r(e)d (manipulation)i(functions)75 1807 y Fs(New)e(functions)h(are)e(pro)o (vided)i(to)f(supplemen)o(t)h(the)f(t)o(yp)q(e)g(manipulation)h (functions)g(that)e(ha)o(v)o(e)h(ad-)75 1864 y(dress)d(sized)i(in)o (teger)e(argumen)o(ts.)k(The)d(new)f(functions)h(will)h(use,)e(in)h (their)g(F)l(ortran)e(binding)j(address)75 1920 y(sized)g Fd(INTEGER)p Fs(s,)d(th)o(us)i(solving)g(problems)g(curren)o(tly)g (encoun)o(tered)h(when)f(the)g(application)h(address)75 1977 y(range)h(is)i Ff(>)e Fs(2)325 1960 y Fc(32)362 1977 y Fs(.)27 b(Also,)18 b(a)g(new,)g(more)f(con)o(v)o(enien)o(t)h(t)o (yp)q(e)g(constructor)f(is)h(pro)o(vided)h(to)e(mo)q(dify)h(the)75 2033 y(lo)o(w)o(er)d(b)q(ound)h(and)f(exten)o(t)g(of)g(a)g(datat)o(yp)q (e.)75 2155 y Fe(10.3.1)49 b(T)l(yp)q(e)17 b(constructo)o(rs)d(with)j (explicit)g(addresses)75 2241 y Fs(The)11 b(four)f(functions)h(b)q(elo) o(w)h(supplemen)o(t)f(the)g(four)f(corresp)q(onding)i(t)o(yp)q(e)f (constructor)e(functions)j(from)75 2297 y Fj(MPI-1)p Fs(.)18 b(The)c(new)f(functions)g(are)g(synon)o(ymous)g(with)g(the)g (old)g(functions)h(in)g(C/C++,)f(or)f(on)h(F)l(ortran)75 2354 y(systems)k(where)h(default)h Fd(INTEGER)p Fs(s)e(are)g(address)h (sized.)29 b(In)19 b(F)l(ortran,)d(they)i(accept)g(argumen)o(ts)f(of)75 2410 y(t)o(yp)q(e)e Fd(INTEGER\(KIND=MPI)p 576 2410 V 13 w(ADDRESS)p 775 2410 V 15 w(SIZE\))p Fs(,)f(wherev)o(er)g(argumen)o (ts)g(of)g(t)o(yp)q(e)h Fd(MPI)p 1558 2410 V 14 w(Aint)g Fs(are)f(used)h(in)75 2467 y(C.)i(On)i(F)l(ortran)d(77)i(systems)f (that)g(do)h(not)f(supp)q(ort)h(the)g(F)l(ortran)f(90)g Fd(KIND)h Fs(notation,)g(and)g(where)75 2523 y(addresses)c(are)g(64)f (bits)h(whereas)g(default)h Fd(INTEGER)p Fs(s)e(are)g(32)h(bits,)g (these)g(argumen)o(ts)f(will)j(b)q(e)e(of)g(t)o(yp)q(e)75 2580 y Fd(INTEGER*8)p Fs(.)19 b(The)c(old)g(functions)h(will)h(con)o (tin)o(ue)e(to)g(b)q(e)g(pro)o(vided,)h(for)e(p)q(ortabilit)o(y)l(.)21 b(Ho)o(w)o(ev)o(er,)14 b(users)75 2636 y(are)h(encouraged)g(to)g(switc) o(h)g(to)g(the)g(new)h(functions,)f(in)h(F)l(ortran.)166 2692 y(The)f(new)h(functions)g(are)f(listed)h(b)q(elo)o(w.)-32 46 y Fk(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 14 16 14 15 bop 75 -100 a Fs(14)1110 b Fg(CHAPTER)15 b(10.)35 b(MISCELLANY)75 45 y Fj(MPI)p 160 45 14 2 v 16 w(TYPE)p 293 45 V 17 w(HVECTOR)p 521 45 V 17 w(X\()16 b(count,)g(blo)q (cklength,)g(stride,)g(oldt)o(yp)q(e,)g(newt)o(yp)q(e\))117 122 y Fn(IN)155 b Fj(count)482 b Fn(n)o(um)o(b)q(er)13 b(of)h(blo)q(c)o(ks)g(\(nonnegativ)o(e)f(in)o(teger\))117 197 y(IN)155 b Fj(blo)q(cklength)371 b Fn(n)o(um)o(b)q(er)16 b(of)h(elemen)o(ts)g(in)f(eac)o(h)i(blo)q(c)o(k)e(\(nonnegativ)o(e)h (in)o(te-)905 254 y(ger\))117 329 y(IN)155 b Fj(stride)484 b Fn(n)o(um)o(b)q(er)13 b(of)g(b)o(ytes)h(b)q(et)o(w)o(een)h(start)f (of)f(eac)o(h)h(blo)q(c)o(k)f(\(in)o(teger\))117 404 y(IN)155 b Fj(oldt)o(yp)q(e)450 b Fn(old)13 b(datat)o(yp)q(e)i (\(handle\))117 479 y(OUT)108 b Fj(newt)o(yp)q(e)433 b Fn(new)15 b(datat)o(yp)q(e)f(\(handle\))75 604 y Fm(int)23 b(MPI)p 245 604 15 2 v 17 w(Type)p 358 604 V 17 w(hvector)p 543 604 V 16 w(x\(int)g(count,)g(int)h(blocklength,)e(MPI)p 1347 604 V 17 w(Aint)h(stride,)393 660 y(MPI)p 468 660 V 17 w(Datatype)g(oldtype,)f(MPI)p 986 660 V 17 w(Datatype)h (*newtype\))75 747 y(MPI)p 150 747 V 17 w(TYPE)p 263 747 V 16 w(HVECTOR)p 447 747 V 17 w(X\(COUNT,)f(BLOCKLENGTH,)h(STIDE,)g (OLDTYPE,)f(NEWTYPE,)h(IERROR\))170 803 y(INTEGER)g(COUNT,)g (BLOCKLENGTH,)g(OLDTYPE,)f(NEWTYPE,)h(IERROR)170 860 y(INTEGER\(KIND=MPI)p 557 860 V 15 w(ADDRESS)p 740 860 V 17 w(SIZE\))g(STRIDE)75 1041 y Fj(MPI)p 160 1041 14 2 v 16 w(TYPE)p 293 1041 V 17 w(HINDEXED)p 537 1041 V 16 w(X\()14 b(count,)h(a)o(rra)o(y)p 843 1041 V 14 w(of)p 894 1041 V 16 w(blo)q(cklengths,)h(a)o(rra)o(y)p 1267 1041 V 15 w(of)p 1319 1041 V 16 w(displacements,)f(oldt)o(yp)q(e,)g (new-)75 1097 y(t)o(yp)q(e\))117 1174 y Fn(IN)155 b Fj(count)482 b Fn(n)o(um)o(b)q(er)34 b(of)f(blo)q(c)o(ks)h({)g(also)f(n)o(um)o(b)q (er)h(of)f(en)o(tries)i(in)905 1231 y Fd(a)o(rra)o(y)p 992 1231 13 2 v 15 w(of)p 1041 1231 V 15 w(displacements)19 b Fn(and)f Fd(a)o(rra)o(y)p 1483 1231 V 15 w(of)p 1532 1231 V 14 w(blo)q(cklengths)i Fn(\(in)o(te-)905 1287 y(ger\))117 1362 y(IN)155 b Fj(a)o(rra)o(y)p 416 1362 14 2 v 15 w(of)p 468 1362 V 16 w(blo)q(cklengths)191 b Fn(n)o(um)o(b)q(er)15 b(of)g(elemen)o(ts)h(in)f(eac)o(h)h(blo)q(c)o (k)g(\(arra)o(y)f(of)g(nonnega-)905 1419 y(tiv)o(e)f(in)o(tegers\))117 1494 y(IN)155 b Fj(a)o(rra)o(y)p 416 1494 V 15 w(of)p 468 1494 V 16 w(displacements)164 b Fn(b)o(yte)14 b(displacemen)o(t)g (of)f(eac)o(h)h(blo)q(c)o(k)g(\(arra)o(y)g(of)f(in)o(teger\))117 1569 y(IN)155 b Fj(oldt)o(yp)q(e)450 b Fn(old)13 b(datat)o(yp)q(e)i (\(handle\))117 1644 y(OUT)108 b Fj(newt)o(yp)q(e)433 b Fn(new)15 b(datat)o(yp)q(e)f(\(handle\))75 1769 y Fm(int)23 b(MPI)p 245 1769 15 2 v 17 w(Type)p 358 1769 V 17 w(hindexed)p 567 1769 V 16 w(x\(int)g(count,)g(int)h(*array)p 1133 1769 V 16 w(of)p 1197 1769 V 17 w(blocklengths,)393 1825 y(MPI)p 468 1825 V 17 w(Aint)f(*array)p 748 1825 V 17 w(of)p 813 1825 V 17 w(displacements,)e(MPI)p 1259 1825 V 17 w(Datatype)i(oldtype,)393 1881 y(MPI)p 468 1881 V 17 w(Datatype)g(*newtype\))75 1968 y(MPI)p 150 1968 V 17 w(TYPE)p 263 1968 V 16 w(HINDEXED)p 471 1968 V 16 w(X\(COUNT,)g(ARRAY)p 822 1968 V 17 w(OF)p 887 1968 V 17 w(BLOCKLENGTHS,)f(ARRAY)p 1358 1968 V 16 w(OF)p 1422 1968 V 17 w(DISPLACEMENTS,)393 2024 y(OLDTYPE,)h(NEWTYPE,)g(IERROR\)) 170 2081 y(INTEGER)g(COUNT,)g(ARRAY)p 651 2081 V 17 w(OF)p 716 2081 V 17 w(BLOCKLENGTHS\(*\),)e(OLDTYPE,)i(NEWTYPE,)g(IERROR)170 2137 y(INTEGER\(KIND=MPI)p 557 2137 V 15 w(ADDRESS)p 740 2137 V 17 w(SIZE\))g(ARRAY)p 1020 2137 V 16 w(OF)p 1084 2137 V 17 w(DISPLACEMENTS\(*\))1967 46 y Fk(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 15 17 15 16 bop 75 -100 a Fg(10.3.)34 b(NEW)15 b(D)o(A)l(T)l(A)l(TYPE)g (MANIPULA)l(TION)i(FUNCTIONS)574 b Fs(15)75 45 y Fj(MPI)p 160 45 14 2 v 16 w(TYPE)p 293 45 V 17 w(STRUCT)p 486 45 V 17 w(X\(count,)12 b(a)o(rra)o(y)p 776 45 V 14 w(of)p 827 45 V 16 w(blo)q(cklengths,)h(a)o(rra)o(y)p 1197 45 V 15 w(of)p 1249 45 V 16 w(displacements,)g(a)o(rra)o(y)p 1646 45 V 14 w(of)p 1697 45 V 16 w(t)o(yp)q(es,)f(new-)75 102 y(t)o(yp)q(e\))117 179 y Fn(IN)155 b Fj(count)482 b Fn(n)o(um)o(b)q(er)18 b(of)h(blo)q(c)o(ks)g(\(in)o(teger\))g({)g (also)f(n)o(um)o(b)q(er)h(of)f(en)o(tries)905 235 y(in)c(arra)o(ys)h Fd(a)o(rra)o(y)p 1167 235 13 2 v 15 w(of)p 1216 235 V 14 w(t)o(yp)q(es)p Fn(,)h Fd(a)o(rra)o(y)p 1432 235 V 15 w(of)p 1481 235 V 14 w(displacements)g Fn(and)e Fd(a)o(r-)905 292 y(ra)o(y)p 959 292 V 15 w(of)p 1008 292 V 15 w(blo)q(cklengths)117 367 y Fn(IN)155 b Fj(a)o(rra)o(y)p 416 367 14 2 v 15 w(of)p 468 367 V 16 w(blo)q(cklength)208 b Fn(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(eac)o(h)g(blo)q(c)o(k)g(\(arra)o(y)g(of)f(in) o(teger\))117 442 y(IN)155 b Fj(a)o(rra)o(y)p 416 442 V 15 w(of)p 468 442 V 16 w(displacements)164 b Fn(b)o(yte)14 b(displacemen)o(t)g(of)f(eac)o(h)h(blo)q(c)o(k)g(\(arra)o(y)g(of)f(in)o (teger\))117 517 y(IN)155 b Fj(a)o(rra)o(y)p 416 517 V 15 w(of)p 468 517 V 16 w(t)o(yp)q(es)327 b Fn(t)o(yp)q(e)20 b(of)f(elemen)o(ts)g(in)g(eac)o(h)h(blo)q(c)o(k)f(\(arra)o(y)g(of)g (handles)g(to)905 574 y(datat)o(yp)q(e)14 b(ob)r(jects\))117 649 y(OUT)108 b Fj(newt)o(yp)q(e)433 b Fn(new)15 b(datat)o(yp)q(e)f (\(handle\))75 773 y Fm(int)23 b(MPI)p 245 773 15 2 v 17 w(Type)p 358 773 V 17 w(struc)p 495 773 V 16 w(xt\(int)g(count,)g (int)h(*array)p 1085 773 V 16 w(of)p 1149 773 V 17 w(blocklengths,)393 830 y(MPI)p 468 830 V 17 w(Aint)f(*array)p 748 830 V 17 w(of)p 813 830 V 17 w(displacements,)e(MPI)p 1259 830 V 17 w(Datatype)i(*array)p 1635 830 V 16 w(of)p 1699 830 V 17 w(types,)393 886 y(MPI)p 468 886 V 17 w(Datatype)g(*newtype\)) 75 972 y(MPI)p 150 972 V 17 w(TYPE)p 263 972 V 16 w(STRUCT)p 423 972 V 17 w(X\(COUNT,)g(ARRAY)p 775 972 V 16 w(OF)p 839 972 V 17 w(BLOCKLENGTHS,)f(ARRAY)p 1310 972 V 17 w(OF)p 1375 972 V 16 w(DISPLACEMENTS,)393 1029 y(ARRAY)p 516 1029 V 17 w(OF)p 581 1029 V 17 w(TYPES,)h(NEWTYPE,)f(IERROR\))170 1085 y(INTEGER)h(COUNT,)g(ARRAY)p 651 1085 V 17 w(OF)p 716 1085 V 17 w(BLOCKLENGTHS\(*\),)e(ARRAY)p 1258 1085 V 17 w(OF)p 1323 1085 V 17 w(TYPES\(*\),)i(NEWTYPE,)170 1142 y(IERROR)170 1198 y(INTEGER\(KIND=MPI)p 557 1198 V 15 w(ADDRESS)p 740 1198 V 17 w(SIZE\))g(ARRAY)p 1020 1198 V 16 w(OF)p 1084 1198 V 17 w(DISPLACEMENTS\(*\),)75 1379 y Fj(MPI)p 160 1379 14 2 v 16 w(ADDRESS)p 378 1379 V 17 w(X\(lo)q(cation,)15 b(address\))117 1456 y Fn(IN)155 b Fj(lo)q(cation)437 b Fn(lo)q(cation)13 b(in)h(caller)f(memory)f(\(c)o (hoice\))117 1532 y(OUT)108 b Fj(address)449 b Fn(address)15 b(of)f(lo)q(cation)f(\(in)o(teger\))75 1656 y Fm(int)23 b(MPI)p 245 1656 15 2 v 17 w(Address)p 430 1656 V 16 w(x\(void*)g(location,)g(MPI)p 948 1656 V 17 w(Aint)g(*address\))75 1743 y(MPI)p 150 1743 V 17 w(ADDRESS)p 335 1743 V 16 w(X\(LOCATION,)f(ADDRESS,)h(IERROR\))170 1799 y()g(LOCATION\(*\)) 170 1855 y(INTEGER)g(IERROR)170 1912 y(INTEGER\(KIND=MPI)p 557 1912 V 15 w(ADDRESS)p 740 1912 V 17 w(SIZE\))g(ADDRESS)189 2048 y Fi(A)n(dvic)n(e)d(to)h(users.)72 b Fs(Curren)o(t)20 b(F)l(ortran)g(MPI)g(co)q(des)h(will)h(run)f(unmo)q(di\014ed,)i(and)e (will)h(p)q(ort)189 2105 y(to)f(an)o(y)g(system.)39 b(Ho)o(w)o(ev)o (er,)22 b(they)g(ma)o(y)f(fail)h(if)g(addresses)g(larger)g(than)f(2) 1558 2088 y Fc(32)1610 2105 y Fl(\000)15 b Fs(1)21 b(are)g(used)189 2161 y(in)f(the)f(program.)31 b(New)19 b(co)q(des)h(should)g(b)q(e)g (written)f(so)g(that)g(they)g(use)h(the)f(new)h(functions.)189 2218 y(This)e(pro)o(vides)h(compatibilit)o(y)g(with)f(C/C++)g(and)h(a)o (v)o(oids)e(errors)g(on)h(64)g(bit)g(arc)o(hitectures.)189 2274 y(Ho)o(w)o(ev)o(er,)13 b(suc)o(h)i(newly)g(written)g(co)q(des)g (ma)o(y)e(need)j(to)e(b)q(e)h(\(sligh)o(tly\))g(rewritten)f(to)g(p)q (ort)h(to)e(old)189 2330 y(F)l(ortran)k(77)g(en)o(vironmen)o(ts)i(that) f(do)g(not)g(supp)q(ort)g Fd(KIND)g Fs(declarations.)30 b(\()p Fi(End)18 b(of)i(advic)n(e)f(to)189 2387 y(users.)p Fs(\))1875 2448 y Fl(?)d Fk(\(Oct\))75 2558 y Fe(10.3.2)49 b(Extent)16 b(and)h(Bounds)f(of)h(Datat)o(yp)q(es)75 2644 y Fs(The)d(follo)o(wing)h(function)g(supplemen)o(ts)h(the)e(three) g(functions)h Fj(MPI)p 1266 2644 14 2 v 16 w(TYPE)p 1399 2644 V 17 w(UB,)f(MPI)p 1586 2644 V 16 w(TYPE)p 1719 2644 V 17 w(LB)g Fs(and)75 2701 y Fj(MPI)p 160 2701 V 16 w(TYPE)p 293 2701 V 17 w(EXTENT)p Fs(.)h(It)g(also)g(returns)h (address)f(sized)h(in)o(tegers,)f(in)h(the)f(F)l(ortran)f(binding.)-32 46 y Fk(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 16 18 16 17 bop 75 -100 a Fs(16)1110 b Fg(CHAPTER)15 b(10.)35 b(MISCELLANY)75 45 y Fj(MPI)p 160 45 14 2 v 16 w(GET)p 264 45 V 17 w(EXTENT)p 459 45 V 17 w(X\(datat)o(yp)q(e,)16 b(lb,)f(extent\))117 122 y Fn(IN)155 b Fj(datat)o(yp)q(e)424 b Fn(datat)o(yp)q(e)14 b(to)g(get)g(information)d(on)117 197 y(OUT)108 b Fj(lb)553 b Fn(lo)o(w)o(er)14 b(b)q(ound)g(of)f(datat)o (yp)q(e)117 273 y(OUT)108 b Fj(extent)471 b Fn(exten)o(t)15 b(of)e(datat)o(yp)q(e)75 397 y Fm(int)23 b(MPI)p 245 397 15 2 v 17 w(Get)p 334 397 V 17 w(extent)p 495 397 V 16 w(x\(MPI)p 631 397 V 17 w(Datatype)g(datatype,)f(MPI)p 1173 397 V 17 w(Aint)h(*lb,)h(MPI)p 1501 397 V 16 w(Aint)g(*extent\))75 483 y(MPI)p 150 483 V 17 w(GET)p 239 483 V 17 w(EXTENT)p 400 483 V 16 w(X\(DATATYPE,)e(LB,)i(EXTENT,)f(IERROR\))170 540 y(INTEGER)g(DATATYPE,)g(IERROR)170 596 y(INTEGER\(KIND)g(=)g(MPI)p 603 596 V 17 w(ADDRESS)p 788 596 V 16 w(SIZE\))h(LB,)f(EXTENT)-117 633 y Fl(>)15 b Fk(\(Oct\))75 683 y Fm(int)23 b(MPI::Datatype::Get)p 605 683 V 15 w(extent)p 764 683 V 17 w(x\(MPI::Aint&)f(lb,)h (MPI::Aint&)g(extent\))g(const)166 769 y Fs(Returns)16 b(the)f(lo)o(w)o(er)g(b)q(ound)h(and)f(the)g(exten)o(t)g(of)g Fj(datat)o(yp)q(e)h Fs(\(as)f(de\014ned)h(b)o(y)f(the)h(MPI)f (standard,)75 826 y(Section)h(3.12.2\).)166 882 y(MPI)21 b(allo)o(ws)h(one)f(to)g(c)o(hange)h(the)f(exten)o(t)g(of)g(a)h(datat)o (yp)q(e,)g(using)g(lo)o(w)o(er)f(b)q(ound)h(and)g(upp)q(er)75 939 y(b)q(ound)e(mark)o(ers)e(\()p Fd(MPI)p 490 939 13 2 v 14 w(LB)h Fs(and)g Fd(MPI)p 740 939 V 14 w(UB)p Fs(\).)f(This)h(is) h(useful,)g(as)f(it)g(allo)o(ws)g(to)f(con)o(trol)h(the)g(stride)g(of) 75 995 y(successiv)o(e)e(datat)o(yp)q(es)e(that)g(are)g(replicated)i(b) o(y)e(datat)o(yp)q(e)g(constructors,)g(or)g(are)g(replicated)i(b)o(y)f (the)75 1052 y Fj(count)k Fs(argumen)o(t)e(in)i(a)e(send)h(or)g(reciev) o(e)g(call.)32 b(Ho)o(w)o(ev)o(er,)19 b(the)f(curren)o(t)h(mec)o (hanism)h(for)e(ac)o(hieving)75 1108 y(it)h(is)h(painful;)i(also)d(it)h (is)f(restrictiv)o(e,)i(as)d Fd(MPI)p 906 1108 V 15 w(LB)h Fs(and)g Fd(MPI)p 1157 1108 V 14 w(UB)g Fs(are)f(\\stic)o(ky":)28 b(once)19 b(presen)o(t)g(in)h(a)75 1165 y(datat)o(yp)q(e,)d(they)h (cannot)g(b)q(e)g(o)o(v)o(eriden)g(\(e.g.,)f(the)h(upp)q(er)h(b)q(ound) f(can)g(b)q(e)g(mo)o(v)o(ed)g(up,)g(b)o(y)g(adding)g(a)75 1221 y(new)13 b Fd(MPI)p 243 1221 V 14 w(UB)g Fs(mark)o(er,)f(but)h (cannot)g(b)q(e)g(mo)o(v)o(ed)g(do)o(wn)f(b)q(elo)o(w)i(an)f(existing)h Fj(MPI)p 1474 1221 14 2 v 15 w(UB)g Fs(mark)o(er\).)k(A)13 b(new)-1992 b Fl(>)15 b Fk(\(Oct\))75 1277 y Fs(t)o(yp)q(e)g (constructor)g(is)g(pro)o(vided)h(to)f(facilitate)h(these)f(c)o (hanges.)75 1428 y Fj(MPI)p 160 1428 V 16 w(TYPE)p 293 1428 V 17 w(RESIZED)p 492 1428 V 16 w(X\(oldt)o(yp)q(e,)h(lb,)f (extent,)i(newt)o(yp)q(e\))117 1506 y Fn(IN)155 b Fj(oldt)o(yp)q(e)450 b Fn(input)14 b(datat)o(yp)q(e)g(\(handle\))117 1581 y(IN)155 b Fj(lb)553 b Fn(new)15 b(lo)o(w)o(er)e(b)q(ound)h(of)f(datat) o(yp)q(e)h(\(in)o(teger\))117 1656 y(IN)155 b Fj(extent)471 b Fn(new)15 b(exten)o(t)f(of)g(datat)o(yp)q(e)g(\(in)o(teger\))117 1731 y(OUT)108 b Fj(newt)o(yp)q(e)433 b Fn(output)14 b(datat)o(yp)q(e)g(\(handle\))75 1855 y Fm(int)23 b(MPI)p 245 1855 15 2 v 17 w(Type)p 358 1855 V 17 w(resized)p 543 1855 V 16 w(x\(MPI)p 679 1855 V 17 w(Datatype)f(oldtype,)h(MPI)p 1197 1855 V 17 w(Aint)g(lb,)h(MPI)p 1501 1855 V 16 w(Aint)g(extent,)393 1912 y(MPI)p 468 1912 V 17 w(Datatype)f(*newtype\))75 1998 y(MPI)p 150 1998 V 17 w(TYPE)p 263 1998 V 16 w(RESIZED)p 447 1998 V 17 w(X\(OLDTYPE,)f(LB,)i(EXTENT,)e(NEWTYPE,)h(IERROR\))170 2055 y(INTEGER)g(DATATYPE,)g(NEWTYPE,)g(IERROR)170 2111 y(INTEGER\(KIND=MPI)p 557 2111 V 15 w(ADDRESS)p 740 2111 V 17 w(SIZE\))g(LB,)g(EXTENT)75 2198 y(int)g(MPI::Datatype::Resized)p 701 2198 V 15 w(x\(const)g(MPI::Datatype&)e(type,)j(const)f(MPI::Aint&) 393 2254 y(lb,)h(const)f(MPI::Aint&)f(extent\))-117 2290 y Fl(?)15 b Fk(\(Oct\))166 2341 y Fs(Returns)f(in)g Fj(newt)o(yp)q(e)i Fs(a)d(handle)i(to)e(a)g(new)h(datat)o(yp)q(e)f(that)g(is)h(iden)o (tical)h(to)e Fj(oldt)o(yp)q(e)p Fs(,)i(except)f(that)75 2397 y(the)k(lo)o(w)o(er)f(b)q(ound)i(of)e(this)h(new)g(datat)o(yp)q(e) g(is)g(set)f(to)g(b)q(e)i Fj(lb)p Fs(,)f(and)g(its)g(upp)q(er)h(b)q (ound)f(is)h(set)e(to)g(b)q(e)i Fj(lb)75 2454 y Fs(+)h Fj(extent)p Fs(.)34 b(This)20 b(is)g(as)f(if)h(an)o(y)f(previous)h Fh(lb)g Fs(and)g Fh(ub)f Fs(mark)o(ers)g(w)o(ere)g(erased,)h(and)g(a)f (new)h(pair)f(of)-1992 b Fl(>)15 b Fk(\(Oct\))75 2510 y Fs(lo)o(w)o(er)i(b)q(ound)i(and)f(upp)q(er)h(b)q(ound)f(mark)o(ers)f (w)o(ere)h(put)g(in)g(the)g(p)q(ositions)h(indicated)g(b)o(y)f(the)g Fj(lb)g Fs(and)75 2566 y Fj(extent)g Fs(argumen)o(ts.)24 b(This)17 b(a\013ects)f(the)h(b)q(eha)o(vior)h(of)e(the)h(datat)o(yp)q (e)f(when)h(used)h(in)f(comm)o(unication)75 2623 y(op)q(erations,)e (with)h Fj(count)g Ff(>)d Fs(1,)h(and)i(when)g(used)f(in)h(the)g (construction)f(of)g(new)g(deriv)o(ed)i(datat)o(yp)q(es.)1967 46 y Fk(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 17 19 17 18 bop 75 -100 a Fg(10.3.)34 b(NEW)15 b(D)o(A)l(T)l(A)l(TYPE)g (MANIPULA)l(TION)i(FUNCTIONS)574 b Fs(17)189 45 y Fi(A)n(dvic)n(e)14 b(to)i(users.)40 b Fs(It)14 b(is)h(strongly)f(recommended)i(that)e (users)g(use)h(these)g(t)o(w)o(o)e(new)i(functions,)189 102 y(rah)o(ter)h(than)h(the)g(old)g(MPI)h(functions)f(to)g(set)g(and)g (access)g(lo)o(w)o(er)g(b)q(ound,)h(upp)q(er)g(b)q(ound)g(and)189 158 y(exten)o(t)d(of)f(datat)o(yp)q(es.)20 b(\()p Fi(End)15 b(of)i(advic)n(e)f(to)g(users.)p Fs(\))1875 211 y Fl(>)g Fk(\(Oct\))75 320 y Fe(10.3.3)49 b(T)l(rue)16 b(Extent)g(of)g(Datat)o (yp)q(es)75 406 y Fs(Supp)q(ose)22 b(w)o(e)f(implemen)o(t)h(gather)f (as)f(a)h(spanning)h(tree)f(implemen)o(ted)i(on)e(top)g(of)f(p)q(oin)o (t-to-p)q(oin)o(t)75 462 y(routines.)f(Since)11 b(the)g(recv)f (bu\013er)g(is)h(only)g(v)m(alid)h(on)e(the)h(ro)q(ot)e(pro)q(cess,)i (one)g(will)g(need)h(to)d(allo)q(cate)i(some)75 519 y(temp)q(orary)i (space)i(for)e(receiving)j(data)d(on)h(in)o(termediate)h(no)q(des.)20 b(The)14 b(di\016cultly)i(is)f(in)f(determining)75 575 y(the)21 b(size)h(one)f(needs)h(to)f(allo)q(cate.)38 b(This)21 b(o)q(ccurs)h(since)g(the)f(user)g(can)h(mo)q(dify)f(the)g (exten)o(t)g(using)75 632 y(the)h Fd(MPI)p 237 632 13 2 v 14 w(UB)g Fs(and)g Fd(MPI)p 500 632 V 14 w(LB)g Fs(v)m(alues.)42 b(The)22 b(writer)g(of)f(the)i(gather)e(routine)h(could)h(determine)h (this)75 688 y(information)16 b(b)o(y)g(deco)q(ding)i(the)e(datat)o(yp) q(e.)21 b(Ho)o(w)o(ev)o(er,)15 b(this)i(is)f(more)g(w)o(ork)f(and)h (more)g(painful)h(than)75 745 y(desired.)k(Th)o(us,)15 b(a)g(new)g(function)h(is)g(pro)o(vided)g(whic)o(h)g(returns)f(the)g (true)g(exten)o(t)g(of)g(the)g(datat)o(yp)q(e.)75 895 y Fj(MPI)p 160 895 14 2 v 16 w(TRUE)p 294 895 V 17 w(EXTENT)p 489 895 V 17 w(X\(datat)o(yp)q(e,)h(true)p 821 895 V 17 w(lb,)f(true)p 975 895 V 17 w(extent\))117 973 y Fn(IN)155 b Fj(datat)o(yp)q(e)424 b Fn(datat)o(yp)q(e)14 b(to)g(get)g (information)d(on)117 1047 y(OUT)108 b Fj(true)p 396 1047 V 17 w(lb)461 b Fn(true)15 b(lo)o(w)o(er)e(b)q(ound)h(of)g(datat)o (yp)q(e)117 1121 y(OUT)108 b Fj(true)p 396 1121 V 17 w(extent)379 b Fn(true)15 b(size)g(of)e(datat)o(yp)q(e)75 1245 y Fm(int)23 b(MPI)p 245 1245 15 2 v 17 w(True)p 358 1245 V 17 w(extent)p 519 1245 V 16 w(x\(MPI)p 655 1245 V 17 w(Datatype)f(datatype,)h(MPI)p 1197 1245 V 17 w(Aint)g(*true)p 1453 1245 V 17 w(lb,)g(MPI)p 1637 1245 V 17 w(Aint)393 1301 y(*true)p 516 1301 V 17 w(extent\))75 1388 y(MPI)p 150 1388 V 17 w(TRUE)p 263 1388 V 16 w(EXTENT)p 423 1388 V 17 w(X\(DATATYPE,)f(TRUE)p 822 1388 V 17 w(LB,)h(TRUE)p 1030 1388 V 17 w(EXTENT,)g(IERROR\))170 1444 y(INTEGER)g(DATATYPE,)g (IERROR)170 1501 y(INTGER\(KIND)g(=)h(MPI)p 580 1501 V 16 w(ADDRESS)p 764 1501 V 17 w(SIZE\))f(TRUE)p 1020 1501 V 16 w(LB,)h(TRUE)p 1228 1501 V 16 w(EXTENT)1875 1537 y Fl(>)16 b Fk(\(Oct\))75 1587 y Fm(int)23 b(MPI::Datatype::True)p 629 1587 V 15 w(extent)p 788 1587 V 16 w(x\(MPI::Aint&)g(true)p 1211 1587 V 16 w(lb,)h(MPI::Aint&)e(true)p 1681 1587 V 17 w(extent\))393 1644 y(const)166 1730 y Fj(true)p 244 1730 14 2 v 17 w(lb)14 b Fs(returns)f(the)g(o\013set)g(of)g(the)g (lo)o(w)o(est)g(unit)h(of)f(store)g(whic)o(h)h(is)g(addressed)g(b)o(y)f (the)h(datat)o(yp)q(e,)75 1787 y(i.e.,)j(the)g(lo)o(w)o(er)f(b)q(ound)i (of)f(the)g(corresp)q(onding)h(t)o(yp)q(emap,)e(ignoring)i Fd(MPI)p 1391 1787 13 2 v 14 w(LB)f Fs(mark)o(ers.)24 b Fj(true)p 1743 1787 14 2 v 17 w(extent)75 1843 y Fs(returns)12 b(the)h(true)f(size)h(of)f(the)g(datat)o(yp)q(e,)g(i.e.,)h(the)f(exten) o(t)g(of)g(the)h(corresp)q(onding)g(t)o(yp)q(emap,)f(ignoring)75 1900 y Fd(MPI)p 152 1900 13 2 v 14 w(LB)17 b Fs(and)h Fj(MPI)p 407 1900 14 2 v 16 w(UB)f Fs(mark)o(ers,)f(and)i(p)q (erforming)f(no)g(rounding)h(for)e(alignmen)o(t.)27 b(If)17 b(the)g(t)o(yp)q(emap)75 1956 y(asso)q(ciated)e(with)h Fj(datat)o(yp)q(e)h Fs(is)189 2044 y Ff(T)6 b(y)r(pemap)12 b Fs(=)h Fl(f)p Fs(\()p Ff(ty)r(pe)562 2051 y Fc(0)581 2044 y Ff(;)8 b(disp)686 2051 y Fc(0)705 2044 y Fs(\))p Ff(;)g(:)g(:)g(:)t(;)g Fs(\()p Ff(ty)r(pe)926 2051 y Fb(n)p Fa(\000)p Fc(1)994 2044 y Ff(;)g(disp)1099 2051 y Fb(n)p Fa(\000)p Fc(1)1166 2044 y Fs(\))p Fl(g)75 2132 y Fs(Then)189 2219 y Ff(tr)q(ue)p 277 2219 V 16 w(l)q(b)p Fs(\()p Ff(T)e(y)r(pemap)p Fs(\))11 b(=)i Ff(min)691 2226 y Fb(j)709 2219 y Fl(f)p Ff(disp)816 2226 y Fb(j)861 2219 y Fs(:)28 b Ff(ty)r(pe)986 2226 y Fb(j)1017 2219 y Fl(6)p Fs(=)13 b Fh(lb)p Fl(g)p Ff(;)189 2317 y(tr)q(ue)p 277 2317 V 16 w(ub)p Fs(\()p Ff(T)6 b(y)r(pemap)p Fs(\))12 b(=)g Ff(max)709 2324 y Fb(j)728 2317 y Fl(f)p Ff(disp)835 2324 y Fb(j)862 2317 y Fs(+)f Ff(siz)r(eof)5 b Fs(\()p Ff(ty)r(pe)1140 2324 y Fb(j)1158 2317 y Fs(\))28 b(:)f Ff(ty)r(pe)1328 2324 y Fb(j)1360 2317 y Fl(6)p Fs(=)13 b Fh(ub)p Fl(g)p Ff(;)75 2415 y Fs(and)189 2503 y Ff(tr)q(ue)p 277 2503 V 16 w(extent)p Fs(\()p Ff(T)6 b(y)r(pemap)p Fs(\))13 b(=)g Ff(tr)q(ue)p 790 2503 V 17 w(ub)p Fs(\()p Ff(T)6 b(y)r(pemap)p Fs(\))j Fl(\000)h Ff(tr)q(ue)p 1216 2503 V 17 w(l)q(b)p Fs(\()p Ff(ty)r(pemap)p Fs(\))p Ff(:)75 2591 y Fs(\(Compares)k(this)i(with)f(the)h(de\014nitions)h(in)f (Section)g(3.12.3)d(of)i(the)g(MPI)g(standard.\))166 2647 y(The)25 b Fj(true)p 347 2647 V 17 w(extent)h Fs(is)e(the)h(minim) o(um)g(n)o(um)o(b)q(er)g(of)f(b)o(ytes)g(of)g(memory)g(necessary)g(to)g (hold)h(a)75 2704 y(datat)o(yp)q(e)15 b(\(uncompressed\).)1285 b Fl(?)16 b Fk(\(Oct\))-32 46 y(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 18 20 18 19 bop 75 -100 a Fs(18)1110 b Fg(CHAPTER)15 b(10.)35 b(MISCELLANY)75 45 y Fp(10.4)59 b(Continuable)19 b(Erro)n(rs)75 147 y Fs(T)o(ypically)l(,)j(MPI)d(sa)o(ys)g(nothing)h(ab)q(out)g(the)f (in)o(ternal)h(state)f(of)g(itself)i(after)d(an)i(error)f(handler)h (has)75 203 y(b)q(een)14 b(in)o(v)o(ok)o(ed.)20 b(There)13 b(are)g(a)g(n)o(um)o(b)q(er)g(of)g(error)f(classes)i(within)g(MPI)f (that)g(should)h(ha)o(v)o(e)f(no)g(negativ)o(e)75 259 y(impact)19 b(on)f(the)g(in)o(ternal)h(state)f(of)g(MPI.)g(That)g(is)g (to)g(sa)o(y)l(,)g(if)h(one)g(of)e(the)i("con)o(tin)o(uable")g(errors)e (is)75 316 y(detected)g(and)h(no)e(other)h(external)g(ev)o(en)o(ts)g (cause)g(degradation,)g(it)g(is)g(safe)g(to)f(con)o(tin)o(ue)h(using)h (MPI)75 372 y(within)e(an)g(application.)166 429 y(The)f("con)o(tin)o (uable")h(error)f(classes)g(are:)75 535 y Fm(MPI_ERR_BUFFER)237 b(/*)23 b(invalid)g(buffer)g(pointer)g(*/)75 592 y(MPI_ERR_COUNT)261 b(/*)23 b(invalid)g(count)g(argument)g(*/)75 648 y(MPI_ERR_TYPE)285 b(/*)23 b(invalid)g(datatype)g(argument)g(*/)75 704 y(MPI_ERR_TAG)309 b(/*)23 b(invalid)g(tag)h(argument)e(*/)75 761 y(MPI_ERR_COMM)285 b(/*)23 b(invalid)g(communicator)f(*/)75 817 y(MPI_ERR_KEYVAL)237 b(/*)23 b(invalid)g(key)h(value)f(*/)75 874 y(MPI_ERR_ROOT)285 b(/*)23 b(invalid)g(root)h(*/)75 930 y(MPI_ERR_RANK)285 b(/*)23 b(invalid)g(rank)h(*/)75 987 y(MPI_ERR_GROUP)261 b(/*)23 b(invalid)g(group)g(*/)75 1043 y(MPI_ERR_TOPOLOGY)189 b(/*)23 b(invalid)g(topology)g(*/)75 1100 y(MPI_ERR_DIMS)285 b(/*)23 b(invalid)g(dimension)g(argument)g(*/)75 1156 y(MPI_ERR_ARG)309 b(/*)23 b(invalid)g(argument)g(*/)75 1262 y Fs(The)13 b(follo)o(wing)h(errors)f(pro)o(vide)g(no)g(guaran)o (tees,)g(although)g(implemen)o(tors)h(are)f(encouraged)g(to)f(mak)o(e) 75 1319 y(errors)i("con)o(tin)o(uable")i(whenev)o(er)g(p)q(ossible:)75 1413 y Fm(MPI_ERR_OP)333 b(/*)23 b(invalid)g(operation)g(*/)75 1469 y(MPI_ERR_REQUEST)213 b(/*)23 b(invalid)g(request)g(handle)g(*/)75 1526 y(MPI_ERR_UNKNOWN)213 b(/*)23 b(unknown)g(error)g(*/)75 1582 y(MPI_ERR_TRUNCATE)189 b(/*)23 b(message)g(truncated)g(on)g (receive)g(*/)75 1638 y(MPI_ERR_OTHER)261 b(/*)23 b(Other)g(error)h(*/) 75 1695 y(MPI_ERR_INTERN)237 b(/*)23 b(internal)g(MPI)h(error)f(*/)75 1751 y(MPI_ERR_IN_STATUS)165 b(/*)23 b(error)g(code)h(is)f(in)h(status) f(*/)75 1808 y(MPI_ERR_PENDING)213 b(/*)23 b(pending)g(request)g(*/)75 1864 y(MPI_ERR_SYSRESOURCE)117 b(/*)23 b(out)h(of)f(system)g(resources) g(*/)h(/*)f(where)g(in)h(MPI?)f(*/)166 1958 y Fs(W)l(e)12 b(also)f(pro)o(vide)i(the)e(function)i(to)e(indicate)i(whether)f(MPI)g (b)q(eliev)o(es)h(an)f(error)f(is)h("con)o(tin)o(uable")75 2015 y(or)j(not:)75 2166 y Fj(MPI)p 160 2166 14 2 v 16 w(ERR)p 261 2166 V 17 w(IS)p 316 2166 V 16 w(CONTINUABLE)h(\(erro)o(r,) d(\015ag\))117 2243 y Fn(IN)155 b Fj(Erro)o(r)490 b Fn(MPI)14 b(Error)h(Status)f(\(actual)g(status)h(or)f(a)f(class\))117 2318 y(OUT)108 b Fj(Flag)505 b Fn(Bo)q(olean)13 b(indication)f(of)h (whether)h(the)g(implemen)o(tation)c(al-)905 2374 y(lo)o(ws)19 b(safe)h(con)o(tin)o(ued)g(op)q(eration)g(in)f(the)i(ev)o(en)o(t)f(of)f (Error)905 2431 y(b)q(eing)14 b(encoun)o(tered)75 2555 y Fm(int)23 b(MPI)p 245 2555 15 2 v 17 w(Err)p 334 2555 V 17 w(is)p 399 2555 V 17 w(continuable\()f(int)i(error,)f(int)g(*flag) g(\))75 2642 y(MPI)p 150 2642 V 17 w(ERR)p 239 2642 V 17 w(IS)p 304 2642 V 16 w(CONTINUABLE\()g(ERROR,)g(FLAG,)g(IERROR)g(\)) 170 2698 y(INTEGER)g(ERROR,)g(IERROR)1967 46 y Fk(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 19 21 19 20 bop 75 -100 a Fg(10.5.)34 b(NEW)15 b(ELEMENT)l(AR)l(Y)h(D)o(A)l (T)l(A)l(TYPES)883 b Fs(19)170 45 y Fm(LOGICAL)23 b(FLAG)75 132 y(static)g(int)g(MPI::Err)p 532 132 15 2 v 17 w(is)p 597 132 V 16 w(continuable\()g(int)g(error,)g(bool&)g(flag)h(\))1875 172 y Fl(>)16 b Fk(\(Oct\))166 218 y Fs(There)c(is)h(a)f(new,)h(con)o (tin)o(uable)g(error)f(class,)h Fj(MPI)p 1026 218 14 2 v 16 w(ERR)p 1127 218 V 17 w(W)o(OULDBLOCK)p Fs(.)f(This)g(error)g (class)h(ma)o(y)75 274 y(b)q(e)j(used)g(b)o(y)g(a)f(non-blo)q(c)o(king) i(function)f(to)f(inform)h(the)f(caller)i(that)e(the)g(op)q(eration)h (cannot)f(pro)q(ceed)75 331 y(without)g(blo)q(c)o(king.)22 b(When)15 b(this)h(error)e(is)i(returned,)f(the)h(state)e(of)h(the)g (application)i(is)f(equiv)m(alen)o(t)h(to)75 387 y(the)e(one)h(prior)f (to)g(the)g(function)h(call.)166 444 y(An)d(implemen)o(tation)h(is)f (not)g(required)g(to)f(use)i(this)f(error)f(class,)h(but)g(if)g(it)g (do)q(es,)g(it)g(m)o(ust)f(pro)o(vide)75 500 y(the)j(functionalit)o(y)i (describ)q(ed.)166 604 y Fo(Discussion:)33 b Fn(What)13 b(do)q(es)h(it)f(mean)f(when)i(a)f(con)o(tin)o(uable)f(error)j(o)q (ccurs)f(in)f(a)g(collectiv)o(e)g(op)q(eration?)166 708 y Fs(There)k(is)g(a)f(prede\014ned)i(error)e(handler)h Fj(MPI)p 969 708 V 16 w(CONT)p 1110 708 V 17 w(ERRORS)p 1299 708 V 18 w(RETURN)p Fs(,)h(whic)o(h)f(allo)o(ws)g(con-)75 764 y(tin)o(uable)e(erorrs)e(to)f(return)i(error)f(co)q(des)h(and)f (non-con)o(tin)o(uable)j(errors)c(to)h(ab)q(ort.)19 b(The)14 b(default)g(error)75 821 y(handler)i(is)g Fj(MPI)p 370 821 V 16 w(ERRORS)p 558 821 V 18 w(ARE)p 662 821 V 17 w(F)l(A)l(T)l(AL)p Fs(.)75 927 y Fh(Example)i(10.11)118 b Fm(MPI_Errhandler_set\(MPI_COMM)o(_WORLD,)20 b (MPI_CONT_ERRORS_RETURN\);)170 983 y(err)k(=)g(MPI_Isend\()e(...)i(,)f (&req\);)170 1040 y(if)h(\()g(err)f(!=)h(MPI_SUCCESS\))e({)266 1096 y(MPI_Error_class\(err,&clas)o(s\);)266 1153 y(if)h(\(class)h(==)f (MPI_ERR_WOULDBLOCK\))e({)361 1209 y(MPI_Send\()i(...)g(\);)266 1266 y(else)361 1322 y(...)266 1379 y(})170 1435 y(})1875 1489 y Fl(?)16 b Fk(\(Oct\))75 1628 y Fp(10.5)59 b(New)20 b(Elementa)n(ry)e(Datat)n(yp)r(es)75 1777 y Fo(Discussion:)34 b Fn(Do)13 b(w)o(e)h(need)h Fd(MPI)p 644 1777 13 2 v 14 w(W)o(CHAR)p Fn(,)e(to)h(tak)o(e)g(care)h(of)e(Unico)q(de?)-32 46 y Fk(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From mpi-comm-human@mcs.anl.gov Fri Sep 20 06:08:35 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id GAA25799; Fri, 20 Sep 1996 06:08:26 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id XAA28152 for mpi-comm-out; Thu, 19 Sep 1996 23:26:32 -0500 Received: from mcs.anl.gov (donner.mcs.anl.gov [140.221.5.134]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id XAA28135; Thu, 19 Sep 1996 23:26:02 -0500 Message-Id: <199609200426.XAA28135@antares.mcs.anl.gov> To: mpi-comm@antares.mcs.anl.gov, mpi-core@antares.mcs.anl.gov Subject: Misc-1.2 Date: Thu, 19 Sep 1996 23:26:00 -0500 From: Rusty Lusk Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk %!PS-Adobe-2.0 %%Creator: dvips 5.528 Copyright 1986, 1994 Radical Eye Software %%Title: temp.dvi %%CreationDate: Thu Sep 19 16:46:18 1996 %%Pages: 23 %%PageOrder: Ascend %%BoundingBox: 0 0 612 792 %%EndComments %DVIPSCommandLine: dvips -o temp.ps temp %DVIPSParameters: dpi=300, comments removed %DVIPSSource: TeX output 1996.09.19:1646 %%BeginProcSet: tex.pro /TeXDict 250 dict def TeXDict begin /N{def}def /B{bind def}N /S{exch}N /X{S N}B /TR{translate}N /isls false N /vsize 11 72 mul N /hsize 8.5 72 mul N /landplus90{false}def /@rigin{isls{[0 landplus90{1 -1}{-1 1} ifelse 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale isls{landplus90{VResolution 72 div vsize mul 0 exch}{Resolution -72 div hsize mul 0}ifelse TR}if Resolution VResolution vsize -72 div 1 add mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@landscape{/isls true N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder}N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail}B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{ pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get}B /ch-xoff{128 ch-data dup length 3 sub get sub}B /ch-yoff{ch-data dup length 2 sub get 127 sub}B /ch-dx{ch-data dup length 1 sub get}B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ ch-image}imagemask restore}B /D{/cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D }B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto /V matrix currentmatrix dup 1 get dup mul exch 0 get dup mul add .99 lt{/QV}{/RV}ifelse load def pop pop}N /eop{SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if pop /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put}for 65781.76 div /vsize X 65781.76 div /hsize X}N /p{show}N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V {}B /RV statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval dup(Display)eq exch 0 4 getinterval(NeXT)eq or}{pop false} ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /QV{gsave transform round exch round exch itransform moveto rulex 0 rlineto 0 ruley neg rlineto rulex neg 0 rlineto fill grestore}B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail} B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{ 3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{ 3 2 roll p a}B /bos{/SS save N}B /eos{SS restore}B end %%EndProcSet TeXDict begin 40258431 52099146 1000 300 300 (/tmp_mnt/Net/antireo/antireo6/lusk/mpi2/report2/temp.dvi) @start /Fa 1 106 df<030007800780030000000000000000007F807F80038003800380 038003800380038003800380038003800380FFFCFFFC0E187D9714>105 D E /Fb 12 122 df<387CFEFEFE7C3807077C860F>46 D<00E00001E0000FE000FFE000 F3E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E000 03E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E000FFFF80 FFFF80111D7C9C1A>49 D<01FC0007FF000E0F801E0FC03F07E03F07E03F07E03F07E01E 0FC0000FC0000F80001F0001FC0001FC00000F800007C00003E00003F00003F83803F87C 03F8FE03F8FE03F8FE03F0FC03F07807E03C0FC01FFF8003FC00151D7E9C1A>51 D69 D<07FC001FFF003F0F803F07 C03F03E03F03E00C03E00003E0007FE007FBE01F03E03C03E07C03E0F803E0F803E0F803 E0FC05E07E0DE03FF8FE0FE07E17147F9319>97 D<01FE0007FF800F83C01E01E03E00F0 7C00F07C00F8FC00F8FFFFF8FFFFF8FC0000FC0000FC00007C00007C00003E00181E0018 0F807007FFE000FF8015147F9318>101 D108 DI112 D<01800180018003800380038007800F803F80FFFCFFFC0F800F800F800F80 0F800F800F800F800F800F800F860F860F860F860F8607CC03F801F00F1D7F9C14>116 D120 DI E /Fc 4 103 df<70F8FCFC74040404080810102040060E7C840D>59 D<00F1800389C00707800E03801C03803C0380380700780700780700780700F00E00F00E 00F00E00F00E10F01C20F01C20703C20705C40308C400F078014147E9318>97 D<07803F8007000700070007000E000E000E000E001C001C001CF01D0C3A0E3C0E380F38 0F700F700F700F700FE01EE01EE01EE01CE03CE038607060E031C01F0010207E9F14>I< 00007C0000CE00019E00039E00030C000700000700000700000700000E00000E00000E00 00FFF0000E00000E00001C00001C00001C00001C00001C00003800003800003800003800 00380000700000700000700000700000700000E00000E00000E00000E00000C00001C000 318000798000F300006200003C000017297E9F16>102 D E /Fd 33 122 df<00E001C0038007000E000E001C001C003800380038007000700070007000E0 00E000E000E000E000E000E000E000E000E000E000E00070007000700070003800380038 001C001C000E000E000700038001C000E00B2A7E9E10>40 DI44 D<001C0000003E0000003E0000 002E0000006700000067000000E7800000C7800000C3800001C3C0000183C0000181C000 0381E0000381E0000700F0000700F0000600F0000E0078000FFFF8000FFFF8001C003C00 1C003C0018003C0038001E0038001E0070001F0070000F0070000F00E0000780191D7F9C 1C>65 DI<003FC000FFF003C0F007 80300F00001E00003C00003C0000780000780000780000F00000F00000F00000F00000F0 0000F00000F00000F00000F000007800007800007800003C00003C00001E00000F000807 801803C07800FFF0003F80151F7D9D1B>III<003F8001FFF003C0F807 80380F00181E00003C00003C0000780000780000780000F00000F00000F00000F00000F0 0000F00000F007F8F007F8F000387800387800387800383C00383C00381E00380F003807 803803C0F801FFF0003F80151F7D9D1C>71 D73 D76 DII<003F000001FFE00003FFF00007C0F8000F807C001E 001E003E001F003C000F00780007807800078078000780F00003C0F00003C0F00003C0F0 0003C0F00003C0F00003C0F00003C0F00003C0F80007C078000780780007807C000F803C 000F003E001F001F003E000F807C0007C0F80003FFF00001FFE000003F00001A1F7E9D1F >II<003F000001FFE00003FFF000 07C0F8000F807C001F003E003E001F003C000F00780007807800078078000780F00003C0 F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C078000780 78000780780E07803C0F0F003E079F001E03DE000F83FC0007C1F80003FFF00001FFF800 003F780000003C0000003E0000001F0000000F801A237E9D1F>II<03F8000FFE001C0F00380700700300600000E00000E000 00E00000E00000F000007800007F00003FE0001FFC0007FE0001FF00001F800007800003 C00003C00001C00001C00001C00001C0C00180E00380F007007C0E001FFC0007F000121F 7E9D17>III<78000E007C001E003C003C001E0038000F0070000F00F0000781E0 0003C1C00001C3C00001E7800000F70000007E0000003E0000003C0000003C0000007E00 000077000000E7800001E3800003C1C0000381E0000700F0000F00F8000E0078001C003C 003C003E0078001F0070000F00F0000F80191D7F9C1C>88 DI<0FC03FF07FF87038401C001C001C00FC0FFC3FFC781CE01CE01CE01CF07C7FFC 7FDC3F1C0E127E9114>97 DI<000E000E000E000E000E000E000E000E000E000E000E0F8E1FEE3FFE7C3E70 0E700EE00EE00EE00EE00EE00EE00EF00E701E7C3E3FFE1FEE0F8E0F1D7E9C15>100 D<07C01FE03FF078787018601CFFFCFFFCFFFCE000E000E000700070043C1C3FFC1FF807 E00E127E9112>I108 D110 D112 D<1C001C001C001C001C001C00FFE0FFE01C001C001C001C001C001C00 1C001C001C001C001C001C001C201FF00FF007C00C187F970F>116 DI<7003807807003C0E001C1C000E1C0007380003F00001E00001C00001 E00003F0000738000E18000E1C001C0E00380700700380F003C01212809113>120 DI E /Fe 24 119 df<00003F03E00000C386700001878CF00003879CF00003031860000700380000 070038000007003800000E003800000E007000000E007000000E00700000FFFFFF80001C 007000001C00E000001C00E000001C00E000001C00E000003800E000003801C000003801 C000003801C000003801C000007001C00000700380000070038000007003800000700380 00006003800000E007000000E007000000E007000000E007000000C006000001C00E0000 01C00E000031860C0000798F180000F31E100000620C6000003C07C000002429829F1C> 11 D<70F8F8F0E005057B840E>46 D<00000200000006000000060000000E0000001E00 00001E0000003F0000002F0000004F0000004F0000008F0000010F0000010F0000020F00 00020F0000040F00000C0F0000080F0000100F0000100F0000200F80003FFF8000400780 00C007800080078001000780010007800200078002000780060007801E000F80FF807FF8 1D207E9F22>65 D<01FFFFFE001E001C001E000C001E0004001E0004003C0004003C0004 003C0004003C00040078080800780800007808000078180000F0300000FFF00000F03000 00F0300001E0200001E0200001E0200001E0001003C0002003C0002003C0004003C00040 078000800780018007800100078007000F001F00FFFFFE001F1F7D9E1F>69 D<01FFFF00001E03C0001E00E0001E0070001E0078003C0078003C0078003C0078003C00 78007800F0007800F0007801E0007801C000F0070000F01E0000FFF00000F0380001E01C 0001E01E0001E00E0001E00F0003C01E0003C01E0003C01E0003C01E0007803C0007803C 0807803C0807803C100F801C10FFF00C20000007C01D207D9E21>82 D<00F1800389C00707800E03801C03803C0380380700780700780700780700F00E00F00E 00F00E00F00E20F01C40F01C40703C40705C40308C800F070013147C9317>97 D<07803F8007000700070007000E000E000E000E001C001C001CF01D0C3A0E3C0E380F38 0F700F700F700F700FE01EE01EE01EE01CE03CE038607060E031C01F0010207B9F15>I< 007E0001C1000300800E07801E07801C07003C0200780000780000780000F00000F00000 F00000F00000F0000070010070020030040018380007C00011147C9315>I<0000780003 F80000700000700000700000700000E00000E00000E00000E00001C00001C000F1C00389 C00707800E03801C03803C0380380700780700780700780700F00E00F00E00F00E00F00E 20F01C40F01C40703C40705C40308C800F070015207C9F17>I<007C01C207010E011C01 3C013802780C7BF07C00F000F000F000F0007000700170023804183807C010147C9315> I<00007800019C00033C00033C000718000700000700000E00000E00000E00000E00000E 0001FFE0001C00001C00001C00001C000038000038000038000038000038000070000070 0000700000700000700000700000E00000E00000E00000E00000C00001C00001C0000180 003180007B0000F300006600003C00001629829F0E>I<003C6000E27001C1E00380E007 00E00F00E00E01C01E01C01E01C01E01C03C03803C03803C03803C03803C07003C07001C 0F001C17000C2E0003CE00000E00000E00001C00001C00301C00783800F0700060E0003F 8000141D7E9315>I<01E0000FE00001C00001C00001C00001C000038000038000038000 038000070000070000071E000763000E81800F01C00E01C00E01C01C03801C03801C0380 1C0380380700380700380700380E10700E20700C20701C20700C40E00CC060070014207D 9F17>I<00C001E001E001C000000000000000000000000000000E003300230043804300 470087000E000E000E001C001C001C003840388030807080310033001C000B1F7C9E0E> I<03C01FC0038003800380038007000700070007000E000E000E000E001C001C001C001C 0038003800380038007000700070007100E200E200E200E200640038000A207C9F0C> 108 D<1C0F80F0002630C318004740640C004780680E004700700E004700700E008E00E0 1C000E00E01C000E00E01C000E00E01C001C01C038001C01C038001C01C038001C01C070 8038038071003803806100380380E10038038062007007006600300300380021147C9325 >I<1C0F802630C04740604780604700704700708E00E00E00E00E00E00E00E01C01C01C 01C01C01C01C03843803883803083807083803107003303001C016147C931A>I<007C00 01C3000301800E01C01E01C01C01E03C01E07801E07801E07801E0F003C0F003C0F003C0 F00780F00700700F00700E0030180018700007C00013147C9317>I<01C1E00262180474 1C04781C04701E04701E08E01E00E01E00E01E00E01E01C03C01C03C01C03C01C0380380 780380700380E003C1C0072380071E000700000700000E00000E00000E00000E00001C00 001C0000FFC000171D809317>I<1C1E002661004783804787804707804703008E00000E 00000E00000E00001C00001C00001C00001C000038000038000038000038000070000030 000011147C9313>114 D<00FC030206010C030C070C060C000F800FF007F803FC003E00 0E700EF00CF00CE008401020601F8010147D9313>I<018001C003800380038003800700 0700FFF007000E000E000E000E001C001C001C001C003800380038003820704070407080 708031001E000C1C7C9B0F>I<0E00C03300E02301C04381C04301C04701C08703800E03 800E03800E03801C07001C07001C07001C07101C0E20180E20180E201C1E200C264007C3 8014147C9318>I<0E03803307802307C04383C04301C04700C08700800E00800E00800E 00801C01001C01001C01001C02001C02001C04001C04001C08000E300003C00012147C93 15>I E /Ff 50 123 df<000078007C7800FC7801FC7803C00003800007800007800007 8000078000078000078000078000078000FFFC78FFFC78FFFC7807807807807807807807 807807807807807807807807807807807807807807807807807807807807807807807807 8078078078078078078078152480A31A>12 D45 DI<00C001C007C0FFC0FFC0FBC003C003C003C003C003C003C0 03C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C0 03C0FFFFFFFFFFFF10227CA118>49 D<03F0000FFC001FFE003C1F003007807007C06003 C0E003E0C001E04001E04001E00001E00001E00001E00003C00003C0000780000780000F 00001E00003C0000780000F00001E00001C0000380000700000E00001C00003800007000 00FFFFE0FFFFE0FFFFE013227EA118>I<01F00007FC001FFF003E0F003807807003C020 03C02003C00003C00003C00003C0000780000780000F00001E0003FC0003F80003FE0000 0F000007800003C00003C00001E00001E00001E00001E00001E08001E0C003C0E003C070 07803C0F801FFF000FFC0003F00013237EA118>I<001F00001F00002F00002F00006F00 00EF0000CF0001CF0001CF00038F00038F00078F00070F000F0F000E0F001E0F003C0F00 3C0F00780F00780F00F00F00FFFFF8FFFFF8FFFFF8000F00000F00000F00000F00000F00 000F00000F00000F00000F0015217FA018>I55 D<01F00007FC000FFE001E0F003C07803C0780 7803C07803C07803C07803C07803C03803803C07801E0F000F1E0007FC0003F8000FFE00 1E0F003C07807803C07803C0F001E0F001E0F001E0F001E0F001E0F001E07803C07803C0 3C07803E0F801FFF0007FC0001F00013237EA118>I<01F00007FC000FFE001E0F003C07 003807807803807803C0F003C0F001C0F001E0F001E0F001E0F001E0F001E0F001E07003 E07803E07803E03C07E03E1FE01FFDE00FF1E003E3C00003C00003C00003800007800007 00000F00101E00183C003FF8001FF00007C00013237EA118>I<001F0000001F0000003F 8000003F8000003B8000007BC0000073C0000071C00000F1E00000F1E00000E0E00001E0 F00001E0F00001C0F00003C0780003C078000380780007803C0007803C0007003C000F00 1E000F001E000FFFFE001FFFFF001FFFFF001C000F003C0007803C000780380007807800 03C0780003C0700003C0F00001E0F00001E0E00001E01B237EA220>65 DI<000FF000007FFC0000FFFF0001F01F0003C00700078000000F0000001E0000 003E0000003C0000003C000000780000007800000078000000F0000000F0000000F00000 00F0000000F0000000F0000000F0000000F0000000F00000007800000078000000780000 003C0000003C0000003E0000001E0000000F0000000780008003C0038001F00F8000FFFF 00007FFC00000FF00019257DA31F>IIII<000FF000003FFE0000 FFFF8001F80F8003E00380078000000F0000001E0000001E0000003C0000003C00000078 0000007800000078000000F0000000F0000000F0000000F0000000F0000000F0000000F0 00FFC0F000FFC0F000FFC0780003C0780003C0780003C03C0003C03C0003C01E0003C01E 0003C00F0003C0078003C003E003C001F807C000FFFFC0003FFF00000FF8001A257DA321 >I73 D76 DII<001FC000007FF00001FFFC0003F07E 0007C01F000F800F801F0007C01E0003C03C0001E03C0001E0780000F0780000F0780000 F070000070F0000078F0000078F0000078F0000078F0000078F0000078F0000078F00000 78F0000078780000F0780000F0780000F07C0001F03C0001E03E0003E01E0003C01F0007 C00F800F8007C01F0003F07E0001FFFC00007FF000001FC0001D257DA324>II< FFFC00FFFF80FFFFC0F003E0F000F0F00078F00038F0003CF0003CF0003CF0003CF0003C F00038F00078F000F0F003E0FFFFC0FFFF80FFFE00F01E00F00F00F00700F00780F00380 F003C0F001E0F001E0F000F0F000F0F00078F00038F0003CF0001EF0001EF0000F18237B A21F>82 D<00FE0003FFC007FFE00F81E01E00603C00003C000078000078000078000078 00007800007C00003C00003F00001FC0000FFC0007FF0001FF80003FC00007E00001F000 00F00000F8000078000078000078000078000078000078C000F0E000F0F801E07E07C03F FF800FFF0001FC0015257EA31B>III90 D<07E01FF83FFC381E201E000F000F000F000F00FF07FF1FFF3E0F780FF00FF00FF00FF0 0FF83F7FFF3FEF1F8F10167E9517>97 DI<01FC0007FF000FFF801F03803C0180 780000780000700000F00000F00000F00000F00000F00000F00000780000780000780000 3C00401F03C00FFFC007FF8001FC0012167E9516>I<0003C00003C00003C00003C00003 C00003C00003C00003C00003C00003C00003C00003C00003C003E3C00FFBC01FFFC03F0F C03C07C07803C07803C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C07803 C07803C03C07C03E0FC01FFFC00FFBC003E3C012237EA219>I<03F00007FC001FFE003E 0F003C0780780380780380F001C0FFFFC0FFFFC0FFFFC0F00000F00000F0000070000078 00007800003C00801F07800FFF8007FF0001F80012167E9516>I<003F00FF01FF03C003 8007800780078007800780078007800780FFF8FFF8FFF807800780078007800780078007 80078007800780078007800780078007800780078007800780102380A20F>I<01F07807 FFF80FFFF81F1F001E0F003C07803C07803C07803C07803C07801E0F001F1F000FFE001F FC0019F0003800003800003C00001FFE001FFFC01FFFE03FFFF07801F07800F8F00078F0 0078F00078F000787800F03E03E01FFFC00FFF8001FC0015217F9518>I 105 D107 DIII<01FC0007FF000FFF801F07C03C01E07800F078 00F0700070F00078F00078F00078F00078F00078F000787800F07800F07C01F03E03E01F 07C00FFF8007FF0001FC0015167F9518>II114 D<07F01FFC3FFE3C0E7806780078007C003F003FF01FF80FFC01FE001F000F000F000FC0 0FF81EFFFE3FFC0FF010167F9513>I<0F000F000F000F000F000F00FFF8FFF8FFF80F00 0F000F000F000F000F000F000F000F000F000F000F000F000F000F080F1C07FC07F803E0 0E1C7F9B12>III<7801F07C01E03E03C01E07C00F 0780078F0007DE0003FC0001FC0000F80000700000F80001FC0003DC00039E00078F000F 07801E07801E03C03C01E07800F0F800F81516809516>120 DI<7FFFC07FFFC07FFFC0000F80000F00 001E00003E00007C0000780000F00001F00003E00003C0000780000F80001F00001E0000 3C00007C0000FFFFC0FFFFC0FFFFC012167F9515>I E /Fg 32 90 df45 D<3078F8787005057C840D>I<007E0001C3000301800701 C00E00C00E00E01C00E01C00E03C01E03801E07801E07801E07801E07801E07801E0F003 C0F003C0F003C0F003C0F003C0F00380F00780E00780E00700E00700E00E00600E00701C 003038003870000FC000131F7C9D17>48 D<000C001C00FC0F3800380038003800380038 00700070007000700070007000E000E000E000E000E000E001C001C001C001C001C001C0 038003C0FFFE0F1E7C9D17>I<003F8000C1E00100F00200780400780400780F007C0F80 7C0F807C0F00780600780000F80000F00001E00001C0000380000700000E00001C000038 0000600000C0000180000300200600200800401000403FFFC07FFF80FFFF80161E7E9D17 >I<007F000183C00201E00400F00700F00F00F00F01F00F01F00001E00001E00003C000 0380000700000E0000F800000E000007000007800007C00003C00007C03007C07807C0F8 07C0F807C0F00780800F00400E00201C0018780007E000141F7D9D17>I<01803001FFE0 03FFC003FF0003FC00020000020000020000040000040000040000047C00058700060380 0C01800801C00001C00001E00001E00001E00001E07003C0F803C0F003C0E00380800780 400700400E00201C0018700007C000141F7D9D17>53 D<2000003FFFE07FFFC07FFF8040 0100C0020080020080040000080000100000200000400000400000800001800003000003 00000700000600000E00000E00001E00001C00001C00003C00003C00003C000078000078 0000780000300000131F799D17>55 D<003F0000C1C00100600200600400300C00300C00 300C00300C00600E00600F80C00FC18007F60003FC0001FC0001FF00063F800C0F801007 C03003C06001C06000C0C000C0C000C0C000C0C00080C0010060030030040018180007E0 00141F7D9D17>I<007E0001C3000301800601C00E01C01C00C03C00E03C00E03C01E078 01E07801E07801E07801E07803E07803E03803C03807C01C0BC00C13C003E38000078000 0780000700000E00600E00F01C00F01800E0300080600041C0003F0000131F7C9D17>I< 0000100000001800000038000000380000007800000078000000FC000001BC0000013C00 00033C0000023C0000063C0000043E0000081E0000081E0000101E0000101E0000201E00 00200F0000400F0000400F0000FFFF0000800F0001000F80010007800200078002000780 04000780040007800C0007C03E0007C0FF807FFC1E207E9F22>65 D<0001F808000E061800380138007000F801E0007803C0007007800030078000300F0000 301F0000301E0000303E0000203C0000007C0000007C0000007C0000007C000000F80000 00F8000000F8000000F8000000F80000007800004078000080780000803C0000803C0001 001C0002000E00020006000C000300100001C0E000003F00001D217B9F21>67 D<07FFFF00007C01E0003C00F0003C00780078003C0078003C0078001E0078001E007800 1E0078001F00F0001F00F0001F00F0001F00F0001F00F0001F00F0001F01E0001E01E000 3E01E0003E01E0003E01E0003C01E0007C03C0007803C000F003C000F003C001E003C003 C003C0078007800F0007803C00FFFFE000201F7E9E23>I<07FFFFF8007C0078003C0038 003C001800780018007800080078000800780008007800080078080800F0100000F01000 00F0100000F0300000FFF00000F0700001E0200001E0200001E0200001E0200001E00008 01E0001003C0001003C0001003C0002003C0002003C0006003C000C0078001C0078007C0 FFFFFF801D1F7E9E1F>I<07FFFFF8007C0078003C0038003C0018007800180078000800 78000800780008007800080078000800F0100000F0100000F0100000F0300000F0700000 FFF00001E0600001E0200001E0200001E0200001E0200001E0000003C0000003C0000003 C0000003C0000003C0000003C000000780000007C00000FFFE00001D1F7E9E1E>I<0001 FC04000F030C003C009C0070007C00E0003C01C0003803800018078000180F0000181F00 00181E0000183E0000103C0000007C0000007C0000007C0000007C000000F8000000F800 0000F8007FFCF80003E0780001E0780001E0780003C0780003C03C0003C03C0003C01C00 03C00E0007C007000B800380118001E06080003F80001E217B9F24>I<07FFC7FFC0007C 00F800003C007800003C007800007800F000007800F000007800F000007800F000007800 F000007800F00000F001E00000F001E00000F001E00000F001E00000FFFFE00000F001E0 0001E003C00001E003C00001E003C00001E003C00001E003C00001E003C00003C0078000 03C007800003C007800003C007800003C007800003C007800007800F000007C00F8000FF F8FFF800221F7E9E22>I<07FFE0007C00003C00003C0000780000780000780000780000 780000780000F00000F00000F00000F00000F00000F00001E00001E00001E00001E00001 E00001E00003C00003C00003C00003C00003C00003C00007800007C000FFFC00131F7F9E 10>I<07FFE07FE0007C001F00003C000C00003C00180000780010000078004000007800 800000780100000078020000007804000000F008000000F010000000F060000000F0F000 0000F1F0000000F278000001E478000001E878000001F03C000001E03C000001E01E0000 01E01E000003C00F000003C00F000003C00F000003C007800003C007800003C003C00007 8003C00007C007E000FFFC3FFC00231F7E9E23>75 D<07FFF000007E0000003C0000003C 000000780000007800000078000000780000007800000078000000F0000000F0000000F0 000000F0000000F0000000F0000001E0000001E0000001E0000001E0000001E0008001E0 010003C0010003C0010003C0030003C0020003C0060003C0060007801E0007807C00FFFF FC00191F7E9E1C>I<07FC0000FFC0007C0000F800003C00017800003C00017800004E00 02F000004E0002F000004E0004F000004E0004F000004E0008F000004E0008F000008700 11E00000870011E00000870021E00000870021E00000870041E00000838041E000010380 83C00001038083C00001038103C00001038203C0000101C203C0000101C403C0000201C4 0780000201C80780000201C80780000201D00780000200F00780000600E00780000600E0 0F00000F00C00F8000FFE0C1FFF8002A1F7E9E2A>I<07FC01FFC0003E003E00003E0018 00003E001800004F001000004F001000004780100000478010000043C010000043C01000 0083C020000081E020000081E020000080F020000080F020000080782000010078400001 007C400001003C400001003C400001001E400001001E400002000F800002000F80000200 0F800002000780000200078000060003800006000300000F00010000FFE0010000221F7E 9E22>I<0003F800001E0E000038070000E0038001C001C003C001E0078000E00F0000F0 0F0000F01E0000F01E0000F83E0000F83C0000F87C0000F87C0000F87C0000F87C0000F8 F80001F0F80001F0F80001F0F80001F0F80003E0780003E0780003C0780007C07C000780 3C000F003C001E001E001C000E0038000700F00003C3C00000FE00001D217B9F23>I<07 FFFF00007C03C0003C01E0003C00F0007800F0007800F8007800F8007800F8007800F800 7800F000F001F000F001E000F003C000F0078000F00F0000FFF80001E0000001E0000001 E0000001E0000001E0000001E0000003C0000003C0000003C0000003C0000003C0000003 C000000780000007C00000FFFC00001D1F7E9E1F>I<07FFFC00007C0700003C03C0003C 01E0007801E0007801F0007801F0007801F0007801F0007801E000F003E000F003C000F0 078000F00F0000F03C0000FFF00001E0300001E0380001E01C0001E01C0001E01C0001E0 1E0003C03E0003C03E0003C03E0003C03E0003C03E0003C03E0207803E0407C01F04FFFC 0F18000003E01F207E9E21>82 D<003F040060CC01803C03801C03001C0700180600080E 00080E00080E00080E00000F00000F80000FE00007FE0003FF8001FFC0007FE00007E000 01E00000E00000F00000F04000E04000E04000E04000E06000C0600180E00380F80300C6 0C0081F80016217D9F19>I<3FFFFFF03C0780F03007803060078030400F0010400F0010 C00F0010800F0010800F0010800F0010001E0000001E0000001E0000001E0000001E0000 001E0000003C0000003C0000003C0000003C0000003C0000003C00000078000000780000 0078000000780000007800000078000000F0000001F800007FFFE0001C1F7A9E21>IIII< 03FFC0FFC0007F007E00003E003800001E003000001E002000000F004000000F00800000 0F81000000078200000007C600000003C400000003E800000001F000000001F000000000 F000000000F800000000F8000000017C000000023C000000043C0000000C1E000000081E 000000101F000000200F000000400F800000C0078000008007C000010003C000070003E0 001F8007E000FFE01FFE00221F7F9E22>II E /Fh 15 117 df<020408103020604040C0C0C0C0C0C0C0C0404060203010 080402071A7F920C>40 D<8040201018080C0404060606060606060604040C0818102040 80071A7E920C>I<1F00318060C04040C060C060C060C060C060C060C060C060404060C0 31801F000B107F8F0F>48 D<0C003C00CC000C000C000C000C000C000C000C000C000C00 0C000C000C00FF8009107E8F0F>I<1F00618040C08060C0600060006000C00180030006 000C00102020207FC0FFC00B107F8F0F>I<1F00218060C060C000C0008001800F000080 00400060C060C060804060801F000B107F8F0F>I<0300030007000F000B001300330023 004300C300FFE003000300030003001FE00B107F8F0F>I<20803F002C00200020002000 2F0030802040006000600060C06080C061801F000B107F8F0F>I<0780184030C060C060 00C000CF00F080E040C060C060C060406060C030801F000B107F8F0F>I<40007FE07FC0 8080808001000200040004000C0008000800180018001800180018000B117E900F>I<1F 00318060C060C060C071803F000F00338061C0C060C060C060404060801F000B107F8F0F >I<1F00318060C0C040C060C060C06040E021E01E600060004060C0608043003E000B10 7F8F0F>I<03E0000C1800100400200200600300400100C00180C00180C00180C00180C0 01806003006003003006003006000C180003E00011117E9017>79 D<1F8030C06000C000C000C000C000C000604030801F000A0B7F8A0E>99 D<10103030FE3030303030323232321C070F7F8E0C>116 D E /Fi 3 64 df<03C00FF01FF83FFC7FFE7FFEFFFFFFFFFFFFFFFFFFFFFFFF7FFE7FFE3FFC1FF8 0FF003C010127D9317>15 D62 D<00040000000C0000000C0000 000C0000000C0000000C0000000C0000000C0000000C0000000C0000000C0000000C0000 000C0000000C0000000C0000000C0000000C0000000C0000000C0000000C0000000C0000 000C0000000C0000000C0000000C0000000C0000FFFFFFE0FFFFFFE01B1C7C9B23>I E /Fj 60 123 df<00F8F001F8F003F8F00780F00700F00F00F00F00F00F00F00F00F00F 00F00F00F00F00F0FFF8F0FFF8F0FFF8F00F00F00F00F00F00F00F00F00F00F00F00F00F 00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F01420809F 18>13 D<007000E001C00380078007000E001E001E003C003C003C007800780078007800 7000F000F000F000F000F000F000F000F000F000F000F000F00070007800780078007800 3C003C003C001E001E000E0007000780038001C000E000700C2E7EA112>40 DI<7878787818303060 60E0050A7D830D>44 DII<03F000 07F8001FFE001E1E003C0F00380700780780780780700380F003C0F003C0F003C0F003C0 F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0780780780780 7807803807003C0F001E1E001FFE000FFC0003F00012207E9E17>48 D<00C001C00FC0FFC0FFC0F3C003C003C003C003C003C003C003C003C003C003C003C003 C003C003C003C003C003C003C003C003C003C003C0FFFEFFFEFFFE0F1F7C9E17>I58 D61 D<001F0000001F0000003F8000003B8000003B8000007BC0000073 C0000071C00000F1E00000E1E00000E0E00001E0F00001E0F00001C0F00003C0780003C0 78000380780007803C0007803C0007003C000FFFFE000FFFFE000FFFFE001E000F001E00 0F003C000F803C0007803C000780780007C0780003C0780003C0F00003E01B207F9F1E> 65 DI<001F C000FFF801FFFC03E03C07800C0F00001E00003E00003C00007C00007800007800007800 00F00000F00000F00000F00000F00000F00000F00000F000007800007800007800007C00 003C00003E00001E00000F000207800E03E03E01FFFC00FFF0001FC017227DA01D>IIII<001FE000FFF801FFFE03E03E07800E0F 00001E00003E00003C00007C0000780000780000780000F00000F00000F00000F00000F0 0000F00000F003FEF003FE7803FE78001E78001E7C001E3C001E3E001E1E001E0F001E07 801E03E03E01FFFE00FFF8001FC017227DA01E>III75 DI II<003F0000 00FFC00003FFF00007E1F8000F807C001F003E001E001E003C000F003C000F0078000780 7800078078000780F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0 F00003C0F00003C0F80007C07800078078000780780007803C000F003C000F001E001E00 1F003E000F807C0007E1F80003FFF00000FFC000003F00001A227DA021>II<003F000000FFC00003FF F00007E1F8000F807C001F003E001E001E003C000F003C000F0078000780780007807800 0780F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F000 03C0F00003C07800078078000780781E07803C0F0F003C0F8F001E079E001F03FE000F83 FC0007E1F80003FFF00000FFF800003F780000007C0000003E0000001E0000001F000000 0F801A277DA021>II<01FC0007FF800FFFC01F03C03C00C03C00007800007800007800007800007800 007C00003C00003F00001FE0000FFC0007FE0001FF00003F800007C00003C00003E00001 E00001E00001E00001E00001E00001C0C003C0F007C0FC0F807FFF001FFE0003F8001322 7EA019>IIIII<780007807C000F003E001F001E001E000F003C000F807C0007807800 03C0F00003E1F00001F1E00000F3C000007FC000007F8000003F0000001F0000001E0000 003F0000007F8000007FC00000F3C00001F1E00001E0F00003C0F80007C0780007803C00 0F003E001F001E001E000F003C000F807C0007C0780003C0F00003E01B207F9F1E>III<07E03FF87FFC701E401F000F000F00 0F003F07FF1FFF7E0FF80FF00FF00FF00FF83F7FFF3FEF1F8F10147E9316>97 DI<03F00FFC 1FFE3E0E3C0278007800F000F000F000F000F000F000780078003C013E0F1FFF0FFE03F0 10147E9314>I<0007800007800007800007800007800007800007800007800007800007 8000078000078007C7800FF7801FFF803E1F807C0780780780F80780F00780F00780F007 80F00780F00780F00780F00780780780780F803E1F801FFF800FF78007C78011207E9F17 >I<03F0000FFC001FFE003E1F003C0700780700700380FFFF80FFFF80FFFF80F00000F0 0000F000007000007800003C01003E07001FFF0007FE0001F80011147F9314>I<007E01 FE03FE078007000F000F000F000F000F000F000F00FFF0FFF0FFF00F000F000F000F000F 000F000F000F000F000F000F000F000F000F000F000F000F000F20809F0E>I<03E0F00F FFF01FFFF03E3E003C1E00780F00780F00780F00780F00780F003C1E003E3E001FFC003F F80033E0003000003800003FFE003FFF801FFFC03FFFE07803F0F000F0F000F0F000F0F8 01F07E07E03FFFC00FFF0003FC00141E7F9317>III107 DIII<01F80007FE001FFF803F0FC03C03C07801E07801E0F000 F0F000F0F000F0F000F0F000F0F000F07801E07801E03C03C03F0FC01FFF8007FE0001F8 0014147F9317>II<03C7800FF780 1FFF803E1F807C0F80780780780780F00780F00780F00780F00780F00780F00780F80780 7807807C0F803E1F801FFF800FF78007C780000780000780000780000780000780000780 000780000780000780111D7E9317>II<07F01FFC3FFC780C78 00780078007C003FC01FF00FF803F8007C003C003CC03CF07CFFF87FF00FC00E147F9311 >I<1E001E001E001E001E001E00FFF0FFF0FFF01E001E001E001E001E001E001E001E00 1E001E001E001E001E001E201FF00FF007C00C1A7F9910>II< F003C0F003C07803807807807807803C0F003C0F003C0F001E0E001E1E001E1E000F1C00 0F3C000F3C0007380007380007B80003F00003F00001E00012147F9315>II<7801E07C03C03E07801E0F000F0F00079E0003FC0003F80001F8 0000F00001F00001F80003FC00079E000F0F000E0F001E07803C03C07801E0F801F01414 809315>II<7FFF7FFF7FFF003E00 3C007800F800F001E003E007C007800F001F001E003C007C00FFFFFFFFFFFF10147F9314 >I E /Fk 73 126 df<030600078F00078F00078F00078F00078F00078F007FFFC0FFFF E0FFFFE07FFFC00F1E000F1E000F1E000F1E000F1E000F1E007FFFC0FFFFE0FFFFE07FFF C01E3C001E3C001E3C001E3C001E3C001E3C000C1800131C7E9B18>35 D<01C00007E0000FF0000E70001C38001C38001C38001C38001C73F01C73F01CE3F00FE3 800FC7000F87000F07001F0E003F0E007B8E0073DC00E1DC00E0F800E0F800E07070E078 7070FC707FFFE03FCFE00F03C0141C7F9B18>38 D<007000F001E003C007800F001E001C 00380038007000700070007000E000E000E000E000E000E000E000E00070007000700070 00380038001C001E000F00078003C001F000F000700C24799F18>40 D<6000F00078003C001E000F000780038001C001C000E000E000E000E000700070007000 70007000700070007000E000E000E000E001C001C0038007800F001E003C007800F00060 000C247C9F18>I<01C00001C00001C00001C000C1C180F1C780F9CF807FFF001FFC0007 F00007F0001FFC007FFF00F9CF80F1C780C1C18001C00001C00001C00001C00011147D97 18>I<1C3E7E7F3F1F070E1E7CF860080C788518>44 D<3078FCFC78300606778518>46 D<000300000780000780000F80000F00001F00001E00001E00003E00003C00007C000078 0000780000F80000F00001F00001E00003E00003C00003C00007C0000780000F80000F00 000F00001F00001E00003E00003C00003C00007C0000780000F80000F00000F000006000 0011247D9F18>I<01800380038007800F803F80FF80FB80438003800380038003800380 038003800380038003800380038003800380038003807FFCFFFE7FFC0F1C7B9B18>49 D<03F0000FFE003FFF007C0F807003C0E001C0F000E0F000E06000E00000E00000E00001 C00001C00003C0000780000F00001E00003C0000780000F00001E00007C0000F80001E00 E03C00E07FFFE0FFFFE07FFFE0131C7E9B18>I<07F8001FFE003FFF007807807803C078 01C03001C00001C00003C0000380000F0003FF0003FE0003FF000007800003C00001C000 00E00000E00000E0F000E0F000E0F001C0F003C07C07803FFF001FFE0003F800131C7E9B 18>I<001F00003F0000770000770000E70001E70001C7000387000787000707000E0700 1E07003C0700380700780700F00700FFFFF8FFFFF8FFFFF8000700000700000700000700 000700000700007FF000FFF8007FF0151C7F9B18>I<03F8000FFE001FFF003E0F803803 807001C07001C07001C07001C03803803C07801FFF0007FC000FFE001F1F003C07807001 C0F001E0E000E0E000E0E000E0E000E07001C07803C03E0F801FFF000FFE0003F800131C 7E9B18>56 D<3078FCFC783000000000000000003078FCFC78300614779318>58 D<183C7E7E3C180000000000000000183C7E7E3E1E0E1C3C78F060071A789318>I<7FFF C0FFFFE0FFFFE0FFFFE0000000000000000000000000FFFFE0FFFFE0FFFFE07FFFC0130C 7E9318>61 D<0FF0003FFC007FFF00700F00F00380F00380600780000F00003E00007C00 01F00001E00003C00003C00003C00003C00003C000038000000000000000000000000000 00000003800007C00007C00007C000038000111C7D9B18>63 D<00700000F80000F80000 D80000D80001DC0001DC0001DC00018C00038E00038E00038E00038E0003060007070007 07000707000707000FFF800FFF800FFF800E03800E03801C01C01C01C07F07F0FF8FF87F 07F0151C7F9B18>65 DI<00F8E003FEE007 FFE00F07E01E03E03C01E03800E07000E07000E0700000E00000E00000E00000E00000E0 0000E00000E00000E000007000007000E07000E03800E03C00E01E01C00F07C007FF8003 FE0000F800131C7E9B18>I<7FF800FFFE007FFF001C0F801C03C01C03C01C01E01C00E0 1C00E01C00F01C00701C00701C00701C00701C00701C00701C00701C00701C00F01C00E0 1C00E01C01E01C01C01C03C01C0F807FFF00FFFE007FF800141C7F9B18>III<01F1C0 03FDC00FFFC01F0FC01C03C03803C03801C07001C07001C0700000E00000E00000E00000 E00000E00000E00FF0E01FF0E00FF07001C07001C07003C03803C03803C01C07C01F0FC0 0FFFC003FDC001F1C0141C7E9B18>I<7F07F0FF8FF87F07F01C01C01C01C01C01C01C01 C01C01C01C01C01C01C01C01C01C01C01FFFC01FFFC01FFFC01C01C01C01C01C01C01C01 C01C01C01C01C01C01C01C01C01C01C01C01C07F07F0FF8FF87F07F0151C7F9B18>I<7F FF00FFFF807FFF0001C00001C00001C00001C00001C00001C00001C00001C00001C00001 C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001 C0007FFF00FFFF807FFF00111C7D9B18>I<7F07F0FF87F87F07F01C03C01C07801C0700 1C0E001C1E001C3C001C38001C70001CF0001DF0001DF0001FB8001FB8001F1C001E1C00 1C0E001C0E001C07001C07001C03801C03801C01C07F03F0FF87F87F03F0151C7F9B18> 75 D<7FE000FFE0007FE0000E00000E00000E00000E00000E00000E00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E00000E00700E00700E0070 0E00700E00707FFFF0FFFFF07FFFF0141C7F9B18>II<7E07F0FF0FF87F07F01D81C01D81C01D81C01DC1C01CC1C01CC1C01CE1C01C E1C01CE1C01C61C01C71C01C71C01C31C01C39C01C39C01C39C01C19C01C19C01C1DC01C 0DC01C0DC01C0DC07F07C0FF87C07F03C0151C7F9B18>I<0FF8003FFE007FFF00780F00 700700F00780E00380E00380E00380E00380E00380E00380E00380E00380E00380E00380 E00380E00380E00380E00380E00380E00380F00780700700780F007FFF003FFE000FF800 111C7D9B18>II<0FF8003FFE007FFF0078 0F00700700F00780E00380E00380E00380E00380E00380E00380E00380E00380E00380E0 0380E00380E00380E00380E00380E1E380E1E380F0E78070F700787F007FFF003FFE000F FC00001C00001E00000E00000F0000070000070011227D9B18>I<7FF800FFFE007FFF00 1C0F801C03801C03C01C01C01C01C01C01C01C03C01C03801C0F801FFF001FFE001FFE00 1C0F001C07001C03801C03801C03801C03801C03801C039C1C039C1C039C7F01F8FF81F8 7F00F0161C7F9B18>I<03F3801FFF803FFF807C0F80700780E00380E00380E00380E000 007000007800003F00001FF00007FE0000FF00000F800003C00001C00000E00000E06000 E0E000E0E001E0F001C0F80780FFFF80FFFE00E7F800131C7E9B18>I<7FFFF8FFFFF8FF FFF8E07038E07038E07038E0703800700000700000700000700000700000700000700000 700000700000700000700000700000700000700000700000700000700000700007FF0007 FF0007FF00151C7F9B18>IIII<7F8FE0 7F9FE07F8FE00E07000F0700070E00078E00039C0003DC0001F80001F80000F00000F000 00700000F00000F80001F80001DC00039E00038E00070F000707000E07800E03801E03C0 7F07F0FF8FF87F07F0151C7F9B18>II<3F FFE07FFFE07FFFE07001C07003C0700780700700000F00001E00001C00003C0000780000 700000F00001E00001C00003C0000780000700000F00001E00E01C00E03C00E07800E070 00E0FFFFE0FFFFE0FFFFE0131C7E9B18>II93 D<7FFF00FFFF80FFFF807FFF0011047D7F18>95 D<1FE0003FF8007FFC00781E00300E00 00070000070000FF0007FF001FFF007F0700780700E00700E00700E00700F00F00781F00 3FFFF01FFBF007E1F014147D9318>97 D<7E0000FE00007E00000E00000E00000E00000E 00000E00000E3E000EFF800FFFC00FC1E00F80E00F00700E00700E00380E00380E00380E 00380E00380E00380F00700F00700F80E00FC1E00FFFC00EFF80063E00151C809B18>I< 01FE0007FF001FFF803E0780380300700000700000E00000E00000E00000E00000E00000 E000007000007001C03801C03E03C01FFF8007FF0001FC0012147D9318>I<001F80003F 80001F8000038000038000038000038000038003E3800FFB801FFF803C1F80380F807007 80700380E00380E00380E00380E00380E00380E00380700780700780380F803C1F801FFF F00FFBF803E3F0151C7E9B18>I<01F00007FC001FFE003E0F00380780700380700380E0 01C0E001C0FFFFC0FFFFC0FFFFC0E000007000007001C03801C03E03C01FFF8007FF0001 FC0012147D9318>I<001F80007FC000FFE000E1E001C0C001C00001C00001C0007FFFC0 FFFFC0FFFFC001C00001C00001C00001C00001C00001C00001C00001C00001C00001C000 01C00001C00001C00001C0007FFF007FFF007FFF00131C7F9B18>I<01E1F007FFF80FFF F81E1E301C0E003807003807003807003807003807001C0E001E1E001FFC001FF80039E0 003800001C00001FFE001FFFC03FFFE07801F0700070E00038E00038E00038E000387800 F07E03F01FFFC00FFF8001FC00151F7F9318>I<7E0000FE00007E00000E00000E00000E 00000E00000E00000E3E000EFF800FFFC00FC1C00F80E00F00E00E00E00E00E00E00E00E 00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E07FC3FCFFE7FE7FC3FC171C809B 18>I<03800007C00007C00007C0000380000000000000000000000000007FC000FFC000 7FC00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C000 01C00001C00001C000FFFF00FFFF80FFFF00111D7C9C18>I<0038007C007C007C003800 000000000000000FFC1FFC0FFC001C001C001C001C001C001C001C001C001C001C001C00 1C001C001C001C001C001C001C001C001C001C001C6038F078FFF07FE03F800E277E9C18 >II<7FE000FFE0007FE00000E00000E000 00E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E000 00E00000E00000E00000E00000E00000E00000E00000E0007FFFC0FFFFE07FFFC0131C7E 9B18>I<7CE0E000FFFBF8007FFFF8001F1F1C001E1E1C001E1E1C001C1C1C001C1C1C00 1C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C00 7F1F1F00FFBFBF807F1F1F001914819318>I<7E3E00FEFF807FFFC00FC1C00F80E00F00 E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E07FC3 FCFFE7FE7FC3FC1714809318>I<01F0000FFE001FFF003E0F803803807001C07001C0E0 00E0E000E0E000E0E000E0E000E0F001E07001C07803C03C07803E0F801FFF000FFE0001 F00013147E9318>I<7E3E00FEFF807FFFC00FC1E00F80E00F00700E00700E00380E0038 0E00380E00380E00380E00380F00700F00700F80E00FC1E00FFFC00EFF800E3E000E0000 0E00000E00000E00000E00000E00000E00007FC000FFE0007FC000151E809318>I<01E3 8007FB801FFF803E1F80380F80700780700780E00380E00380E00380E00380E00380E003 80700780700780380F803C1F801FFF800FFB8003E3800003800003800003800003800003 80000380000380003FF8003FF8003FF8151E7E9318>I<7F87E0FF9FF07FBFF803F87803 F03003E00003C00003C00003800003800003800003800003800003800003800003800003 80007FFE00FFFF007FFE0015147F9318>I<07F7003FFF007FFF00780F00E00700E00700 E007007C00007FE0001FFC0003FE00001F00600780E00380E00380F00380F80F00FFFF00 FFFC00E7F00011147D9318>I<0180000380000380000380000380007FFFC0FFFFC0FFFF C00380000380000380000380000380000380000380000380000380000380400380E00380 E00380E001C1C001FFC000FF80003E0013197F9818>I<7E07E0FE0FE07E07E00E00E00E 00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E01E00F 03E007FFFC03FFFE01FCFC1714809318>I<7F8FF0FF8FF87F8FF01E03C00E03800E0380 0E0380070700070700070700038E00038E00038E00038E0001DC0001DC0001DC0000F800 00F80000700015147F9318>II<7F8FF07F9FF07F8FF0070700078E00039E0001DC0001F80000F80000 700000F00000F80001DC00039E00038E000707000F07807F8FF0FF8FF87F8FF015147F93 18>I<7F8FF0FF8FF87F8FF00E01C00E03800E0380070380070700070700038700038600 038E0001CE0001CE0000CC0000CC0000DC00007800007800007800007000007000007000 00F00000E00079E0007BC0007F80003F00001E0000151E7F9318>I<3FFFF07FFFF07FFF F07001E07003C0700780000F00001E00003C0000F80001F00003C0000780000F00701E00 703C0070780070FFFFF0FFFFF0FFFFF014147F9318>I<0007E0001FE0007FE000780000 E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00001E0007F C000FF8000FF80007FC00001E00000E00000E00000E00000E00000E00000E00000E00000 E00000E00000E00000E000007800007FE0001FE00007E013247E9F18>I<7C0000FF0000 FFC00003C00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E000 00E00000F000007FC0003FE0003FE0007FC000F00000E00000E00000E00000E00000E000 00E00000E00000E00000E00000E00000E00003C000FFC000FF00007C000013247E9F18> 125 D E /Fl 70 123 df<007E1F0001C1B1800303E3C00703C3C00E03C1800E01C0000E 01C0000E01C0000E01C0000E01C0000E01C000FFFFFC000E01C0000E01C0000E01C0000E 01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E 01C0000E01C0000E01C0000E01C0007F87FC001A1D809C18>11 D<007E0001C180030180 0703C00E03C00E01800E00000E00000E00000E00000E0000FFFFC00E01C00E01C00E01C0 0E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0 0E01C07F87F8151D809C17>I<007FC001C1C00303C00703C00E01C00E01C00E01C00E01 C00E01C00E01C00E01C0FFFFC00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01 C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C07FCFF8151D809C17>I<60 60F0F0F8F86868080808080808101010102020404080800D0C7F9C15>34 D<00E0000001900000030800000308000007080000070800000708000007080000071000 0007100000072000000740000003C03FE003800F00038006000380040005C0040009C008 0010E0100030E010006070200060702000E0384000E03C4000E01C8000E00F0020E00700 20700780403009C0401830E18007C03E001B1F7E9D20>38 D<004000800100020006000C 000C0018001800300030007000600060006000E000E000E000E000E000E000E000E000E0 00E000E000E000600060006000700030003000180018000C000C00060002000100008000 400A2A7D9E10>40 D<800040002000100018000C000C0006000600030003000380018001 80018001C001C001C001C001C001C001C001C001C001C001C001C0018001800180038003 000300060006000C000C00180010002000400080000A2A7E9E10>I<60F0F07010101010 20204080040C7C830C>44 DI<60F0F06004047C830C>I<03C00C 301818300C300C700E60066006E007E007E007E007E007E007E007E007E007E007E007E0 07E00760066006700E300C300C18180C3007E0101D7E9B15>48 D<030007003F00C70007 000700070007000700070007000700070007000700070007000700070007000700070007 000700070007000F80FFF80D1C7C9B15>I<07C01830201C400C400EF00FF80FF807F807 7007000F000E000E001C001C00380070006000C00180030006010C01180110023FFE7FFE FFFE101C7E9B15>I<07E01830201C201C781E780E781E381E001C001C00180030006007 E00030001C001C000E000F000F700FF80FF80FF80FF00E401C201C183007E0101D7E9B15 >I<000C00000C00001C00003C00003C00005C0000DC00009C00011C00031C00021C0004 1C000C1C00081C00101C00301C00201C00401C00C01C00FFFFC0001C00001C00001C0000 1C00001C00001C00001C0001FFC0121C7F9B15>I<300C3FF83FF03FC020002000200020 002000200023E024302818301C200E000E000F000F000F600FF00FF00FF00F800E401E40 1C2038187007C0101D7E9B15>I<00F0030C06040C0E181E301E300C700070006000E3E0 E430E818F00CF00EE006E007E007E007E007E007600760077006300E300C18180C3003E0 101D7E9B15>I<4000007FFF807FFF007FFF004002008004008004008008000010000010 0000200000600000400000C00000C00001C0000180000180000380000380000380000380 00078000078000078000078000078000078000030000111D7E9B15>I<03E00C30100820 0C20066006600660067006780C3E083FB01FE007F007F818FC307E601E600FC007C003C0 03C003C00360026004300C1C1007E0101D7E9B15>I<03C00C301818300C700C600EE006 E006E007E007E007E007E0076007700F300F18170C2707C700060006000E300C780C7818 7010203030C00F80101D7E9B15>I<60F0F0600000000000000000000060F0F06004127C 910C>I<60F0F0600000000000000000000060F0F0701010101020204080041A7C910C>I< 7FFFFFC0FFFFFFE000000000000000000000000000000000000000000000000000000000 00000000FFFFFFE07FFFFFC01B0C7E8F20>61 D<0FE03038401CE00EF00EF00EF00E000C 001C0030006000C000800180010001000100010001000100000000000000000000000300 0780078003000F1D7E9C14>63 D<000600000006000000060000000F0000000F0000000F 00000017800000178000001780000023C0000023C0000023C0000041E0000041E0000041 E0000080F0000080F0000180F8000100780001FFF80003007C0002003C0002003C000600 3E0004001E0004001E000C001F001E001F00FF80FFF01C1D7F9C1F>65 DI<001F808000E061800180198007000780 0E0003801C0003801C00018038000180780000807800008070000080F0000000F0000000 F0000000F0000000F0000000F0000000F0000000F0000000700000807800008078000080 380000801C0001001C0001000E000200070004000180080000E03000001FC000191E7E9C 1E>IIII<001F808000E0618001801980070007800E000380 1C0003801C00018038000180780000807800008070000080F0000000F0000000F0000000 F0000000F0000000F0000000F000FFF0F0000F8070000780780007807800078038000780 1C0007801C0007800E00078007000B800180118000E06080001F80001C1E7E9C21>II I<1FFF00F800780078007800780078007800780078007800780078007800780078007800 780078007800787078F878F878F878F0F040E021C01F00101D7F9B15>I76 DII<003F800000E0E000 0380380007001C000E000E001C0007003C00078038000380780003C0780003C0700001C0 F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0700001C0 780003C0780003C0380003803C0007801C0007000E000E0007001C000380380000E0E000 003F80001B1E7E9C20>II82 D<07E0801C1980300580700380600180E00180E00080E00080E000 80F00000F800007C00007FC0003FF8001FFE0007FF0000FF80000F800007C00003C00001 C08001C08001C08001C0C00180C00180E00300D00200CC0C0083F800121E7E9C17>I<7F FFFFC0700F01C0600F00C0400F0040400F0040C00F0020800F0020800F0020800F002000 0F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F000000 0F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000001F800003 FFFC001B1C7F9B1E>II<0808101020204040404080 8080808080B0B0F8F8787830300D0C7A9C15>92 D<1FC000307000783800781C00301C00 001C00001C0001FC000F1C00381C00701C00601C00E01C40E01C40E01C40603C40304E80 1F870012127E9115>97 DI<07E00C 301878307870306000E000E000E000E000E000E00060007004300418080C3007C00E127E 9112>I<003F000007000007000007000007000007000007000007000007000007000007 0003E7000C1700180F00300700700700600700E00700E00700E00700E00700E00700E007 00600700700700300700180F000C370007C7E0131D7E9C17>I<03E00C301818300C700E 6006E006FFFEE000E000E000E00060007002300218040C1803E00F127F9112>I<00F801 8C071E061E0E0C0E000E000E000E000E000E00FFE00E000E000E000E000E000E000E000E 000E000E000E000E000E000E000E000E007FE00F1D809C0D>I<00038003C4C00C38C01C 3880181800381C00381C00381C00381C001818001C38000C300013C00010000030000018 00001FF8001FFF001FFF803003806001C0C000C0C000C0C000C06001803003001C0E0007 F800121C7F9215>II<18003C003C 0018000000000000000000000000000000FC001C001C001C001C001C001C001C001C001C 001C001C001C001C001C001C001C00FF80091D7F9C0C>I<00C001E001E000C000000000 000000000000000000000FE000E000E000E000E000E000E000E000E000E000E000E000E0 00E000E000E000E000E000E000E000E060E0F0C0F1C061803E000B25839C0D>III II<03F0000E1C0018060030 0300700380600180E001C0E001C0E001C0E001C0E001C0E001C060018070038030030018 06000E1C0003F00012127F9115>II<03C1000C3300180B 00300F00700700700700E00700E00700E00700E00700E00700E00700600700700700300F 00180F000C370007C700000700000700000700000700000700000700000700003FE0131A 7E9116>II<1F9030704030C010C010E010F8007F803FE00FF000F88038 8018C018C018E010D0608FC00D127F9110>I<04000400040004000C000C001C003C00FF E01C001C001C001C001C001C001C001C001C001C101C101C101C101C100C100E2003C00C 1A7F9910>IIII<7F8FF00F03800F0300 07020003840001C80001D80000F00000700000780000F800009C00010E00020E00060700 0403801E07C0FF0FF81512809116>II<7FFC7038603840 7040F040E041C003C0038007000F040E041C043C0C380870087038FFF80E127F9112>I E /Fm 17 119 df<78FCFCFCFC7800000000000078FCFCFCFC7806127D910D>58 D<00038000000380000007C0000007C0000007C000000FE000000FE000001FF000001BF0 00001BF0000031F8000031F8000061FC000060FC0000E0FE0000C07E0000C07E0001803F 0001FFFF0003FFFF8003001F8003001F8006000FC006000FC00E000FE00C0007E0FFC07F FEFFC07FFE1F1C7E9B24>65 D<001FE02000FFF8E003F80FE007C003E00F8001E01F0000 E03E0000E03E0000607E0000607C000060FC000000FC000000FC000000FC000000FC0000 00FC000000FC000000FC0000007C0000607E0000603E0000603E0000C01F0000C00F8001 8007C0030003F80E0000FFFC00001FE0001B1C7D9B22>67 DI< 07F8201FFEE03C07E07801E07000E0F000E0F00060F00060F80000FE0000FFE0007FFE00 3FFF003FFF800FFFC007FFE0007FE00003F00001F00000F0C000F0C000F0C000E0E000E0 F001C0FC03C0EFFF0083FC00141C7D9B1B>83 D<0FF8001C1E003E0F803E07803E07C01C 07C00007C0007FC007E7C01F07C03C07C07C07C0F807C0F807C0F807C0780BC03E13F80F E1F815127F9117>97 D<03FC000E0E001C1F003C1F00781F00780E00F80000F80000F800 00F80000F80000F800007800007801803C01801C03000E0E0003F80011127E9115>99 D<01FC000F07001C03803C01C07801C07801E0F801E0F801E0FFFFE0F80000F80000F800 007800007C00603C00601E00C00F038001FC0013127F9116>101 D<1E003F003F003F003F001E00000000000000000000000000FF00FF001F001F001F001F 001F001F001F001F001F001F001F001F001F001F00FFE0FFE00B1E7F9D0E>105 D108 D110 D<01FC000F07801C01C03C01E07800F07800F0F800F8F800F8F800F8F800F8F800F8F800 F87800F07800F03C01E01E03C00F078001FC0015127F9118>I114 D<1FD830786018E018E018F000FF807FE07FF01F F807FC007CC01CC01CE01CE018F830CFC00E127E9113>I<030003000300030007000700 0F000F003FFCFFFC1F001F001F001F001F001F001F001F001F001F0C1F0C1F0C1F0C0F08 079803F00E1A7F9913>II< FFC1FCFFC1FC1F00601F80E00F80C00FC0C007C18007C18003E30003E30001F60001F600 01FE0000FC0000FC0000780000780000300016127F9119>I E /Fn 53 123 df<00000F80003F8F80007F8F8000FF8F8001FF8F8003E0000003C0000007C000 0007C0000007C0000007C0000007C0000007C0000007C0000007C0000007C00000FFFF8F 80FFFF8F80FFFF8F8007C00F8007C00F8007C00F8007C00F8007C00F8007C00F8007C00F 8007C00F8007C00F8007C00F8007C00F8007C00F8007C00F8007C00F8007C00F8007C00F 8007C00F8007C00F8007C00F8007C00F8007C00F8007C00F8007C00F8007C00F80192B7F AA20>12 D45 DI<007F000001 FFC00007FFF0000FFFF8000FC1F8001F007C003F007E003E003E003C001E007C001F007C 001F007C001F0078000F00F8000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8 000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8000F8078 000F007C001F007C001F007C001F003E003E003E003E003F007E001F80FC000FC1F8000F FFF80007FFF00001FFC000007F000019297EA71E>48 D<00180000380000F80007F800FF F800FFF800F8F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000 F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000 F80000F80000F80000F80000F80000F80000F80000F80000F8007FFFF07FFFF07FFFF014 287BA71E>I<00FE0003FFC007FFE00FFFF01F03F83C00FC38007E78003E70003EF0001F F0001F60001F20001F00001F00001F00001F00003E00003E00007C00007C0000F80001F0 0001E00003C0000780000F00001E00003C0000780000F00001E00003C0000780000F0000 1E00003C00007FFFFF7FFFFF7FFFFF7FFFFF18287EA71E>I<007F000001FFC00007FFF0 000FFFF8001FC1F8003E007C003C003E0078003E0038003E0010003E0000003E0000003E 0000003C0000007C000000FC000001F8000007F00000FFE00000FFC00000FFE00000FFF0 000001FC0000007C0000003E0000001F0000001F0000000F8000000F8000000F8000000F 8000000F8040000F8060001F00F0001F00F8003F007E007E003F81FC001FFFF8000FFFF0 0003FFE000007F000019297EA71E>I<0003F0000007F0000005F000000DF000000DF000 001DF0000039F0000039F0000079F0000079F00000F1F00000F1F00001E1F00003E1F000 03E1F00007C1F00007C1F0000F81F0000F81F0001F01F0001F01F0003E01F0007C01F000 7C01F000F801F000FFFFFF80FFFFFF80FFFFFF80FFFFFF800001F0000001F0000001F000 0001F0000001F0000001F0000001F0000001F0000001F0000001F00019277EA61E>I<3F FFFC3FFFFC3FFFFC3FFFFC3E00003E00003E00003E00003E00003E00003E00003E00003E 00003E00003E3F003EFFC03FFFE03FFFF03FE1F83F807C3F003E3E003E00003E00001F00 001F00001F00001F00001F00001F00001F20001F60003E70003EF8007C7C00FC3F03F81F FFF00FFFE007FF8000FE0018287EA61E>I<000FF000003FFC0000FFFC0001FFFC0003F8 0C0007E000000FC000000F8000001F0000001E0000003E0000003C0000007C0000007C00 00007C3FE000F8FFF000F9FFF800FBFFFC00FF807E00FF003E00FE003F00FC001F00FC00 1F00FC000F80F8000F80F8000F80F8000F80F8000F8078000F807C000F807C000F807C00 0F003E001F003E001F001F003E001F807C000FC1FC0007FFF80003FFF00001FFC000007F 000019297EA71E>II<007F000001FFC000 07FFF0000FFFF8001FC1FC003F007E003E003E007E003F007C001F007C001F007C001F00 7C001F007C001F003E003E003E003E001F007C000FC1F80007FFF00003FFE00003FFE000 0FFFF8001FC1FC003F007E003E003E007C001F007C001F00F8000F80F8000F80F8000F80 F8000F80F8000F80F8000F807C001F007C001F007E003F003F007E001FC1FC000FFFF800 07FFF00003FFE000007F000019297EA71E>I<007F000001FFC00003FFE0000FFFF0000F C1F8001F007C003E007C007C003E007C001E007C001F00F8001F00F8001F00F8000F00F8 000F80F8000F80F8000F80F8000F80F8001F807C001F807C001F807E003F803E007F803F 00FF801FFFEF800FFFCF8007FF8F8003FE1F0000001F0000001F0000001E0000003E0000 003E0000007C0000007C000000F8001801F0001E07E0003FFFC0001FFF80000FFE000003 F8000019297EA71E>I<0001F000000003F800000003F800000007FC00000007BC000000 07BC0000000F3E0000000F1E0000000F1E0000001F1F0000001E1F0000001E0F0000003E 0F8000003C0F8000003C078000007C07C000007807C00000F803E00000F803E00000F003 E00001F001F00001F001F00001E001F00003E000F80003E000F80003C000F80007FFFFFC 0007FFFFFC000FFFFFFE000F80003E000F80003E001F00003F001F00001F001E00001F00 3E00000F803E00000F803C00000F807C000007C07C000007C078000007C0F8000003E0F8 000003E0232A7EA928>65 D<0001FF00000FFFE0003FFFF8007FFFF800FE01F801F80030 03F0001007C000000F8000001F8000001F0000003E0000003E0000007E0000007C000000 7C0000007C000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000 F8000000F8000000F80000007C0000007C0000007C0000007E0000003E0000003E000000 1F0000001F8000000F80000007C0000003F0000401F8001C00FE00FC007FFFFC003FFFF8 000FFFE00001FF001E2C7CAA26>67 DIII72 DI75 D77 DI<0001FC0000000FFF8000003FFFE000 007FFFF00001FE03FC0003F800FE0007E0003F0007C0001F000F80000F801F000007C01F 000007C03E000003E03E000003E07C000001F07C000001F07C000001F078000000F0F800 0000F8F8000000F8F8000000F8F8000000F8F8000000F8F8000000F8F8000000F8F80000 00F8F8000000F8F8000000F87C000001F07C000001F07C000001F07E000003F03E000003 E03F000007E01F000007C01F80000FC00FC0001F8007E0003F0007F0007F0003F800FE00 01FE03FC0000FFFFF800003FFFE000000FFF80000001FC0000252C7DAA2C>II82 D<007FC00001FFF80007FFFE000F FFFF001FC07F003F000F007E0006007C0000007C000000F8000000F8000000F8000000F8 000000F8000000FC0000007E0000007F0000003F8000001FF800000FFF800007FFE00003 FFF80000FFFC00000FFE000000FF0000003F0000001F8000000F8000000FC0000007C000 0007C0000007C0000007C0000007C0000007C000000F8060000F80F0001F00FC003F00FF 80FE007FFFFC001FFFF80007FFE00000FF80001A2C7DAA21>II86 DI<01FE000FFF803FFFC03FFFE03C03F03001F00001F800 00F80000F80000F80000F80000F8007FF807FFF81FFFF83FE0F87F00F8FC00F8F800F8F8 00F8F800F8FC01F87E07F87FFFF83FFFF81FFCF80FE0F8151B7E9A1D>97 DI<007FC001FFF007FFFC0F FFFC1FC07C1F00083E00007C00007C00007C0000F80000F80000F80000F80000F80000F8 0000F800007C00007C00007E00003E00001F000C1FC07C0FFFFC07FFFC01FFF0007F8016 1B7E9A1B>I<00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E 00003E00003E00003E00003E00003E00FC3E03FF3E07FFFE0FFFFE1FC1FE3F007E3E003E 7C003E7C003EFC003EF8003EF8003EF8003EF8003EF8003EF8003EF8003EFC003E7C003E 7C003E3E007E3F00FE1FC1FE0FFFFE07FFBE03FF3E00FC3E172A7EA91F>I<007E0003FF 8007FFC00FFFE01F83F03F00F03E00787C00787C003878003CFFFFFCFFFFFCFFFFFCFFFF FCF80000F80000F800007800007C00007C00003E00003F000C1FC07C0FFFFC07FFFC01FF F0007F80161B7E9A1B>I<001FC0007FC000FFC001FFC003F00003E00007C00007C00007 C00007C00007C00007C00007C00007C00007C000FFFE00FFFE00FFFE0007C00007C00007 C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007 C00007C00007C00007C00007C00007C00007C00007C00007C00007C000122A7FA912>I< 00F8078003FE7FC00FFFFFC01FFFFFC01F07C0003E03E0003E03E0007C01F0007C01F000 7C01F0007C01F0007C01F0007C01F0003E03E0003E03E0001F07C0001FFFC0003FFF8000 3BFE000038F8000078000000780000003C0000003FFFC0003FFFF8001FFFFC001FFFFE00 3FFFFF007C007F00F8001F80F8000F80F8000F80F8000F80FC001F807E003F003F80FE00 3FFFFE000FFFF80007FFF00000FF80001A287E9A1E>III108 DII<007F000001FFC00007FFF0000FFFF8001FC1 FC003F007E003E003E007C001F007C001F0078000F00F8000F80F8000F80F8000F80F800 0F80F8000F80F8000F80F8000F807C001F007C001F007E003F003E003E003F007E001FC1 FC000FFFF80007FFF00001FFC000007F0000191B7E9A1E>II114 D<03FC001FFF803FFFC07FFFC07C07C0F80080F80000F80000F80000FC00007F80007FF8 003FFE001FFF0007FF8000FFC0000FE00007E00003E00003E04003E0E007E0FC0FC0FFFF C07FFF801FFE0003F800131B7E9A17>I<07C00007C00007C00007C00007C00007C00007 C000FFFFC0FFFFC0FFFFC007C00007C00007C00007C00007C00007C00007C00007C00007 C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C04007E1C003 FFE003FFE001FF8000FC0013227FA116>IIII<7C000FC03E001F803F001F001F803E000F807C0007C0FC0003E0F80001F1F00001 FBE00000FFC000007FC000003F8000001F0000001F0000003F8000007FC00000FBC00000 F3E00001F1F00003E0F80007C07C000F807C000F803E001F001F003E000F807E000FC0FC 0007E01B1B809A1C>III E /Fo 15 122 df<07E00FF03FFC3FFC7FFEFFFFFFFFFFFFFFFFFF FFFFFF7FFE3FFC3FFC0FF007E01010788F21>46 D<000001E00000000003F0000000000F F0000000003FF000000000FFF00000000FFFF0000003FFFFF00000FFFFFFF00000FFFFFF F00000FFFFFFF00000FFF0FFF00000FC00FFF000000000FFF000000000FFF000000000FF F000000000FFF000000000FFF000000000FFF000000000FFF000000000FFF000000000FF F000000000FFF000000000FFF000000000FFF000000000FFF000000000FFF000000000FF F000000000FFF000000000FFF000000000FFF000000000FFF000000000FFF000000000FF F000000000FFF000000000FFF000000000FFF000000000FFF000000000FFF000000000FF F000000000FFF000000000FFF000000000FFF000000000FFF000000000FFF000000000FF F000000000FFF000000000FFF000000000FFF000000000FFF000000000FFF000000000FF F000000000FFF000000000FFF000000000FFF000000000FFF000000000FFF000000000FF F000000000FFF000000000FFF000000000FFF000000000FFF000000000FFF000000000FF F000003FFFFFFFFFC03FFFFFFFFFC03FFFFFFFFFC03FFFFFFFFFC03FFFFFFFFFC02A4478 C33B>49 D<0000FFE00000000FFFFE0000003FFFFFC00000FFFFFFF00003FFFFFFFC0007 FC01FFFE000FE0007FFF001FC0001FFF803F80000FFFC03FE00007FFE07FF80007FFE07F FC0003FFF0FFFC0003FFF0FFFE0001FFF8FFFE0001FFF8FFFE0001FFF8FFFE0000FFFCFF FE0000FFFC7FFC0000FFFC7FFC0000FFFC3FF80000FFFC1FF00000FFFC0FE00000FFFC00 000001FFFC00000001FFF800000001FFF800000001FFF000000003FFF000000003FFE000 000003FFE000000007FFC000000007FF800000000FFF800000001FFF000000001FFE0000 00003FFC000000007FF8000000007FE000000000FFC000000001FF8000000003FF000000 0007FC000000000FF8000000001FF0000000003FC0000000007F80007C0000FF00007C00 00FE00007C0001FC00007C0003F00000F80007E00000F8000FC00000F8001F800000F800 3F000001F8007C000003F800FFFFFFFFF801FFFFFFFFF003FFFFFFFFF007FFFFFFFFF00F FFFFFFFFF01FFFFFFFFFF03FFFFFFFFFF07FFFFFFFFFF07FFFFFFFFFF0FFFFFFFFFFE0FF FFFFFFFFE0FFFFFFFFFFE0FFFFFFFFFFE02E447AC33B>I77 D<0007FFFC000000007FFFFFC0000001FFFF FFF8000003FFFFFFFE000007FE001FFF000007FF0003FFC0000FFF8001FFE0000FFF8000 FFF0000FFF80007FF0000FFF80007FF8000FFF80007FF80007FF00003FFC0007FF00003F FC0003FE00003FFC0000F800003FFC00000000003FFC00000000003FFC00000000003FFC 00000000003FFC00000007FFFFFC000000FFFFFFFC000007FFFFFFFC00003FFFE03FFC00 00FFFE003FFC0003FFF0003FFC0007FFC0003FFC000FFF00003FFC001FFE00003FFC003F FC00003FFC007FF800003FFC007FF800003FFC00FFF000003FFC00FFF000003FFC00FFF0 00003FFC00FFF000003FFC00FFF000003FFC00FFF000007FFC007FF80000FFFC007FF800 01EFFC003FFC0003EFFC003FFF0007CFFF000FFFC03F8FFFF807FFFFFF07FFFC01FFFFFC 03FFFC007FFFF001FFFC0003FF80007FF8362E7DAD3A>97 D<00001FFFC0000000FFFFF8 000007FFFFFE00001FFFFFFF80007FFC00FFC000FFE001FFC001FFC003FFE003FF8003FF E007FF0003FFE00FFE0003FFE00FFE0003FFE01FFC0001FFC01FFC0001FFC03FFC0000FF 803FFC00003E007FF8000000007FF8000000007FF800000000FFF800000000FFF8000000 00FFF800000000FFF800000000FFF800000000FFF800000000FFF800000000FFF8000000 00FFF800000000FFF8000000007FF8000000007FF8000000007FFC000000003FFC000000 003FFC000000001FFC000000F81FFE000000F80FFE000000F80FFF000001F007FF800003 F003FFC00007E001FFE0000FC000FFF0001F80007FFE00FF00001FFFFFFE000007FFFFF8 000000FFFFE00000001FFE00002D2E7CAD35>99 D<00001FFE00000001FFFFE0000007FF FFF800001FFFFFFE00007FFC07FF0000FFE001FF8001FFC0007FC003FF80003FE007FF00 003FF00FFE00001FF01FFE00000FF81FFC00000FF83FFC00000FFC3FFC000007FC7FFC00 0007FC7FF8000007FC7FF8000007FE7FF8000007FEFFF8000007FEFFF8000007FEFFFFFF FFFFFEFFFFFFFFFFFEFFFFFFFFFFFEFFFFFFFFFFFCFFF800000000FFF800000000FFF800 000000FFF8000000007FF8000000007FF8000000007FFC000000003FFC000000003FFC00 0000003FFC0000001C1FFE0000003E0FFE0000003E07FF0000007E07FF000000FC03FF80 0001F801FFC00003F0007FF0001FE0003FFE00FFC0001FFFFFFF800007FFFFFE000000FF FFF80000000FFF80002F2E7DAD36>101 D<000000FFC000000007FFF80000003FFFFC00 0000FFFFFF000001FFC1FF000007FF03FF80000FFC03FF80000FF807FFC0001FF807FFC0 003FF007FFC0003FF007FFC0003FE003FF80007FE003FF80007FE001FF00007FE000FE00 007FE0003800007FE0000000007FE0000000007FE0000000007FE0000000007FE0000000 007FE0000000007FE0000000007FE0000000007FE0000000007FE0000000FFFFFFFE0000 FFFFFFFE0000FFFFFFFE0000FFFFFFFE0000FFFFFFFE0000007FF0000000007FF0000000 007FF0000000007FF0000000007FF0000000007FF0000000007FF0000000007FF0000000 007FF0000000007FF0000000007FF0000000007FF0000000007FF0000000007FF0000000 007FF0000000007FF0000000007FF0000000007FF0000000007FF0000000007FF0000000 007FF0000000007FF0000000007FF0000000007FF0000000007FF0000000007FF0000000 007FF0000000007FF0000000007FF0000000007FF0000000007FF0000000007FF0000000 007FF0000000007FF0000000007FF0000000007FF00000003FFFFFF800003FFFFFF80000 3FFFFFF800003FFFFFF800003FFFFFF800002A487DC724>I<00FC0001FE0003FF0007FF 800FFFC01FFFE01FFFE01FFFE01FFFE01FFFE01FFFE00FFFC007FF8003FF0001FE0000FC 00000000000000000000000000000000000000000000000000000000000000000000007F C0FFFFC0FFFFC0FFFFC0FFFFC0FFFFC003FFC001FFC001FFC001FFC001FFC001FFC001FF C001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FF C001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FF C001FFC001FFC001FFC001FFC0FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF18497CC820>105 D<007FC000FFFFC000FFFFC000FFFFC000FFFFC000FFFFC00003FFC00001FFC00001FFC0 0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 0001FFC00001FFC00001FFC00001FFC000FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF 8019487CC720>108 D<007FC001FFC00000FFFFC00FFFF80000FFFFC03FFFFE0000FFFF C0FFFFFF0000FFFFC1FC07FF8000FFFFC3E003FFC00003FFC7C001FFC00001FFCF0001FF E00001FFDE0000FFE00001FFDC0000FFE00001FFFC0000FFF00001FFF80000FFF00001FF F00000FFF00001FFF00000FFF00001FFF00000FFF00001FFE00000FFF00001FFE00000FF F00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FF E00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FF F00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FF E00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FF F00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FF E00000FFF00001FFE00000FFF000FFFFFFC07FFFFFE0FFFFFFC07FFFFFE0FFFFFFC07FFF FFE0FFFFFFC07FFFFFE0FFFFFFC07FFFFFE03B2E7CAD42>110 D<00000FFF0000000000 FFFFF000000007FFFFFE0000001FFFFFFF8000003FFC03FFC00000FFE0007FF00001FF80 001FF80003FF00000FFC0007FE000007FE000FFE000007FF000FFC000003FF001FFC0000 03FF803FFC000003FFC03FF8000001FFC03FF8000001FFC07FF8000001FFE07FF8000001 FFE07FF8000001FFE0FFF8000001FFF0FFF8000001FFF0FFF8000001FFF0FFF8000001FF F0FFF8000001FFF0FFF8000001FFF0FFF8000001FFF0FFF8000001FFF0FFF8000001FFF0 FFF8000001FFF07FF8000001FFE07FF8000001FFE07FF8000001FFE07FF8000001FFE03F FC000003FFC03FFC000003FFC01FFC000003FF801FFE000007FF800FFE000007FF0007FF 00000FFE0003FF80001FFC0001FFC0003FF80000FFE0007FF000007FFC03FFE000001FFF FFFF80000007FFFFFE00000000FFFFF0000000000FFF000000342E7DAD3B>I<00FF803F 8000FFFF80FFF000FFFF83FFFC00FFFF87FFFE00FFFF8FC3FF00FFFF8F07FF0003FF9E0F FF8001FFBC0FFF8001FFB80FFF8001FFF80FFF8001FFF00FFF8001FFF007FF0001FFF007 FF0001FFE003FE0001FFE000F80001FFE000000001FFE000000001FFC000000001FFC000 000001FFC000000001FFC000000001FFC000000001FFC000000001FFC000000001FFC000 000001FFC000000001FFC000000001FFC000000001FFC000000001FFC000000001FFC000 000001FFC000000001FFC000000001FFC000000001FFC000000001FFC000000001FFC000 000001FFC000000001FFC000000001FFC000000001FFC0000000FFFFFFE00000FFFFFFE0 0000FFFFFFE00000FFFFFFE00000FFFFFFE00000292E7CAD31>114 D<000FFF00E0007FFFF3E001FFFFFFE007FFFFFFE00FF800FFE01FC0001FE03F80000FE0 3F000007E07F000003E07F000003E0FF000003E0FF000003E0FF800003E0FFC0000000FF F0000000FFFE000000FFFFF800007FFFFFC0007FFFFFF0003FFFFFFC001FFFFFFF000FFF FFFF8007FFFFFFC003FFFFFFE000FFFFFFF0003FFFFFF00003FFFFF800001FFFF8000000 FFFC0000001FFC7800000FFCF8000007FCF8000003FCFC000003FCFC000003FCFE000003 F8FE000003F8FF000003F8FF800007F0FFC0000FF0FFF0001FE0FFFC00FFC0FFFFFFFF80 FC7FFFFE00F81FFFF800E003FF8000262E7CAD2F>I<7FFFFFC000FFFF807FFFFFC000FF FF807FFFFFC000FFFF807FFFFFC000FFFF807FFFFFC000FFFF8000FFF000000FE00000FF F800000FC00000FFF800000FC000007FFC00000F8000007FFC00001F8000003FFC00001F 0000003FFE00003F0000001FFE00003E0000001FFF00007E0000000FFF00007C0000000F FF8000FC00000007FF8000F800000007FFC001F800000003FFC001F000000003FFE003F0 00000003FFE003F000000001FFF003E000000001FFF007E000000000FFF007C000000000 FFF80FC0000000007FF80F80000000007FFC1F80000000003FFC1F00000000003FFE3F00 000000001FFE3E00000000001FFF7E00000000000FFF7C00000000000FFFFC0000000000 0FFFFC000000000007FFF8000000000007FFF8000000000003FFF0000000000003FFF000 0000000001FFE0000000000001FFE0000000000000FFC0000000000000FFC00000000000 007F800000000000007F800000000000003F000000000000003F000000000000003F0000 00000000003E000000000000007E000000000000007C00000000000000FC000000001F80 00F8000000003FC001F8000000007FE001F000000000FFF003F000000000FFF003E00000 0000FFF007E000000000FFF00FC000000000FFF01F8000000000FFF03F80000000007FE0 7F00000000007F43FE00000000003FFFF800000000001FFFF0000000000007FFC0000000 000001FE00000000000039427EAD3F>121 D E /Fp 8 117 df<0003FF8000001FFFF000 007FFFFE0000FE03FF0001F000FF8003C000FFC00780007FE00FF0007FF00FF8007FF01F FC007FF81FFE007FF81FFE007FF81FFE007FF81FFE007FF81FFE007FF80FFC007FF007F8 007FF003F0007FF0000000FFE0000000FFC0000001FF80000001FF00000003FE00000007 FC0000001FF000000FFFC000000FFF8000000FFFF800000003FE00000000FF800000007F E00000003FF00000003FF80000003FFC0000001FFC0000001FFE0000001FFE0200001FFF 1FC0001FFF3FE0001FFF7FF0001FFF7FF0001FFFFFF8001FFFFFF8001FFFFFF8001FFEFF F8001FFEFFF0001FFE7FF0003FFC7FE0003FFC3FC0003FF81F80007FF01FE000FFE007FC 03FFC003FFFFFF0001FFFFFE00003FFFF0000007FF800028397CB731>51 D<0000001FFF000030000001FFFFE000F000000FFFFFFC01F000007FFFFFFE03F00001FF FE007F87F00003FFE0000FCFF0000FFF000003FFF0001FFC000001FFF0003FF80000007F F0007FF00000003FF000FFC00000003FF001FFC00000001FF003FF800000000FF007FF00 0000000FF00FFF0000000007F00FFE0000000007F01FFE0000000003F01FFE0000000003 F03FFC0000000003F03FFC0000000001F03FFC0000000001F07FFC0000000001F07FF800 00000001F07FF80000000000007FF8000000000000FFF8000000000000FFF80000000000 00FFF8000000000000FFF8000000000000FFF8000000000000FFF8000000000000FFF800 0000000000FFF8000000000000FFF8000000000000FFF8000000000000FFF80000000000 007FF80000000000007FF80000000000007FF80000000000007FFC0000000000F03FFC00 00000000F03FFC0000000000F03FFC0000000000F01FFE0000000000F01FFE0000000001 E00FFE0000000001E00FFF0000000001E007FF0000000003C003FF8000000003C001FFC0 000000078000FFE00000000F00007FF00000001F00003FF80000003E00001FFC0000007C 00000FFF000001F8000003FFE00007F0000001FFFE003FC00000007FFFFFFF000000000F FFFFFC0000000001FFFFF000000000001FFF0000003C3D7BBB47>67 D<001FFF00000001FFFFF0000003FFFFFC000007F007FE00000FF801FF00001FFC00FF80 001FFC007FC0001FFC007FE0001FFC003FE0000FF8003FF0000FF8003FF00007F0003FF0 0001C0003FF0000000003FF0000000003FF0000000003FF0000000FFFFF000000FFFFFF0 00007FF83FF00001FF803FF00007FE003FF0000FF8003FF0001FF0003FF0003FE0003FF0 007FE0003FF0007FE0003FF000FFC0003FF000FFC0003FF000FFC0003FF000FFC0003FF0 00FFC0007FF0007FE0007FF0007FE000DFF0003FF0039FF8001FFC0F0FFFF007FFFE0FFF F001FFFC07FFF0003FE000FFF02C267DA530>97 D<0001FFC000000FFFF800003FFFFE00 00FF80FF0001FE003F8007FC001FC00FF8000FE00FF8000FF01FF00007F03FF00007F83F F00007F87FE00007F87FE00003FC7FE00003FC7FE00003FCFFE00003FCFFFFFFFFFCFFFF FFFFFCFFFFFFFFFCFFE0000000FFE0000000FFE0000000FFE00000007FE00000007FE000 00007FE00000003FE00000003FF000003C1FF000003C1FF000003C0FF800007807FC0000 F803FE0001F001FF0007E000FFC03FC0003FFFFF000007FFFC000000FFE00026267DA52D >101 D<00FF00000000FFFF00000000FFFF00000000FFFF00000000FFFF0000000007FF 0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF 0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF 0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF007FC00003FF 01FFF80003FF07FFFC0003FF0F03FE0003FF1C01FF0003FF3001FF8003FF6000FF8003FF E000FFC003FFC000FFC003FF8000FFC003FF8000FFC003FF8000FFC003FF0000FFC003FF 0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF 0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF 0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF 0000FFC003FF0000FFC003FF0000FFC0FFFFFC3FFFFFFFFFFC3FFFFFFFFFFC3FFFFFFFFF FC3FFFFF303C7CBB37>104 D<00FF01FF8000FFFF0FFFF000FFFF3FFFFC00FFFFFE03FF 00FFFFF000FF8003FFC0007FC003FF80003FE003FF00003FF003FF00001FF803FF00001F FC03FF00000FFC03FF00000FFE03FF00000FFE03FF000007FE03FF000007FF03FF000007 FF03FF000007FF03FF000007FF03FF000007FF03FF000007FF03FF000007FF03FF000007 FF03FF000007FF03FF000007FE03FF000007FE03FF00000FFE03FF00000FFC03FF00000F FC03FF00001FF803FF00001FF803FF00003FF003FF80003FE003FFC0007FC003FFF001FF 8003FFFC07FF0003FF3FFFFC0003FF0FFFF00003FF01FF000003FF0000000003FF000000 0003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF000000 0003FF0000000003FF0000000003FF0000000003FF0000000003FF00000000FFFFFC0000 00FFFFFC000000FFFFFC000000FFFFFC00000030377DA537>112 D<00FE03F000FFFE0FFE00FFFE1FFF00FFFE3C3F80FFFE707FC007FE60FFE003FEE0FFE0 03FEC0FFE003FFC0FFE003FF807FC003FF807FC003FF803F8003FF800E0003FF00000003 FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF 00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00 000003FF00000003FF00000003FF00000003FF00000003FF000000FFFFFE0000FFFFFE00 00FFFFFE0000FFFFFE000023267DA529>114 D<00078000000780000007800000078000 00078000000F8000000F8000000F8000000F8000001F8000001F8000003F8000003F8000 007F800000FF800001FF800007FF80001FFFFFF0FFFFFFF0FFFFFFF0FFFFFFF001FF8000 01FF800001FF800001FF800001FF800001FF800001FF800001FF800001FF800001FF8000 01FF800001FF800001FF800001FF800001FF800001FF800001FF800001FF800001FF8000 01FF803C01FF803C01FF803C01FF803C01FF803C01FF803C01FF803C01FF803C00FF8078 00FFC078007FC070003FE0E0001FFFC00007FF800001FF001E377EB626>116 D E /Fq 83 124 df<001F83E000F06E3001C078780380F8780300F03007007000070070 000700700007007000070070000700700007007000FFFFFF800700700007007000070070 000700700007007000070070000700700007007000070070000700700007007000070070 000700700007007000070070000700700007007000070070007FE3FF001D20809F1B>11 D<003F0000E0C001C0C00381E00701E00701E00700000700000700000700000700000700 00FFFFE00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700 E00700E00700E00700E00700E00700E00700E00700E07FC3FE1720809F19>I<003FE000 E0E001C1E00381E00700E00700E00700E00700E00700E00700E00700E00700E0FFFFE007 00E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E007 00E00700E00700E00700E00700E00700E07FE7FE1720809F19>I<001F81F80000F04F04 0001C07C06000380F80F000300F00F000700F00F00070070000007007000000700700000 070070000007007000000700700000FFFFFFFF0007007007000700700700070070070007 007007000700700700070070070007007007000700700700070070070007007007000700 700700070070070007007007000700700700070070070007007007000700700700070070 07007FE3FE3FF02420809F26>I<001F81FF0000F06F070001C07C0F000380F80F000300 F00700070070070007007007000700700700070070070007007007000700700700070070 0700FFFFFFFF000700700700070070070007007007000700700700070070070007007007 000700700700070070070007007007000700700700070070070007007007000700700700 070070070007007007000700700700070070070007007007007FE3FE3FF02420809F26> I<70F8F8F8F8F8F8F8707070707070707070702020202020000000000070F8F8F8700521 7CA00D>33 D<7038F87CFC7EFC7E743A0402040204020804080410081008201040200F0E 7E9F17>I<00780000008400000184000003020000070200000702000007020000070200 00070400000704000007080000070800000310000003A00FFC03C003E0038001C001C000 8001C0010003E0010004E0020008F00200187004003078080070380800701C1000F01E10 00F00E2000F0074000F003C0087003C0087801C010380670301C18386007E00F801E227E A023>38 D<70F8FCFC74040404080810102040060E7C9F0D>I<00200040008001000200 06000C000C00180018003000300030007000600060006000E000E000E000E000E000E000 E000E000E000E000E000E0006000600060007000300030003000180018000C000C000600 020001000080004000200B2E7DA112>I<800040002000100008000C0006000600030003 0001800180018001C000C000C000C000E000E000E000E000E000E000E000E000E000E000 E000E000C000C000C001C001800180018003000300060006000C00080010002000400080 000B2E7DA112>I<01800180018001800180C183F18F399C0FF003C003C00FF0399CF18F C1830180018001800180018010147DA117>I<0006000000060000000600000006000000 060000000600000006000000060000000600000006000000060000000600000006000000 06000000060000FFFFFFF0FFFFFFF0000600000006000000060000000600000006000000 060000000600000006000000060000000600000006000000060000000600000006000000 0600001C207D9A23>I<70F8FCFC74040404080810102040060E7C840D>II<70F8F8F87005057C840D>I<000100030003000600060006000C000C000C0018 0018001800300030003000600060006000C000C000C00180018001800300030003000600 060006000C000C000C00180018001800300030003000600060006000C000C000C000102D 7DA117>I<03F0000E1C001C0E00180600380700700380700380700380700380F003C0F0 03C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C070 03807003807003807807803807001806001C0E000E1C0003F000121F7E9D17>I<018003 800F80F38003800380038003800380038003800380038003800380038003800380038003 800380038003800380038003800380038007C0FFFE0F1E7C9D17>I<03F0000C1C00100E 00200700400780800780F007C0F803C0F803C0F803C02007C00007C0000780000780000F 00000E00001C0000380000700000600000C0000180000300000600400C00401800401000 803FFF807FFF80FFFF80121E7E9D17>I<03F0000C1C00100E00200F00780F8078078078 0780380F80000F80000F00000F00000E00001C0000380003F000003C00000E00000F0000 07800007800007C02007C0F807C0F807C0F807C0F00780400780400F00200E001C3C0003 F000121F7E9D17>I<000600000600000E00000E00001E00002E00002E00004E00008E00 008E00010E00020E00020E00040E00080E00080E00100E00200E00200E00400E00C00E00 FFFFF0000E00000E00000E00000E00000E00000E00000E0000FFE0141E7F9D17>I<1803 001FFE001FFC001FF8001FE00010000010000010000010000010000010000011F000161C 00180E001007001007800003800003800003C00003C00003C07003C0F003C0F003C0E003 80400380400700200600100E000C380003E000121F7E9D17>I<007C000182000701000E 03800C07801C0780380300380000780000700000700000F1F000F21C00F40600F80700F8 0380F80380F003C0F003C0F003C0F003C0F003C07003C07003C070038038038038070018 07000C0E00061C0001F000121F7E9D17>I<4000007FFFC07FFF807FFF80400100800200 80020080040000080000080000100000200000200000400000400000C00000C00001C000 018000038000038000038000038000078000078000078000078000078000078000078000 030000121F7D9D17>I<03F0000C0C001006003003002001806001806001806001807001 807803003E03003F06001FC8000FF00003F80007FC000C7E00103F00300F806003804001 C0C001C0C000C0C000C0C000C0C000806001802001001002000C0C0003F000121F7E9D17 >I<03F0000E18001C0C00380600380700700700700380F00380F00380F003C0F003C0F0 03C0F003C0F003C07007C07007C03807C0180BC00E13C003E3C000038000038000038000 0700300700780600780E00700C002018001070000FC000121F7E9D17>I<70F8F8F87000 00000000000000000070F8F8F87005147C930D>I<70F8F8F87000000000000000000000 70F0F8F878080808101010202040051D7C930D>I<7FFFFFE0FFFFFFF000000000000000 00000000000000000000000000000000000000000000000000FFFFFFF07FFFFFE01C0C7D 9023>61 D<0FC0307040384038E03CF03CF03C603C0038007000E000C001800180010003 000200020002000200020002000000000000000000000007000F800F800F8007000E207D 9F15>63 D<000100000003800000038000000380000007C0000007C0000007C0000009E0 000009E0000009E0000010F0000010F0000010F00000207800002078000020780000403C 0000403C0000403C0000801E0000801E0000FFFE0001000F0001000F0001000F00020007 800200078002000780040003C00E0003C01F0007E0FFC03FFE1F207F9F22>65 DI<000FC040007030C001C009C0 038005C0070003C00E0001C01E0000C01C0000C03C0000C07C0000407C00004078000040 F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000 780000007C0000407C0000403C0000401C0000401E0000800E0000800700010003800200 01C0040000703800000FC0001A217D9F21>IIII72 D I<0FFFC0007C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C 00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00203C 00F83C00F83C00F83C00F0380040780040700030E0000F800012207E9E17>I76 DII<001F800000F0 F00001C0380007801E000F000F000E0007001E0007803C0003C03C0003C07C0003E07800 01E0780001E0F80001F0F80001F0F80001F0F80001F0F80001F0F80001F0F80001F0F800 01F0F80001F0780001E07C0003E07C0003E03C0003C03C0003C01E0007800E0007000F00 0F0007801E0001C0380000F0F000001F80001C217D9F23>II82 D<07E0800C1980100780300380600180600180E00180E00080E00080E00080F000 00F000007800007F00003FF0001FFC000FFE0003FF00001F800007800003C00003C00001 C08001C08001C08001C08001C0C00180C00380E00300F00600CE0C0081F80012217D9F19 >I<7FFFFFE0780F01E0600F0060400F0020400F0020C00F0030800F0010800F0010800F 0010800F0010000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F 0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F 0000000F0000000F0000001F800007FFFE001C1F7E9E21>IIII<7FF83FF80FE00FC007C0 070003C0020001E0040001F00C0000F0080000781000007C1000003C2000003E4000001E 4000000F8000000F8000000780000003C0000007E0000005E0000009F0000018F8000010 780000207C0000603C0000401E0000801F0001800F0001000780020007C0070003C01F80 07E0FFE01FFE1F1F7F9E22>II< FEFEC0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0 C0C0C0C0C0C0C0FEFE072D7CA10D>91 D<080410082010201040204020804080408040B8 5CFC7EFC7E7C3E381C0F0E7B9F17>II<1FE00030 3000781800781C00300E00000E00000E00000E0000FE00078E001E0E00380E00780E00F0 0E10F00E10F00E10F01E10781E103867200F83C014147E9317>97 D<0E0000FE00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00 000E3E000EC3800F01C00F00E00E00E00E00700E00700E00780E00780E00780E00780E00 780E00780E00700E00700E00E00F00E00D01C00CC300083E0015207F9F19>I<03F80E0C 1C1E381E380C70007000F000F000F000F000F000F00070007000380138011C020E0C03F0 10147E9314>I<000380003F800003800003800003800003800003800003800003800003 8000038000038003E380061B801C0780380380380380700380700380F00380F00380F003 80F00380F00380F003807003807003803803803807801C07800E1B8003E3F815207E9F19 >I<03F0000E1C001C0E00380700380700700700700380F00380F00380FFFF80F00000F0 0000F000007000007000003800801800800C010007060001F80011147F9314>I<007C00 C6018F038F07060700070007000700070007000700FFF007000700070007000700070007 00070007000700070007000700070007000700070007007FF01020809F0E>I<0000E003 E3300E3C301C1C30380E00780F00780F00780F00780F00780F00380E001C1C001E380033 E0002000002000003000003000003FFE001FFF800FFFC03001E0600070C00030C00030C0 0030C000306000603000C01C038003FC00141F7F9417>I<0E0000FE00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E3E000E43000E81800F01C0 0F01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0 0E01C00E01C00E01C0FFE7FC16207F9F19>I<1C001E003E001E001C0000000000000000 00000000000E007E000E000E000E000E000E000E000E000E000E000E000E000E000E000E 000E000E000E00FFC00A1F809E0C>I<00E001F001F001F000E000000000000000000000 0000007007F000F000700070007000700070007000700070007000700070007000700070 00700070007000700070007000706070F060F0C061803F000C28829E0E>I<0E0000FE00 000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E0FF00E03 C00E03000E02000E04000E08000E10000E30000E70000EF8000F38000E1C000E1E000E0E 000E07000E07800E03800E03C00E03E0FFCFF815207F9F18>I<0E00FE000E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00 0E000E000E000E000E000E000E000E00FFE00B20809F0C>I<0E1F01F000FE618618000E 81C81C000F00F00E000F00F00E000E00E00E000E00E00E000E00E00E000E00E00E000E00 E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E0 0E000E00E00E000E00E00E00FFE7FE7FE023147F9326>I<0E3E00FE43000E81800F01C0 0F01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0 0E01C00E01C00E01C0FFE7FC16147F9319>I<01F800070E001C03803801C03801C07000 E07000E0F000F0F000F0F000F0F000F0F000F0F000F07000E07000E03801C03801C01C03 80070E0001F80014147F9317>I<0E3E00FEC3800F01C00F00E00E00E00E00F00E00700E 00780E00780E00780E00780E00780E00780E00700E00F00E00E00F01E00F01C00EC3000E 3E000E00000E00000E00000E00000E00000E00000E00000E0000FFE000151D7F9319>I< 03E0800619801C05803C0780380380780380700380F00380F00380F00380F00380F00380 F003807003807803803803803807801C0B800E138003E380000380000380000380000380 000380000380000380000380003FF8151D7E9318>I<0E78FE8C0F1E0F1E0F0C0E000E00 0E000E000E000E000E000E000E000E000E000E000E000E00FFE00F147F9312>I<1F9030 704030C010C010C010E00078007F803FE00FF00070803880188018C018C018E030D0608F 800D147E9312>I<020002000200060006000E000E003E00FFF80E000E000E000E000E00 0E000E000E000E000E000E000E080E080E080E080E080610031001E00D1C7F9B12>I<0E 01C0FE1FC00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E03C00603C0030DC001F1FC16147F9319>III<7FC3FC0F01E00701C007018003810001C20000E40000EC00007800003800003C 00007C00004E000087000107000303800201C00601E01E01E0FF07FE1714809318>II<3FFF380E200E201C40384078407000 E001E001C00380078007010E011E011C0338027006700EFFFE10147F9314>II E /Fr 46 122 df<00200040008001000300060004000C000C0018001800 3000300030007000600060006000E000E000E000E000E000E000E000E000E000E000E000 E000E000E0006000600060007000300030003000180018000C000C000400060003000100 0080004000200B327CA413>40 D<800040002000100018000C0004000600060003000300 01800180018001C000C000C000C000E000E000E000E000E000E000E000E000E000E000E0 00E000E000E000C000C000C001C0018001800180030003000600060004000C0018001000 2000400080000B327DA413>I<70F8FCFC7404040404080810102040060F7C840E>44 DI<70F8F8F87005057C840E>I<008003800F80F3800380038003 800380038003800380038003800380038003800380038003800380038003800380038003 8003800380038003800380038007C0FFFE0F217CA018>49 D<03F0000C1C001007002007 804003C04003C08003E0F003E0F801E0F801E0F801E02003E00003E00003C00003C00007 80000700000E00001C0000180000300000600000C0000180000100000200200400200800 201800603000403FFFC07FFFC0FFFFC013217EA018>I<000200000600000E00000E0000 1E00001E00002E00004E00004E00008E00008E00010E00020E00020E00040E00040E0008 0E00100E00100E00200E00200E00400E00800E00FFFFF8000E00000E00000E00000E0000 0E00000E00000E00001F0001FFF015217FA018>52 D<1000801E07001FFF001FFE001FF8 0013E00010000010000010000010000010000010000010F800130E001407001803801003 800001C00001C00001E00001E00001E00001E07001E0F001E0F001E0E001C08001C04003 C04003802007001006000C1C0003F00013227EA018>I<007E0001C1000300800601C00E 03C01C03C0180180380000380000780000700000700000F0F800F30C00F40600F40300F8 0380F801C0F001C0F001E0F001E0F001E0F001E0F001E07001E07001E07001E03801C038 01C01803801C03000C0600070C0001F00013227EA018>I<01F800060E00080300100180 2001802000C06000C06000C06000C07000C07801803E01003F02001FC4000FF80003F800 03FC00067F00083F80100F803007C06001C06000E0C000E0C00060C00060C00060C00060 6000406000C03000801803000E0E0003F00013227EA018>56 D<01F000060C000C060018 0700380380700380700380F001C0F001C0F001C0F001E0F001E0F001E0F001E0F001E070 01E07003E03803E01805E00C05E00619E003E1E00001C00001C00001C000038000038030 0300780700780600700C002018001030000FC00013227EA018>I<000180000001800000 0180000003C0000003C0000003C0000005E0000005E000000DF0000008F0000008F00000 10F800001078000010780000203C0000203C0000203C0000401E0000401E0000401E0000 800F0000800F0000FFFF000100078001000780030007C0020003C0020003C0040003E004 0001E0040001E00C0000F00C0000F03E0001F8FF800FFF20237EA225>65 D<0007E0100038183000E0063001C00170038000F0070000F00E0000701E0000701C0000 303C0000303C0000307C0000107800001078000010F8000000F8000000F8000000F80000 00F8000000F8000000F8000000F800000078000000780000107C0000103C0000103C0000 101C0000201E0000200E000040070000400380008001C0010000E0020000381C000007E0 001C247DA223>67 DIII72 DI77 DI80 D82 D<03F0200C0C601802603001E07000E0600060E00060E00060E00020 E00020E00020F00000F000007800007F00003FF0001FFE000FFF0003FF80003FC00007E0 0001E00000F00000F0000070800070800070800070800070C00060C00060E000C0F000C0 C80180C6070081FC0014247DA21B>I<7FFFFFF878078078600780184007800840078008 40078008C007800C80078004800780048007800480078004000780000007800000078000 000780000007800000078000000780000007800000078000000780000007800000078000 000780000007800000078000000780000007800000078000000780000007800000078000 000FC00003FFFF001E227EA123>I<0FE0001838003C0C003C0E00180700000700000700 00070000FF0007C7001E07003C0700780700700700F00708F00708F00708F00F08781708 3C23900FC1E015157E9418>97 D<0E0000FE00001E00000E00000E00000E00000E00000E 00000E00000E00000E00000E00000E00000E00000E1F000E61C00E80600F00300E00380E 003C0E001C0E001E0E001E0E001E0E001E0E001E0E001E0E001E0E001C0E003C0E00380F 00700C80600C41C0083F0017237FA21B>I<01FE000703000C07801C0780380300780000 700000F00000F00000F00000F00000F00000F00000F000007000007800403800401C0080 0C010007060001F80012157E9416>I<0000E0000FE00001E00000E00000E00000E00000 E00000E00000E00000E00000E00000E00000E00000E001F8E00704E00C02E01C01E03800 E07800E07000E0F000E0F000E0F000E0F000E0F000E0F000E0F000E07000E07800E03800 E01801E00C02E0070CF001F0FE17237EA21B>I<01FC000707000C03801C01C03801C078 01E07000E0F000E0FFFFE0F00000F00000F00000F00000F000007000007800203800201C 00400E008007030000FC0013157F9416>I<003C00C6018F038F030F0700070007000700 07000700070007000700FFF8070007000700070007000700070007000700070007000700 07000700070007000700070007807FF8102380A20F>I<00007001F198071E180E0E181C 07001C07003C07803C07803C07803C07801C07001C07000E0E000F1C0019F00010000010 00001800001800001FFE000FFFC00FFFE03800F0600030400018C00018C00018C0001860 00306000303800E00E038003FE0015217F9518>I<0E0000FE00001E00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E00000E1F800E60C00E80E0 0F00700F00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E0070 0E00700E00700E00700E00700E0070FFE7FF18237FA21B>I<1C001E003E001E001C0000 0000000000000000000000000000000E00FE001E000E000E000E000E000E000E000E000E 000E000E000E000E000E000E000E000E000E00FFC00A227FA10E>I<01C003E003E003E0 01C00000000000000000000000000000000001E00FE001E000E000E000E000E000E000E0 00E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E060E0 F0C0F18061803E000B2C82A10F>I<0E0000FE00001E00000E00000E00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E03FC0E01F00E01C00E01800E0200 0E04000E08000E10000E38000EF8000F1C000E1E000E0E000E07000E07800E03C00E01C0 0E01E00E00F00E00F8FFE3FE17237FA21A>I<0E1FC07F00FE60E183801E807201C00F00 3C00E00F003C00E00E003800E00E003800E00E003800E00E003800E00E003800E00E0038 00E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800 E00E003800E00E003800E0FFE3FF8FFE27157F942A>109 D<0E1F80FE60C01E80E00F00 700F00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00 700E00700E00700E00700E0070FFE7FF18157F941B>I<01FC000707000C01801800C038 00E0700070700070F00078F00078F00078F00078F00078F00078F000787000707800F038 00E01C01C00E038007070001FC0015157F9418>I<0E1F00FE61C00E80600F00700E0038 0E003C0E001C0E001E0E001E0E001E0E001E0E001E0E001E0E001E0E003C0E003C0E0038 0F00700E80E00E41C00E3F000E00000E00000E00000E00000E00000E00000E00000E0000 0E0000FFE000171F7F941B>I<0E3CFE461E8F0F0F0F060F000E000E000E000E000E000E 000E000E000E000E000E000E000E000F00FFF010157F9413>114 D<0F8830786018C018C008C008E008F0007F803FE00FF001F8003C801C800C800CC00CC0 08E018D0308FC00E157E9413>I<02000200020002000600060006000E001E003E00FFF8 0E000E000E000E000E000E000E000E000E000E000E000E040E040E040E040E040E040708 030801F00E1F7F9E13>I<0E0070FE07F01E00F00E00700E00700E00700E00700E00700E 00700E00700E00700E00700E00700E00700E00700E00700E00F00E00F006017003827800 FC7F18157F941B>I119 D121 D E /Fs 20 118 df45 D68 D73 D77 D80 D<007F802001FFE02007C078600F001C601E0006E03C0003E0380001E078 0000E0700000E070000060F0000060F0000060F0000020F0000020F0000020F8000020F8 0000007C0000007E0000003F0000003FC000001FF800000FFF800007FFF80003FFFC0000 FFFF00000FFF800000FFC000001FE0000007E0000003F0000001F0000000F0000000F800 0000F88000007880000078800000788000007880000078C0000078C0000070E00000F0E0 0000E0F00000E0F80001C0EC000380C7000700C1F01E00807FFC00800FF0001D337CB125 >83 D<00FE00000303C0000C00E00010007000100038003C003C003E001C003E001E003E 001E0008001E0000001E0000001E0000001E00000FFE0000FC1E0003E01E000F801E001F 001E003E001E003C001E007C001E00F8001E04F8001E04F8001E04F8003E04F8003E0478 003E047C005E043E008F080F0307F003FC03E01E1F7D9E21>97 D<003F8000E060038018 0700040F00041E001E1C003E3C003E7C003E7C0008780000F80000F80000F80000F80000 F80000F80000F80000F80000F800007800007C00007C00003C00011E00011E00020F0002 07000403801800E060003F80181F7D9E1D>99 D<000001E000003FE000003FE0000003E0 000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0 000001E0000001E0000001E0000001E0000001E0000001E0001F81E000F061E001C019E0 078005E00F0003E00E0003E01E0001E03C0001E03C0001E07C0001E0780001E0F80001E0 F80001E0F80001E0F80001E0F80001E0F80001E0F80001E0F80001E0F80001E0780001E0 780001E03C0001E03C0001E01C0001E01E0003E00E0005E0070009E0038011F000E061FF 003F81FF20327DB125>I<003F800000E0E0000380380007003C000E001E001E001E001C 000F003C000F007C000F0078000F8078000780F8000780F8000780FFFFFF80F8000000F8 000000F8000000F8000000F8000000F8000000780000007C0000003C0000003C0000801E 0000800E0001000F0002000780020001C00C0000F03000001FC000191F7E9E1D>I<0007 E0001C1000383800707C00E07C01E07C01C03803C00003C00003C00003C00003C00003C0 0003C00003C00003C00003C00003C00003C000FFFFC0FFFFC003C00003C00003C00003C0 0003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C0 0003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00007E0007FFF 007FFF0016327FB114>I<000000F0007F030801C1C41C0380E81C070070080F0078001E 003C001E003C003E003E003E003E003E003E003E003E003E003E003E003E001E003C001E 003C000F007800070070000780E00009C1C000087F000018000000180000001800000018 000000180000001C0000000E0000000FFFF80007FFFF0003FFFF800E000FC0180001E030 0000F070000070E0000038E0000038E0000038E0000038E0000038700000707000007038 0000E01C0001C00700070001C01C00003FE0001E2F7E9F21>I<07000F801F801F800F80 0700000000000000000000000000000000000000000000000780FF80FF800F8007800780 078007800780078007800780078007800780078007800780078007800780078007800780 07800780078007800FC0FFF8FFF80D307EAF12>105 D<0780FE001FC000FF83078060F0 00FF8C03C18078000F9001E2003C0007A001E4003C0007A000F4001E0007C000F8001E00 07C000F8001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E0007 8000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E000780 00F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000 F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0 001E000FC001F8003F00FFFC1FFF83FFF0FFFC1FFF83FFF0341F7E9E38>109 D<0780FE0000FF83078000FF8C03C0000F9001E00007A001E00007A000F00007C000F000 07C000F000078000F000078000F000078000F000078000F000078000F000078000F00007 8000F000078000F000078000F000078000F000078000F000078000F000078000F0000780 00F000078000F000078000F000078000F000078000F000078000F000078000F0000FC001 F800FFFC1FFF80FFFC1FFF80211F7E9E25>I<001FC00000F0780001C01C00070007000F 0007801E0003C01C0001C03C0001E03C0001E0780000F0780000F0780000F0F80000F8F8 0000F8F80000F8F80000F8F80000F8F80000F8F80000F8F80000F8780000F07C0001F03C 0001E03C0001E01E0003C01E0003C00F00078007800F0001C01C0000F07800001FC0001D 1F7E9E21>I<0783E0FF8C18FF907C0F907C07A07C07C03807C00007C00007C000078000 078000078000078000078000078000078000078000078000078000078000078000078000 0780000780000780000780000780000780000FC000FFFE00FFFE00161F7E9E19>114 D<01FC100E03301800F0300070600030E00030E00010E00010E00010F00010F800007E00 003FF0001FFF000FFFC003FFE0003FF00001F80000F880003C80003C80001CC0001CC000 1CE0001CE00018F00038F00030CC0060C301C080FE00161F7E9E1A>I<00400000400000 400000400000400000C00000C00000C00001C00001C00003C00007C0000FC0001FFFE0FF FFE003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003 C00003C00003C00003C00003C00003C01003C01003C01003C01003C01003C01003C01003 C01001C02001E02000E0400078C0001F00142C7FAB19>I<078000F000FF801FF000FF80 1FF0000F8001F000078000F000078000F000078000F000078000F000078000F000078000 F000078000F000078000F000078000F000078000F000078000F000078000F000078000F0 00078000F000078000F000078000F000078000F000078000F000078000F000078001F000 078001F000078001F000038002F00003C004F00001C008F800007030FF80001FC0FF8021 1F7E9E25>I E /Ft 5 85 df<00000000C00000000000E00000000001E00000000003E0 0000000003E00000000007E00000000007E0000000000FE0000000000FE0000000001FE0 000000001FE00000000037E00000000067E00000000067E000000000C7E000000000C7F0 0000000183F00000000183F00000000303F00000000703F00000000603F00000000C03F0 0000000C03F00000001803F00000001803F00000003003F00000003003F00000006003F0 000000C003F0000000C003F00000018003F00000018003F8000003FFFFF8000003FFFFF8 0000060001F800000E0001F800000C0001F80000180001F80000180001F80000300001F8 0000300001F80000600001F80000E00001F80000C00001F80001C00001F80001C00001F8 0007C00001FC001FC00003FC00FFF8007FFFE0FFF8007FFFE02B327BB135>65 D<000FFFFFFE0000000FFFFFFF800000007F000FE00000007E0003F00000007E0000F800 00007E0000FC0000007E00007C000000FC00003E000000FC00003E000000FC00003F0000 00FC00001F000001F800001F000001F800001F800001F800001F800001F800001F800003 F000001F800003F000001F800003F000001F800003F000001F800007E000003F800007E0 00003F800007E000003F800007E000003F80000FC000003F00000FC000007F00000FC000 007F00000FC000007F00001F8000007E00001F800000FE00001F800000FE00001F800000 FC00003F000001FC00003F000001F800003F000001F800003F000003F000007E000003E0 00007E000007E000007E00000FC000007E00000F800000FC00001F800000FC00003F0000 00FC00007E000000FC0000FC000001F80001F0000001F80003E0000001F8000FC0000003 F8007F000000FFFFFFFC000000FFFFFFE000000031317BB036>68 D<000FFFFFFFFC000FFFFFFFFC00007F0001FC00007E00007C00007E00003C00007E0000 3C00007E0000180000FC0000180000FC0000180000FC0000180000FC0000180001F80000 180001F80000180001F80000180001F80000180003F00080100003F00180000003F00180 000003F00180000007E00300000007E00300000007E00700000007E01F0000000FFFFE00 00000FFFFE0000000FC01E0000000FC00E0000001F800C0000001F800C0000001F800C00 00001F800C0000003F00180000003F00080000003F00000000003F00000000007E000000 00007E00000000007E00000000007E0000000000FC0000000000FC0000000000FC000000 0000FC0000000001F80000000001F80000000001F80000000003F800000000FFFFF00000 00FFFFF00000002E317BB02F>70 D<000FFFFFF000000FFFFFFE0000007F003F8000007E 000FC000007E0007E000007E0003F000007E0001F80000FC0001F80000FC0001F80000FC 0001F80000FC0001F80001F80003F80001F80003F80001F80003F80001F80003F00003F0 0007F00003F00007E00003F0000FC00003F0000FC00007E0001F000007E0007E000007E0 00FC000007E007F000000FFFFFC000000FFFFF0000000FC00F8000000FC003C000001F80 03E000001F8001F000001F8001F000001F8001F800003F0001F800003F0001F800003F00 01F800003F0001F800007E0003F800007E0003F800007E0003F000007E0003F00000FC00 07F00000FC0007F00000FC0007F00800FC0007F00C01F80007F01801F80007F01801F800 03F03003F80003F030FFFFE001F0E0FFFFE000FFC0000000003F002E327BB034>82 D<07FFFFFFFFF00FFFFFFFFFF00FC00FE003F01E000FC000F01C000FC000E018000FC000 E038000FC0006030001F8000E030001F8000E060001F8000C060001F8000C060003F0000 C0C0003F0000C0C0003F0000C0C0003F0000C080007E00008000007E00000000007E0000 0000007E0000000000FC0000000000FC0000000000FC0000000000FC0000000001F80000 000001F80000000001F80000000001F80000000003F00000000003F00000000003F00000 000003F00000000007E00000000007E00000000007E00000000007E0000000000FC00000 00000FC0000000000FC0000000000FC0000000001F80000000001F80000000001F800000 00001F80000000003F00000000003F00000000003F0000000000FF00000000FFFFFF0000 00FFFFFF0000002C3173B033>84 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300dpi TeXDict begin %%EndSetup %%Page: 0 1 0 0 bop 795 947 a Ft(D)26 b(R)g(A)f(F)h(T)225 1038 y Fs(Do)r(cumen)n(t)20 b(for)i(a)f(Standard)g(Message-P)n(assing)f(In)n (terface)621 1232 y Fr(Message)c(P)o(assing)h(In)o(terface)e(F)l(orum) 766 1359 y(Septem)o(b)q(er)g(19,)h(1996)190 1417 y(This)h(w)o(ork)f(w)o (as)h(supp)q(orted)g(in)f(part)g(b)o(y)g(NSF)g(and)h(ARP)l(A)e(under)h (NSF)g(con)o(tract)283 1475 y(CD)o(A-9115428)j(and)e(Esprit)f(under)h (pro)s(ject)e(HPC)i(Standards)g(\(21111\).)p eop %%Page: 1 2 1 1 bop 166 45 a Fq(This)20 b(is)h(the)f(result)g(of)f(a)h(LaT)l(eX)g (run)g(of)g(a)f(draft)g(of)h(a)f(single)j(c)o(hapter)d(of)h(the)g(MPIF) f(Final)75 102 y(Rep)q(ort)d(do)q(cumen)o(t.)969 2828 y(i)p eop %%Page: 1 3 1 2 bop 75 356 a Fp(Chapter)34 b(3)75 564 y Fo(Miscellan)m(y)41 b(for)g(1.2)75 805 y Fn(3.1)59 b(V)n(ersion)20 b(Numb)r(er)75 953 y Fm(Curren)o(t)12 b(Status:)k Fl(P)o(assed)d(once.)20 b Fq(In)14 b(order)f(to)f(cop)q(e)i(with)g(c)o(hanges)f(to)f(the)i(MPI) f(Standard,)g(there)g(are)75 1057 y(b)q(oth)j(compile-time)h(and)f (run-time)g(w)o(a)o(ys)e(to)h(determine)h(whic)o(h)g(v)o(ersion)g(of)f (the)h(standard)e(is)i(in)h(use)75 1114 y(in)f(the)f(en)o(vironmen)o(t) h(one)f(is)h(using.)166 1170 y(The)e(\\v)o(ersion")g(will)h(b)q(e)g (represen)o(ted)f(b)o(y)g(t)o(w)o(o)e(separate)i(in)o(tegers,)g(for)f (the)h(v)o(ersion)g(and)g(sub)o(v)o(er-)75 1227 y(sion:)166 1283 y(In)i(C,)170 1367 y Fk(#define)23 b(MPI_VERSION)94 b(1)170 1423 y(#define)23 b(MPI_SUBVERSION)f(2)75 1507 y Fq(in)16 b(F)l(ortran,)170 1591 y Fk(INTEGER)23 b(MPI_VERSION,)g (MPI_SUBVERSION)170 1648 y(PARAMETER)g(\(MPI_VERSION)94 b(=)24 b(1\))170 1704 y(PARAMETER)f(\(MPI_SUBVERSION)f(=)i(2\))75 1788 y Fq(and)15 b(in)h(C++,)170 1872 y Fk(const)24 b(int)f (MPI::VERSION)f(=)i(1)170 1928 y(const)g(int)f(MPI::SUBVERSION)f(=)i(2) 75 2012 y Fq(F)l(or)15 b(run)o(time)g(determination,)75 2163 y Fj(MPI)p 160 2163 14 2 v 16 w(GET)p 264 2163 V 17 w(VERSION\()h(version,)f(subversion)h(\))117 2240 y Fl(OUT)108 b Fj(version)456 b Fl(v)o(ersion)14 b(n)o(um)o(b)q(er)117 2313 y(OUT)108 b Fj(subversion)393 b Fl(sub)o(v)o(ersion)15 b(n)o(um)o(b)q(er)75 2438 y Fk(int)23 b(MPI)p 245 2438 15 2 v 17 w(Get)p 334 2438 V 17 w(version\()g(int)g(*version,)g(int)g (*subversion)g(\))75 2524 y(MPI)p 150 2524 V 17 w(GET)p 239 2524 V 17 w(VERSION\()f(VERSION,)h(SUBVERSION,)f(IERROR)h(\))170 2581 y(INTEGER)g(VERSION,)g(SUBVERSION,)f(IERROR)1875 2617 y Fi(>)16 b Fh(\(Oct\))75 2667 y Fk(static)23 b(int)g(MPI::Get)p 532 2667 V 17 w(version\()f(int&)i(version,)e(int&)i(subversion)e(\)) 1875 2704 y Fi(?)16 b Fh(\(Oct\))964 2828 y Fq(1)p eop %%Page: 2 4 2 3 bop 75 -100 a Fq(2)970 b Fg(CHAPTER)15 b(3.)35 b(MISCELLANY)17 b(F)o(OR)e(1.2)75 45 y Fn(3.2)59 b(T)-5 b(reatment)18 b(of)i(MPI)p 670 45 18 2 v 21 w(Status)75 147 y Fq(The)25 b(follo)o(wing)g(prop)q(osals)g(add)g(to,)g(but)g(do)g(not)f(c)o (hange,)i(the)f(functionalit)o(y)h(asso)q(ciated)f(with)75 203 y Fj(MPI)p 160 203 14 2 v 16 w(ST)l(A)l(TUS)p Fq(.)75 325 y Ff(3.2.1)49 b(P)o(assing)17 b(MPI)p 481 325 15 2 v 18 w(ST)l(A)l(TUS)p 676 325 V 18 w(IGNORE)g(fo)o(r)f(MPI)p 1044 325 V 18 w(Status)75 458 y Fm(Curren)o(t)e(Status:)j Fl(P)o(assed)e(once.)166 561 y Fq(Ev)o(ery)10 b(call)i(to)e Fj(MPI)p 507 561 14 2 v 16 w(RECV)h Fq(includes)i(a)e Fj(status)h Fq(argumen)o(t,)f(where)g(the)f(system)h(can)f(return)h (details)75 618 y(ab)q(out)18 b(the)g(message)g(receiv)o(ed.)31 b(There)18 b(are)g(also)g(a)g(n)o(um)o(b)q(er)h(of)f(other)g(MPI)g (calls,)i(particularly)f(in)-1991 b Fi(>)15 b Fh(\(Oct\))75 674 y Fq(MPI-2,)20 b(where)f Fj(status)i Fq(is)f(returned.)33 b(An)19 b(ob)s(ject)g(of)g(t)o(yp)q(e)g Fj(MPI)p 1234 674 V 16 w(ST)l(A)l(TUS)h Fq(is)g(not)f(an)g Fj(MPI)g Fq(opaque)-117 676 y Fi(?)c Fh(\(Oct\))75 731 y Fq(ob)s(ject;)h(its)h (structure)g(is)g(declared)h(in)f Fk(mpi.h)f Fq(and)h Fk(mpif.h)p Fq(,)e(and)i(it)g(exists)g(in)g(the)g(users')f(program.)75 787 y(In)i(man)o(y)e(cases)h(application)i(programs)d(are)g (constructed)i(so)e(that)h(it)g(is)g(unnecessary)h(for)f(them)g(to)75 844 y(examine)j(the)g Fk(status)f Fq(\014elds.)35 b(In)20 b(these)g(cases)f(it)h(is)g(a)g(w)o(aste)e(for)h(the)h(user)g(to)f (allo)q(cate)h(a)g(status)75 900 y(ob)s(ject,)f(and)h(it)f(is)h (particularly)g(w)o(asteful)f(for)g(the)g Fj(MPI)f Fq(implemen)o (tation)j(to)d(\014ll)j(in)f(\014elds)h(in)f(this)75 957 y(ob)s(ject.)166 1013 y(T)l(o)h(cop)q(e)h(with)g(this)g(problem,)i (there)d(is)h(a)g(pre-de\014ned)h(constan)o(t,)f Fj(MPI)p 1502 1013 V 16 w(ST)l(A)l(TUS)p 1683 1013 V 17 w(IGNORE)p Fq(,)75 1070 y(whic)o(h,)i(when)e(passed)g(to)f(a)h(receiv)o(e)h(or)e (test)g(function,)j(informs)e(the)g(implemen)o(tation)h(that)e(the)75 1126 y(status)13 b(\014elds)h(are)f(not)g(to)g(b)q(e)h(\014lled)i(in.)k (Note)13 b(that)g Fj(MPI)p 1059 1126 V 15 w(ST)l(A)l(TUS)p 1239 1126 V 18 w(IGNORE)i Fq(is)e(not)h(a)f(sp)q(ecial)i(t)o(yp)q(e)e (of)75 1183 y Fj(MPI)p 160 1183 V 16 w(ST)l(A)l(TUS)i Fq(ob)s(ject;)f(rather,)f(it)h(is)h(a)f(sp)q(ecial)i(v)m(alue)f(for)f (the)g(argumen)o(t.)k(That)c(is,)h(in)f(C)g(one)h(w)o(ould)75 1239 y(exp)q(ect)h(it)f(to)g(b)q(e)h(NULL,)g(not)e(the)i(address)f(of)g (a)g(sp)q(ecial)i Fj(MPI)p 1179 1239 V 15 w(ST)l(A)l(TUS)p Fq(.)166 1295 y(In)12 b(general,)h(this)f(optimization)g(can)g(apply)g (to)f(all)i(functions)f(for)f(whic)o(h)h Fj(MPI)p 1514 1295 V 16 w(Status)i Fq(or)d(an)g(arra)o(y)75 1352 y(of)k Fj(MPI)p 212 1352 V 16 w(Status)p Fq('s)i(is)f(an)g(argumen)o(t.)k (These)c(are)f(all)i(the)e(v)m(arious)h(forms)f(of)g Fj(MPI)p 1493 1352 V 16 w(RECV)p Fq(,)h Fj(MPI)p 1735 1352 V 16 w(TEST)p Fq(,)75 1408 y(and)i Fj(MPI)p 251 1408 V 15 w(W)l(AIT)p Fq(.)f(When)h(an)f(arra)o(y)g(is)g(passed,)h(as)f (in)h(the)p 1132 1408 V 34 w Fj(ANY)g Fq(and)p 1349 1408 V 34 w Fj(ALL)f Fq(functions,)h(a)f(separate)75 1465 y(constan)o(t,)d Fj(MPI)p 356 1465 V 16 w(ST)l(A)l(TUSES)p 589 1465 V 18 w(IGNORE)p Fq(,)i(is)f(passed)h(for)e(the)i(arra)o(y)e (argumen)o(t.)166 1521 y Fj(MPI)p 251 1521 V 16 w(ST)l(A)l(TUS)p 432 1521 V 17 w(IGNORE)g Fq(and)e Fj(MPI)p 794 1521 V 16 w(ST)l(A)l(TUSES)p 1027 1521 V 18 w(IGNORE)i Fq(are)e(not)g (required)h(to)f(ha)o(v)o(e)g(the)g(same)-1992 b Fi(>)15 b Fh(\(Oct\))75 1578 y Fq(v)m(alues)h(in)g(C)f(and)h(F)l(ortran.)166 1634 y(It)d(is)g(not)g(allo)o(w)o(ed)g(to)f(ha)o(v)o(e)g(some)h(of)f (the)h(statuses)f(in)i(an)f(arra)o(y)e(of)i(statuses)f(for)p 1574 1634 V 28 w Fj(ANY)i Fq(and)p 1782 1634 V 29 w Fj(ALL)75 1691 y Fq(functions)j(set)g(to)f Fj(MPI)p 487 1691 V 16 w(ST)l(A)l(TUS)p 668 1691 V 17 w(IGNORE)p Fq(;)i(one)e(either)i(sp)q (eci\014es)g(ignoring)f Fe(al)r(l)34 b Fq(of)17 b(the)f(statuses)g(in) 75 1747 y(suc)o(h)e(a)g(call)h(with)f Fj(MPI)p 482 1747 V 16 w(ST)l(A)l(TUSES)p 715 1747 V 18 w(IGNORE)p Fq(,)h(or)f Fe(none)i Fq(of)e(themm)g(b)o(y)f(passing)i(normal)f(statuses)f(in)75 1804 y(alll)k(p)q(ositions)f(in)g(the)f(arra)o(y)f(of)h(statuses.)-932 b Fi(?)15 b Fh(\(Oct\))75 1925 y Ff(3.2.2)49 b(Non-destructive)16 b(T)l(est)f(of)i(MPI)p 808 1925 15 2 v 18 w(Status)75 2058 y Fm(Curren)o(t)d(Status:)j Fl(P)o(assed)e(once.)166 2162 y Fq(This)22 b(call)g(is)g(useful)g(for)f(accessing)h(the)f (information)h(asso)q(ciated)f(with)h(a)f(request,)h(without)75 2218 y(deleting)c(the)f(request)f(\(in)h(case)g(the)g(user)f(is)i(exp)q (ected)f(to)f(access)h(it)g(after)f(the)g(handler\).)25 b(It)17 b(allo)o(ws)75 2275 y(one)11 b(to)e(la)o(y)o(er)i(libraries)h (more)e(con)o(v)o(enien)o(tly)l(,)i(since)g(m)o(ultiple)g(la)o(y)o(ers) e(of)h(soft)o(w)o(are)d(ma)o(y)i(access)h(the)f(same)75 2331 y(completed)17 b(request)f(and)g(extract)f(from)g(it)h(the)g (status)f(information.)21 b(This)c(will)g(also)f(b)q(e)g(imp)q(ortan)o (t)75 2388 y(for)g(language)i(in)o(terop)q(erabilit)o(y)h(if)e(w)o(e)g (decide)i(that)d(status)g(ob)s(jects)h(cannot)f(b)q(e)i(transferred)f (across)75 2444 y(language)d(b)q(oundaries,)g(since)h(with)f(this)g (function)g(the)g(request)f(ob)s(ject)g(can)h(b)q(e)g(transferred)f (instead.)1967 46 y Fh(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 3 5 3 4 bop 75 -100 a Fg(3.3.)34 b(ERR)o(OR)17 b(CLASS)f(F)o(OR)f(INV)-5 b(ALID)16 b(KEYV)-5 b(AL)818 b Fq(3)75 45 y Fj(MPI)p 160 45 14 2 v 16 w(GET)p 264 45 V 17 w(ST)l(A)l(TUS\()16 b(request,)g(\015ag,)f(status)i(\))117 122 y Fl(IN)155 b Fj(request)452 b Fl(an)14 b Fd(MPI)p 1040 122 13 2 v 14 w(REQUEST)f Fl(ob)r(ject)117 197 y(OUT)108 b Fj(\015ag)518 b Fl(\015ag,)13 b(same)g(as)h(from)e Fd(MPI)p 1325 197 V 15 w(TEST)117 273 y Fl(OUT)108 b Fj(status)476 b Fd(MPI)p 982 273 V 15 w(ST)m(A)m(TUS)12 b Fl(ob)r(ject)j(if)e(\015ag)g(is)h (true)75 397 y Fk(int)23 b(MPI)p 245 397 15 2 v 17 w(Get)p 334 397 V 17 w(status\(MPI)p 591 397 V 16 w(Request)g(request,)f(int)i (*flag,)f(MPI)p 1347 397 V 17 w(Status)g(*status\))75 483 y(MPI)p 150 483 V 17 w(GET)p 239 483 V 17 w(STATUS\()f(REQUEST,)h (FLAG,)g(STATUS,)g(IERROR\))170 540 y(INTEGER)g(REQUEST,)g(FLAG,)g (STATUS\(MPI)p 962 540 V 16 w(STATUS)p 1122 540 V 16 w(SIZE\),)h(IERROR)75 626 y(int)f(MPI::Request::Get)p 581 626 V 15 w(status\(bool&)g(flag,)g(MPI::Status&)f(status\))h(const) 166 713 y Fq(Sets)18 b(\015ag=true)g(if)h(the)f(request)h(has)f (completed,)h(and,)g(if)g(so,)f(returns)g(in)h(status)f(the)g(request) 75 769 y(status.)34 b(Ho)o(w)o(ev)o(er,)20 b(unlik)o(e)i(test)d(or)h(w) o(ait,)g(it)h(do)q(es)f(not)g(deallo)q(cate)h(or)f(inactiv)m(ate)h(the) f(request;)i(a)75 826 y(subsequen)o(t)16 b(call)g(to)f(test,)f(w)o(ait) h(or)f(free)i(should)g(b)q(e)g(executed)g(with)f(that)g(request)75 969 y Fn(3.3)59 b(Erro)n(r)21 b(Class)f(fo)n(r)h(Invalid)e(Keyval)1875 1026 y Fi(>)d Fh(\(Oct\))75 1118 y Fm(Curren)o(t)e(Status:)j Fl(P)o(assed)e(once.)166 1221 y Fq(There)d(is)f(a)h(new)f(MPI)h(error)e (class:)19 b Fj(MPI)p 875 1221 14 2 v 15 w(ERR)p 975 1221 V 18 w(KEYV)l(AL)p Fq(.)11 b(It)h(can)f(b)q(e)h(returned)g(b)o(y)f Fj(MPI)p 1692 1221 V 16 w(A)o(ttr)p 1785 1221 V 17 w(put)p Fq(,)75 1278 y Fj(MPI)p 160 1278 V 16 w(A)o(ttr)p 253 1278 V 17 w(get)p Fq(,)16 b Fj(MPI)p 440 1278 V 16 w(A)o(ttr)p 533 1278 V 16 w(delete)p Fq(,)i Fj(MPI)p 772 1278 V 16 w(Keyval)p 915 1278 V 15 w(free)p Fq(,)e Fj(MPI)p 1111 1278 V 16 w(Comm)p 1253 1278 V 14 w(dup)p Fq(,)i(and)e Fj(MPI)p 1538 1278 V 16 w(Comm)p 1680 1278 V 14 w(free)p Fq(.)23 b(The)75 1334 y(last)15 b(t)o(w)o(o)f(are)h(b)q(ecause)h Fj(k)o(eyval)f Fq(is)h(an)f(argumen)o(t)f(to)h(the)g(cop)o(y)g(and)h (delete)g(functions)g(for)e(attributes.)8 b Fi(?)16 b Fh(\(Oct\))1875 1393 y Fi(>)g Fh(\(Oct\))1875 1451 y Fi(>)g Fh(\(Oct\))75 1478 y Fn(A)k(new)f(reduce)g(op)r(eration)75 1626 y Fm(Curren)o(t)14 b(Status:)j Fl(New)d(for)g(Octob)q(er.)166 1730 y Fq(A)e(new)h(prede\014ned)h(op)q(eration,)f Fd(MPI)p 811 1730 13 2 v 14 w(REPLA)o(CE)p Fq(,)d(is)j(de\014ned.)20 b(It)13 b(corresp)q(onds)f(to)g(the)h(asso)q(ciativ)o(e)75 1786 y(function)18 b Fc(f)5 b Fq(\()p Fc(a;)j(b)p Fq(\))15 b(=)h Fc(b)p Fq(;)i(i.e.,)f(the)g(curren)o(t)h(v)m(alue)g(in)g(the)g (target)e(memory)h(is)h(replaced)g(b)o(y)f(the)h(v)m(alue)75 1843 y(supplied)g(b)o(y)d(the)g(origin.)21 b(This)15 b(op)q(eration)h(can)f(b)q(e)h(applied)h(on)e(an)o(y)g(basic)h(datat)o (yp)q(e.)166 1947 y Fm(Discussion:)35 b Fl(This)15 b(needs)g(a)f (rationale.)19 b(It)14 b(can)h(b)q(e)g(used)g(to)f(do)g(one-sided)h (broadcasts)g(\(should)f(it)g(b)q(e)75 2003 y(f\(a,b\))i(=)h(a\)?)27 b(It)17 b(w)o(ould)f(b)q(e)i(wierd)f(in)f(a)h(reduce)i(op)q(eration,)e (and)f(could)h(b)q(e)h(easily)e(done)h(b)o(y)g(the)g(user)h(as)f(a)75 2059 y(user-de\014ned)f(function.)1875 2107 y Fi(?)g Fh(\(Oct\))75 2250 y Fn(3.4)59 b(A)20 b(F)n(o)n(rtran)h(Problem)e(with) h(Register)e(Optimization)75 2399 y Fm(Curren)o(t)c(Status:)j Fl(A)d(new)g(discussion.)189 2552 y Fe(A)n(dvic)n(e)h(to)i(users.)189 2627 y Fq(MPI)d(pro)o(vides)g(op)q(erations)g(whic)o(h)h(ma)o(y)e(b)q (e)i(hidden)h(from)d(the)h(user)g(co)q(de)h(and)f(run)h(in)f(parallel) 189 2684 y(with)g(it,)h(accessing)g(the)f(same)g(memory)g(as)g(user)g (co)q(de.)20 b(Examples)15 b(include)i(the)d(data)g(transfer)-32 46 y Fh(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 4 6 4 5 bop 75 -100 a Fq(4)970 b Fg(CHAPTER)15 b(3.)35 b(MISCELLANY)17 b(F)o(OR)e(1.2)189 45 y Fq(for)10 b(an)h Fj(MPI)p 398 45 14 2 v 16 w(RECV)h Fq(The)f(optimizer)i(of)d(a)h(compiler)i(will)f (assume)f(that)g(it)g(can)h(recognize)g(p)q(erio)q(ds)189 102 y(when)j(a)g(cop)o(y)g(of)g(a)f(v)m(ariable)j(can)e(b)q(e)h(k)o (ept)f(in)h(a)e(register)h(without)g(reloading)h(from)f(or)f(storing) 189 158 y(to)e(memory)l(.)18 b(When)13 b(the)g(user)g(co)q(de)g(is)g(w) o(orking)f(with)h(a)f(register)h(cop)o(y)f(of)g(some)h(v)m(ariable)g (while)189 214 y(the)i(hidden)i(op)q(eration)f(reads)f(or)g(writes)g (the)h(memory)f(cop)o(y)l(,)g(problems)h(o)q(ccur.)k(This)c(section)189 271 y(discusses)g(register)f(optimization)h(pitfalls.)189 346 y(When)f(a)f(v)m(ariable)i(is)f(lo)q(cal)h(to)e(a)g(F)l(ortran)f (subroutine)j(\(i.e.)k(not)14 b(in)h(a)g(COMMON)f(blo)q(c)o(k\),)h(the) 189 403 y(compiler)g(will)i(assume)d(that)g(it)h(cannot)f(b)q(e)h(mo)q (di\014ed)h(b)o(y)f(a)f(called)i(subroutine)f(unless)h(it)f(is)g(an)189 459 y(actual)g(argumen)o(t)g(of)h(the)f(call.)23 b(In)16 b(the)g(most)e(common)i(link)m(age)h(con)o(v)o(en)o(tion,)e(the)h (subroutine)189 515 y(is)d(exp)q(ected)i(to)d(sa)o(v)o(e)h(and)g (restore)g(certain)h(registers.)19 b(Th)o(us,)13 b(the)g(optimizer)i (will)g(assume)e(that)189 572 y(a)k(register)h(whic)o(h)h(held)g(a)e(v) m(alid)j(cop)o(y)d(of)h(suc)o(h)g(a)g(v)m(ariable)h(b)q(efore)f(the)g (call)h(will)g(still)h(hold)e(a)189 628 y(v)m(alid)e(cop)o(y)g(on)f (return.)189 703 y(Normally)22 b(users)g(are)g(not)g(a\017icted)g(with) h(this.)41 b(But)22 b(the)g(user)g(should)h(pa)o(y)f(atten)o(tion)g(to) 189 760 y(this)f(section)g(if)g(in)h(his/her)f(program)e(a)i(bu\013er)f (argumen)o(t)g(to)g(an)h Fj(MPI)p 1498 760 V 16 w(SEND)p Fq(,)f Fj(MPI)p 1746 760 V 16 w(RECV)189 816 y Fq(etc.)25 b(uses)18 b(a)f(name)g(whic)o(h)h(hides)g(the)f(actual)h(v)m(ariables)g (in)o(v)o(olv)o(ed.)27 b Fj(MPI)p 1493 816 V 16 w(BOTTOM)18 b Fq(with)f(an)189 873 y(MPI)p 281 873 V 16 w(Datat)o(yp)q(e)i(con)o (taining)i(absolute)g(addresses)f(is)h(one)f(example.)36 b(Creating)20 b(a)f(datat)o(yp)q(e)189 929 y(whic)o(h)14 b(uses)f(one)g(v)m(ariable)i(as)e(an)g(anc)o(hor)f(and)i(brings)f (along)h(others)e(b)o(y)h(using)h Fj(MPI)p 1659 929 V 16 w(ADDRESS)189 986 y Fq(to)h(determine)j(their)f(o\013sets)e(from)g (the)i(anc)o(hor)f(is)h(another.)22 b(The)17 b(anc)o(hor)f(v)m(ariable) i(w)o(ould)e(b)q(e)189 1042 y(the)g(only)h(one)f(men)o(tioned)h(in)g (the)g(call.)24 b(Also)16 b(atten)o(tion)g(m)o(ust)g(b)q(e)h(pa)o(y)o (ed)f(if)h(MPI)f(op)q(erations)189 1099 y(are)f(used)g(that)g(run)g(in) h(parallel)h(with)f(the)f(user's)g(application.)189 1174 y(The)g(follo)o(wing)h(example)g(sho)o(ws,)e(what)h(F)l(ortran)f (compilers)i(are)f(allo)o(w)o(ed)h(to)e(do:)224 1299 y(This)i(source)f(...)663 b(can)15 b(b)q(e)h(compiled)h(as:)224 1363 y(call)f(MPI)p 399 1363 V 17 w(ADDRESS\(buf,bufaddr,ierr\))189 b(call)16 b(MPI)p 1344 1363 V 17 w(ADDRESS\(buf,...\))224 1420 y(call)g(MPI)p 399 1420 V 17 w(TYPE)p 545 1420 V 16 w(STR)o(UCT\(1,1,bufaddr,)163 b(call)16 b(MPI)p 1344 1420 V 17 w(TYPE)p 1490 1420 V 16 w(STR)o(UCT\(...\))590 1476 y(MPI)p 682 1476 V 17 w(REAL,t)o(yp,ierr\))224 1533 y(call)g(MPI)p 399 1533 V 17 w(TYPE)p 545 1533 V 16 w(COMMIT\(t)o(yp\)) 308 b(call)16 b(MPI)p 1344 1533 V 17 w(TYPE)p 1490 1533 V 16 w(COMMIT\(...\))224 1589 y(v)m(al)p 283 1589 V 17 w(old)g(=)g(buf)681 b(register)15 b(=)h(buf)1169 1646 y(v)m(al)p 1228 1646 V 17 w(old)g(=)f(register)224 1702 y(call)h(MPI)p 399 1702 V 17 w(RECV\(MPI)p 654 1702 V 16 w(BOTTOM,1,t)o(yp,...\))107 b(call)16 b(MPI)p 1344 1702 V 17 w(RECV\(MPI)p 1599 1702 V 16 w(BOTTOM,...\))224 1759 y(v)m(al)p 283 1759 V 17 w(new)g(=)f(buf)664 b(v)m(al)p 1228 1759 V 17 w(new)16 b(=)f(register)189 1884 y(The)j(compiler)g(do)q (es)g(not)g(in)o(v)m(alidate)h(the)f(register)g(b)q(ecause)g(it)g (cannot)f(see)h(that)f Fj(MPI)p 1746 1884 V 16 w(RECV)189 1940 y Fq(c)o(hanges)h(the)g(v)m(alue)i(of)e Fj(buf)p Fq(.)30 b(The)19 b(access)f(of)g Fj(buf)i Fq(is)f(hidden)h(b)o(y)e(the) h(use)f(of)g Fj(MPI)p 1659 1940 V 16 w(ADDRESS)189 1997 y Fq(and)d Fj(MPI)p 362 1997 V 16 w(BOTTOM)p Fq(.)189 2072 y(The)g(next)g(example)h(sho)o(ws)f(extreme,)g(but)g(allo)o(w)o (ed)h(p)q(ossibilities!)224 2197 y(Source)424 b(compiled)17 b(as)322 b(or)15 b(compiled)i(as)224 2261 y(call)f(MPI)p 399 2261 V 17 w(IRECV\(buf,..req\))i(call)f(MPI)p 955 2261 V 16 w(IRECV\(buf,..req\))h(call)f(MPI)p 1510 2261 V 16 w(IRECV\(buf,..req\))779 2318 y(register)e(=)h(buf)278 b(b1)15 b(=)h(buf)224 2374 y(call)g(MPI)p 399 2374 V 17 w(W)-5 b(AIT\(req,..\))104 b(call)17 b(MPI)p 955 2374 V 16 w(W)-5 b(AIT\(req,..\))104 b(call)17 b(MPI)p 1510 2374 V 16 w(W)-5 b(AIT\(req,..\))224 2431 y(b1)15 b(=)h(buf)377 b(b1)15 b(:=)g(register)189 2556 y Fj(MPI)p 274 2556 V 15 w(W)l(AIT)e Fq(or)f(a)g(parallel)i(thread)e(mo)q(di\014es)i Fj(buf)f Fq(b)q(et)o(w)o(een)g(the)f(in)o(v)o(o)q(cation)h(of)f Fj(MPI)p 1648 2556 V 16 w(IRECV)h Fq(and)189 2612 y(the)j(\014nish)h (of)f Fj(MPI)p 528 2612 V 16 w(W)l(AIT)p Fq(.)g(But)g(the)g(compiler)i (cannot)e(see)g(an)o(y)g(p)q(ossibilit)o(y)i(that)e Fj(buf)h Fq(can)f(b)q(e)189 2669 y(c)o(hanged)g(after)e Fj(MPI)p 557 2669 V 16 w(IRECV)i Fq(has)g(\014nished.)23 b(And)16 b(therefore)f(it)h(can)g(sc)o(hedule)h(the)f(load)g(of)f Fj(buf)1967 46 y Fh(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 5 7 5 6 bop 75 -100 a Fg(3.5.)34 b(TR)o(UE)15 b(EXTENT)g(OF)g(D)o(A)l(T)l (A)l(TYPES)958 b Fq(5)189 45 y(earlier)14 b(than)g(t)o(yp)q(ed)g(in)h (the)f(source.)20 b(It)13 b(has)h(no)g(reason)f(to)h(a)o(v)o(oid)f (using)i(a)e(register)h(to)f(hold)i Fj(buf)189 102 y Fq(across)f(the)h(call)i(to)d Fj(MPI)p 625 102 14 2 v 16 w(W)l(AIT)p Fq(.)h(It)g(also)g(can)h(reorder)f(the)g(instructions)h (as)f(in)h(the)f(left)h(case.)189 177 y(T)l(o)c(prev)o(en)o(t)h (instruction)h(reordering)g(or)e(the)h(allo)q(cation)h(of)f(a)g(v)m (ariable)h(or)f(bu\013er)g(in)h(a)e(register,)189 233 y(there)j(are)g(three)g(p)q(ossibilities)k(in)d(p)q(ortable)f(F)l (ortran)f(co)q(de:)243 327 y Fi(\017)23 b Fq(One)d(can)f(put)h(the)g(v) m(ariables)g(and)g(bu\013ers)f(in)o(to)h(common)f(blo)q(c)o(ks;)j(then) e(they)f(will)i(b)q(e)289 383 y(allo)q(cated)d(in)g(the)g(memory)e (while)j(an)o(y)e(library)h(subroutine)h(\(e.g.)25 b(an)18 b(MPI)f(routine\))g(is)289 440 y(called)c(pro)o(vided)f(that)f(the)h (compiler)h(cannot)e(analyze)h(that)f(the)h(library)g(subroutine)h(do)q (es)289 496 y(not)h(use)i(these)f(common)g(blo)q(c)o(ks,)h(or)243 569 y Fi(\017)23 b Fq(one)11 b(can)h(declare)g(them)f(as)g(V)o(OLA)l (TILE,)h(but)g(this)g(is)f(not)g(part)g(of)g(the)g(standards)g(F)l (ortran)289 626 y(77)f(and)h(90,)g(and)g(y)o(our)f(co)q(de)i(will)h (not)d(b)q(e)i(p)q(ortable)f(b)q(ecause)h(there)f(are)g(compilers)h (without)289 682 y(V)o(OLA)l(TILE,)k(and)f(normally)h(it)g(prev)o(en)o (ts)e(an)o(y)h(optimization,)h(or)243 755 y Fi(\017)23 b Fq(one)14 b(can)g(call)h(a)f(dumm)o(y)g(routine)g(with)g(the)h(v)m (ariable)g(or)e(bu\013er)h(as)g(an)g(actual)g(argumen)o(t;)289 812 y(if)22 b(the)h(dumm)o(y)f(routine)h(is)f(not)g(part)g(of)g(the)g (application)i(then)f(the)f(compiler)i(m)o(ust)289 868 y(allo)q(cate)16 b(the)f(v)m(ariable)i(or)e(bu\013er)h(in)g(the)f (memory)g(while)i(that)e(subroutine)h(is)g(executed;)289 925 y Fj(MPI)p 374 925 V 15 w(ADDRESS)h Fq(can)e(b)q(e)h(used)g(as)e (dumm)o(y)h(routine.)189 1019 y(High)e(qualit)o(y)g(F)l(ortran)f (implemen)o(tations)i(ma)o(y)e(pro)o(vide)i(mec)o(hanisms)f(to)f(a)o(v) o(oid)h(the)g(problem,)189 1075 y(without)18 b(disabling)j (optimizations.)31 b(Ho)o(w)o(ev)o(er,)18 b(co)q(de)h(that)f(do)q(es)h (not)f(use)h(one)g(of)f(the)h(three)189 1131 y(solutions)d(describ)q (ed)h(ab)q(o)o(v)o(e)d(ma)o(y)h(not)g(b)q(e)h(p)q(ortable.)189 1207 y(In)h(C,)f(subroutines)i(whic)o(h)g(mo)q(dify)f(v)m(ariables)h (that)e(are)h(not)f(in)i(the)f(parameter)f(list)i(will)g(not)189 1263 y(cause)e(register)h(optimization)g(problems.)25 b(This)17 b(is)g(b)q(ecause)g(taking)g(p)q(oin)o(ters)g(to)f(storage)f (ob-)189 1319 y(jects)d(b)o(y)g(using)h(the)g(&)f(op)q(erator)g(and)g (later)h(referencing)g(the)g(ob)s(jects)f(b)o(y)g(w)o(a)o(y)f(of)h(the) h(p)q(oin)o(ter)f(is)189 1376 y(an)j(in)o(tegral)g(part)g(of)f(the)i (language.)k(A)15 b(C)g(compiler)h(understands)g(the)f(implications,)i (so)e(that)189 1432 y(the)g(problem)h(should)h(not)e(o)q(ccur,)h(in)g (general.)21 b(Ho)o(w)o(ev)o(er,)14 b(some)i(compilers)g(do)g(o\013er)e (optional)189 1489 y(aggressiv)o(e)g(optimization)j(lev)o(els)f(whic)o (h)g(ma)o(y)f(not)f(b)q(e)i(safe.)189 1564 y(\()p Fe(End)f(of)i(advic)n (e)f(to)g(users.)p Fq(\))p 75 1670 827 2 v 75 1727 a(The)f(follo)o (wing)h(advice)h(m)o(ust)d(b)q(e)i(added)g(in)75 1795 y(-)f(MPI)g(2,)g(c)o(hapter)g(?.?)21 b(Handlers,)75 1851 y(-)15 b Fj(MPI)p 190 1851 14 2 v 16 w(ADDRESS)h Fq(\(MPI)f(1.1,)f (page)h(70,)f(line)j(16\),)d(and)h(in)75 1908 y(-)g Fj(MPI)p 190 1908 V 16 w(IRECV)h Fq(\(MPI)f(1.1,)e(\\Non)o(blo)q(c)o(king)k (comm)o(unication",)e(end)h(of)e(page)h(40\))189 2014 y Fe(A)n(dvic)n(e)i(to)i(users.)53 b Fq(T)l(o)17 b(prev)o(en)o(t)g (troubles)h(with)g(the)f(register)g(optimization)i(of)e(the)g(F)l (ortran)189 2071 y(compilers)i(please)g(pa)o(y)e(atten)o(tion)h(to)f (the)h(hin)o(ts)g(in)h(section)g(3.4)e(\\A)g(F)l(ortran)g(Problem)i (with)189 2127 y(register)c(optimization")h(on)f(page)g(3.)k(\()p Fe(End)d(of)g(advic)n(e)g(to)h(users.)p Fq(\))1875 2188 y Fi(?)f Fh(\(Oct\))75 2320 y Fn(3.5)59 b(T)-5 b(rue)20 b(Extent)d(of)k(Datat)n(yp)r(es)75 2469 y Fm(Curren)o(t)14 b(Status:)j Fl(P)o(assed)e(once.)166 2573 y Fq(Supp)q(ose)23 b(w)o(e)f(implemen)o(t)h(gather)f(as)g(a)f(spanning)i(tree)f(implemen)o (ted)i(on)e(top)g(of)g(p)q(oin)o(t-to-)75 2629 y(p)q(oin)o(t)e (routines.)34 b(Since)22 b(the)d(recv)h(bu\013er)g(is)g(only)h(v)m (alid)g(on)f(the)g(ro)q(ot)f(pro)q(cess,)i(one)f(will)h(need)g(to)75 2685 y(allo)q(cate)f(some)f(temp)q(orary)g(space)h(for)f(receiving)i (data)e(on)g(in)o(termediate)i(no)q(des.)33 b(The)20 b(di\016cultly)-32 46 y Fh(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 6 8 6 7 bop 75 -100 a Fq(6)970 b Fg(CHAPTER)15 b(3.)35 b(MISCELLANY)17 b(F)o(OR)e(1.2)75 45 y Fq(is)20 b(in)g(determining)h(the)f(size)g(one)f (needs)i(to)d(allo)q(ciate.)34 b(This)20 b(o)q(ccurs)g(since)h(the)e (user)h(can)f(mo)q(dify)75 102 y(the)g(exten)o(t)h(using)g(the)f Fd(MPI)p 583 102 13 2 v 14 w(UB)g Fq(and)h Fd(MPI)p 841 102 V 14 w(LB)f Fq(v)m(alues.)34 b(The)20 b(writer)f(of)g(the)g(gather) g(routine)h(could)75 158 y(determine)14 b(this)f(information)g(b)o(y)g (deco)q(ding)h(the)f(datat)o(yp)q(e.)18 b(Ho)o(w)o(ev)o(er,)12 b(this)h(is)g(more)g(w)o(ork)e(and)i(more)75 214 y(painful)j(than)e (desired.)20 b(Th)o(us,)14 b(a)g(new)g(function)h(is)g(pro)o(vided)g (whic)o(h)f(returns)g(the)h(true)e(exten)o(t)h(of)g(the)75 271 y(datat)o(yp)q(e.)75 422 y Fj(MPI)p 160 422 14 2 v 16 w(TRUE)p 294 422 V 17 w(EXTENT\(datat)o(yp)q(e,)j(true)p 775 422 V 17 w(lb,)e(true)p 929 422 V 17 w(extent\))117 499 y Fl(IN)155 b Fj(datat)o(yp)q(e)424 b Fl(datat)o(yp)q(e)14 b(to)g(get)g(information)d(on)117 574 y(OUT)108 b Fj(true)p 396 574 V 17 w(lb)461 b Fl(true)15 b(lo)o(w)o(er)e(b)q(ound)h(of)g (datat)o(yp)q(e)117 649 y(OUT)108 b Fj(true)p 396 649 V 17 w(extent)379 b Fl(true)15 b(size)g(of)e(datat)o(yp)q(e)75 774 y Fk(int)23 b(MPI)p 245 774 15 2 v 17 w(True)p 358 774 V 17 w(extent\(MPI)p 615 774 V 16 w(Datatype)f(datatype,)h(MPI)p 1156 774 V 17 w(Aint)g(*true)p 1412 774 V 17 w(lb,)g(MPI)p 1596 774 V 17 w(Aint)393 830 y(*true)p 516 830 V 17 w(extent\))75 917 y(MPI)p 150 917 V 17 w(TRUE)p 263 917 V 16 w(EXTENT\(DATATYPE,)f (TRUE)p 781 917 V 17 w(LB,)h(TRUE)p 989 917 V 17 w(EXTENT,)g(IERROR\)) 170 973 y(INTEGER)g(DATATYPE,)g(TRUE)p 699 973 V 17 w(LB,)g(TRUE)p 907 973 V 17 w(EXTENT,)g(IERROR)166 1060 y Fj(true)p 244 1060 14 2 v 17 w(lb)14 b Fq(returns)f(the)g(o\013set)g(of)g(the)g (lo)o(w)o(est)g(unit)h(of)f(store)g(whic)o(h)h(is)g(addressed)g(b)o(y)f (the)h(datat)o(yp)q(e.)75 1116 y Fj(true)p 153 1116 V 17 w(extent)k Fq(returns)f(the)g(true)f(size)i(of)e(the)h(datat)o(yp)q (e.)24 b(The)17 b Fj(true)p 1244 1116 V 17 w(extent)i Fq(is)e(the)g(minim)o(um)h(n)o(um)o(b)q(er)75 1173 y(of)13 b(b)o(ytes)g(of)h(memory)f(necessary)g(to)g(hold)i(a)e(datat)o(yp)q(e)g (and)h(ignores)g(the)f Fj(LB)h Fq(and)g Fj(UB)g Fq(that)e(ma)o(y)h(ha)o (v)o(e)75 1229 y(b)q(een)j(used)g(in)g(creating)g(the)f(datat)o(yp)q (e.)166 1326 y Fm(Alternativ)o(es)o(:)166 1376 y Fl(If)e(w)o(e)i(w)o (an)o(t)e(to)h(b)q(e)g(consisten)o(t)h(with)f(the)g(MPI1)g(exten)o(t)h (functions,)f(w)o(e)g(w)o(ould)f(need)i(three)g(functions:)166 1426 y Fd(MPI)p 243 1426 13 2 v 14 w(TRUE)p 366 1426 V 15 w(LB\(datat)o(yp)q(e,)g(lb\))166 1475 y(MPI)p 243 1475 V 14 w(TRUE)p 366 1475 V 15 w(UB\(datat)o(yp)q(e,)f(ub\))166 1525 y(MPI)p 243 1525 V 14 w(TRUE)p 366 1525 V 15 w(EXTENT\(datat)o(yp) q(e,)h(extent\))166 1575 y Fl(On)f(the)h(other)f(hand,)f(w)o(e)i(ma)o (y)c(not)j(w)o(an)o(t)g(to)f(mak)o(e)g(the)h(same)f(mistak)o(e)g(t)o (wice...)75 1766 y Fn(3.6)59 b(Datat)n(yp)r(e)18 b(extent)75 1914 y Fm(Curren)o(t)c(Status:)j Fl(P)o(assed)e(once.)166 2018 y Fq(MPI)21 b(allo)o(ws)h(one)f(to)g(c)o(hange)h(the)f(exten)o(t)g (of)g(a)h(datat)o(yp)q(e,)g(using)g(lo)o(w)o(er)f(b)q(ound)h(and)g(upp) q(er)75 2074 y(b)q(ound)e(mark)o(ers)e(\()p Fd(MPI)p 490 2074 V 14 w(LB)h Fq(and)g Fd(MPI)p 740 2074 V 14 w(UB)p Fq(\).)f(This)h(is)h(useful,)g(as)f(it)g(allo)o(ws)g(to)f(con)o (trol)h(the)g(stride)g(of)75 2131 y(successiv)o(e)e(datat)o(yp)q(es)e (that)g(are)g(replicated)i(b)o(y)e(datat)o(yp)q(e)g(constructors,)g(or) g(are)g(replicated)i(b)o(y)f(the)75 2187 y Fj(count)k Fq(argumen)o(t)e(in)i(a)e(send)h(or)g(reciev)o(e)g(call.)32 b(Ho)o(w)o(ev)o(er,)19 b(the)f(curren)o(t)h(mec)o(hanism)h(for)e(ac)o (hieving)75 2244 y(it)h(is)h(painful;)i(also)d(it)h(is)f(restrictiv)o (e,)i(as)d Fd(MPI)p 906 2244 V 15 w(LB)h Fq(and)g Fd(MPI)p 1157 2244 V 14 w(UB)g Fq(are)f(\\stic)o(ky":)28 b(once)19 b(presen)o(t)g(in)h(a)75 2300 y(datat)o(yp)q(e,)d(they)h(cannot)g(b)q (e)g(o)o(v)o(eriden)g(\(e.g.,)f(the)h(upp)q(er)h(b)q(ound)f(can)g(b)q (e)g(mo)o(v)o(ed)g(up,)g(b)o(y)g(adding)g(a)75 2357 y(new)13 b Fd(MPI)p 243 2357 V 14 w(UB)g Fq(mark)o(er,)f(but)h(cannot)g(b)q(e)g (mo)o(v)o(ed)g(do)o(wn)f(b)q(elo)o(w)i(an)f(existing)h Fj(MPI)p 1474 2357 14 2 v 15 w(UB)g Fq(mark)o(er\).)k(A)13 b(new)75 2413 y(function)j(is)g(pro)o(vided)g(to)e(facilitate)i(these)g (c)o(hanges.)1967 46 y Fh(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 7 9 7 8 bop 75 -100 a Fg(3.7.)34 b(MPI-1.0)14 b(AND)i(MPI-1.1)e(CLARIFICA)l (TIONS)802 b Fq(7)75 45 y Fj(MPI)p 160 45 14 2 v 16 w(SET)p 259 45 V 17 w(EXTENT\(datat)o(yp)q(e,)16 b(lb,)f(extent\))117 122 y Fl(INOUT)62 b Fj(datat)o(yp)q(e)424 b Fl(datat)o(yp)q(e)14 b(to)g(up)q(date)h(exten)o(t)117 197 y(IN)155 b Fj(true)p 396 197 V 17 w(lb)461 b Fl(new)15 b(lo)o(w)o(er)e(b)q(ound)h(of)f (datat)o(yp)q(e)117 273 y(IN)155 b Fj(true)p 396 273 V 17 w(extent)379 b Fl(new)15 b(exten)o(t)f(of)g(datat)o(yp)q(e)75 397 y Fk(int)23 b(MPI)p 245 397 15 2 v 17 w(Set)p 334 397 V 17 w(extent\(MPI)p 591 397 V 16 w(Datatype)g(datatype,)f(MPI)p 1132 397 V 17 w(Aint)h(lb,)h(MPI)p 1436 397 V 17 w(Aint)f(extent\))75 483 y(MPI)p 150 483 V 17 w(TRUE)p 263 483 V 16 w(EXTENT\(DATATYPE,)f (LB,)i(EXTENT,)e(IERROR\))170 540 y(INTEGER)h(DATATYPE,)g(LB,)g (EXTENT,)g(IERROR)166 626 y Fq(Resets)13 b(the)g(lo)o(w)o(er)g(b)q (ound)h(and)f(the)h(exten)o(t)e(\(and,)h(hence,)h(the)g(upp)q(er)g(b)q (ound\))f(of)g Fj(datat)o(yp)q(e)p Fq(.)21 b(The)75 683 y(new)e(settings)f(will)i(a\013ect)e(the)h(outcome)f(of)g(send)h(or)f (receiv)o(e)h(op)q(erations)g(that)e(use)i Fj(datat)o(yp)q(e)p Fq(,)i(and)75 739 y(a\013ects)14 b(the)i(v)m(alue)g(of)f(datat)o(yp)q (es)g(constructed)g(using)h Fj(datat)o(yp)q(e)h Fq(as)e(an)g(agumen)o (t,)f(after)h(the)g(call)i(w)o(as)75 796 y(made.)i(Ho)o(w)o(ev)o(er,)11 b(datat)o(yp)q(es)h(that)f(w)o(ere)h(previously)i(constructed)e(using)h Fj(datat)o(yp)q(e)h Fq(are)e(not)g(a\013ected.)189 902 y Fe(R)n(ationale.)189 977 y Fq(This)h(is)g(consisten)o(t)f(to)g(the)h (MPI)f(statemen)o(t)g(that)g(\\the)g(system)g(b)q(eha)o(v)o(es)h(as)f (if)h(input)g(datat)o(yp)q(e)189 1034 y(argumen)o(ts)h(to)h(deriv)o(ed) h(datat)o(yp)q(e)e(constructors)h(are)g(passed)g(b)o(y)g(v)m(alue".)189 1109 y(\()p Fe(End)g(of)i(r)n(ationale.)p Fq(\))189 1215 y Fe(A)n(dvic)n(e)g(to)h(users.)51 b Fq(It)17 b(is)g(recommended)h (that)f(uses)g(use)g(the)g(function)h Fj(MPI)p 1571 1215 14 2 v 16 w(SET)p 1670 1215 V 17 w(EXTENT)p Fq(,)189 1271 y(rather)c(than)h Fd(MPI)p 508 1271 13 2 v 15 w(LB)g Fq(and)g Fd(MPI)p 751 1271 V 15 w(UB)f Fq(mark)o(ers.)189 1347 y(\()p Fe(End)h(of)i(advic)n(e)f(to)g(users.)p Fq(\))1875 1408 y Fi(?)g Fh(\(Oct\))75 1540 y Fn(3.7)59 b(MPI-1.0)19 b(and)h(MPI-1.1)f(Cla)n(ri\014cations)75 1643 y Ff(3.7.1)49 b(Cla)o(ri\014cation)18 b(of)e(MPI)p 627 1643 15 2 v 18 w(FINALIZE)1875 1672 y Fi(>)g Fh(\(Oct\))75 1776 y Fm(Curren)o(t)e(Status:)j Fl(Not)d(v)o(oted)g(on.)166 1880 y Fq(This)j(routine)f(cleans)h(up)f(all)h(MPI)f(state.)22 b(Eac)o(h)16 b(pro)q(cess)g(m)o(ust)f(call)i Fj(MPI)p 1485 1880 14 2 v 16 w(FINALIZE)e Fq(b)q(efore)i(it)75 1936 y(exits.)j(Once)c(this)f(routine)g(returns,)f(no)h(MPI)g(routine)g (\(ev)o(en)g(not)f Fj(MPI)p 1338 1936 V 16 w(INIT)p Fq(\))f(ma)o(y)h(b) q(e)i(called.)21 b(Eac)o(h)75 1992 y(pro)q(cess)11 b(m)o(ust)e (complete)j(an)o(y)e(p)q(ending)i(comm)o(unication)f(it)g(initiated)h (b)q(efore)e(it)h(calls)g Fj(MPI)p 1656 1992 V 16 w(FINALIZE)p Fq(.)75 2049 y(If)16 b(the)g(call)h(returns,)e(eac)o(h)h(pro)q(cess)g (ma)o(y)f(con)o(tin)o(ue)i(lo)q(cal)f(computations,)g(or)f(exit,)h (without)g(partici-)75 2105 y(pating)g(in)g(further)g Fj(MPI)f Fq(comm)o(unication)h(with)g(other)f(pro)q(cesses.)22 b Fj(MPI)p 1362 2105 V 16 w(FINALIZE)14 b Fq(is)j(collectiv)o(e)g(on)75 2162 y Fj(MPI)p 160 2162 V 16 w(COMM)p 318 2162 V 16 w(W)o(ORLD)p Fq(.)1369 b Fi(>)16 b Fh(\(Oct\))166 2218 y Fq(It)f(is)h(not)f(required)h(that)e Fj(MPI)p 703 2218 V 16 w(Finalize)h Fq(return,)g(ev)o(en)g(on)h(one)f(pro)q(cess.)1875 2220 y Fi(?)h Fh(\(Oct\))189 2325 y Fe(A)n(dvic)n(e)11 b(to)i(implementors.)38 b Fq(Ev)o(en)11 b(though)g(a)g(pro)q(cess)h (has)f(completed)i(all)f(the)f(comm)o(unication)189 2381 y(it)j(initiated,)h(suc)o(h)g(comm)o(unication)f(ma)o(y)f(not)h(y)o(et) g(b)q(e)g(completed)h(from)e(the)h(viewp)q(oin)o(t)h(of)f(the)189 2437 y(underlying)j(MPI)f(system.)k(E.g.,)14 b(a)h(blo)q(c)o(king)i (send)f(ma)o(y)f(ha)o(v)o(e)g(completed,)i(ev)o(en)e(though)h(the)189 2494 y(data)k(is)h(still)h(bu\013ered)f(at)f(the)h(sender.)37 b(The)20 b(MPI)h(implemen)o(tation)h(m)o(ust)e(ensure)h(that)f(a)189 2550 y(pro)q(cess)14 b(has)g(completed)h(an)o(y)f(in)o(v)o(olv)o(emen)o (t)g(in)h(MPI)f(comm)o(unication)h(b)q(efore)f Fj(MPI)p 1669 2550 V 16 w(FINALIZE)189 2607 y Fq(returns.)26 b(Th)o(us,)17 b(if)h(a)e(pro)q(cess)i(exits)g(after)e(the)h(call)i(to)d Fj(MPI)p 1271 2607 V 16 w(FINALIZE)p Fq(,)g(this)i(will)h(not)e(cause) 189 2663 y(an)e(ongoing)g(comm)o(unication)h(to)e(fail.)21 b(\()p Fe(End)16 b(of)g(advic)n(e)g(to)h(implementors.)p Fq(\))-32 46 y Fh(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 8 10 8 9 bop 75 -100 a Fq(8)970 b Fg(CHAPTER)15 b(3.)35 b(MISCELLANY)17 b(F)o(OR)e(1.2)189 45 y Fe(A)n(dvic)n(e)i(to)i(implementors.)54 b Fq(Although)18 b(it)g(is)g(not)f(required,)i(it)f(has)f(b)q(een)i (found)f(that)f(users)189 102 y(exp)q(ect)f(that)f(at)f(least)i(one)f (pro)q(cess)h(return,)f(so)g(that)g(they)h(can)f(kno)o(w)g(that)g(the)g (MPI)h(p)q(ortion)189 158 y(of)e(the)g(computation)g(is)h(o)o(v)o(er.)k (In)c(addition,)g(in)g(a)f(POSIX)i(en)o(vironmen)o(t,)e(they)h(ma)o(y)e (desire)j(to)189 214 y(supply)f(an)f(exit)g(co)q(de)h(for)e(eac)o(h)h (pro)q(cess)g(that)f(returns)h(from)f Fj(MPI)p 1357 214 14 2 v 16 w(FINALIZE)p Fq(.)g(\()p Fe(End)h(of)h(advic)n(e)189 271 y(to)h(implementors.)p Fq(\))-117 332 y Fi(>)f Fh(\(Oct\))75 464 y Fn(3.8)59 b(Determining)19 b(Whether)g(MPI)h(Has)g(Finished)75 613 y Fm(Curren)o(t)14 b(Status:)j Fl(No)d(v)o(otes.)166 716 y Fq(One)i(of)f(the)h(goals)f(of)h Fj(MPI)f Fq(w)o(as)f(to)h(allo)o (w)h(for)f(la)o(y)o(ered)h(libraries.)23 b(In)16 b(order)f(for)g(a)h (library)g(to)f(do)75 773 y(this)k(cleanly)l(,)i(it)e(needs)g(to)f(kno) o(w)g(if)h Fj(MPI)f Fq(is)i(activ)o(e.)30 b(In)19 b Fj(MPI-1)f Fq(the)h(function)g Fj(MPI)p 1590 773 V 16 w(Initialized)g Fq(w)o(as)75 829 y(pro)o(vided)c(to)g(tell)g(if)g Fj(MPI)f Fq(had)h(b)q(een)h(initialzed.)23 b(The)15 b(problem)g(arises)g(in)g (kno)o(wing)g(if)g Fj(MPI)f Fq(has)h(b)q(een)75 886 y(\014nalized.)31 b(Once)19 b Fj(MPI)f Fq(has)g(b)q(een)h(\014nalized)h(it)f(is)g(no)f (longer)g(activ)o(e)h(and)f(cannot)g(b)q(e)h(restarted.)28 b(A)75 942 y(library)16 b(needs)g(to)e(b)q(e)i(able)g(to)e(determine)i (this)f(to)g(act)f(accordingly)l(.)22 b(Do)14 b(ac)o(hiev)o(e)i(this)f (the)g(follo)o(wing)75 999 y(function)h(is)g(needed:)75 1150 y Fj(MPI)p 160 1150 V 16 w(FINALIZED\(\015ag\))117 1227 y Fl(OUT)108 b Fj(\015ag)518 b Fl(true)15 b(if)e Fd(MPI)g Fl(w)o(as)h(\014nalized)g(\(logical\).)75 1351 y Fk(int)23 b(MPI)p 245 1351 15 2 v 17 w(Finalized\(int)f(*flag\))75 1438 y(MPI)p 150 1438 V 17 w(FINALIZED\(flag,)g(IERROR\)\))170 1494 y(FLAG,)i(IERROR)75 1581 y(int)f(MPI::Finalized\(bool)f (&flag\)\)?????)166 1667 y Fq(This)17 b(function)g(returns)f(true)g(if) h Fj(MPI)p 833 1667 14 2 v 16 w(FINALIZE)e Fq(has)h(returned.)23 b(Th)o(us,)16 b(it)h(will)h(return)e(false)g(if)75 1724 y(called)k(in)f(a)f(callbac)o(k)h(routine)f(in)o(v)o(ok)o(ed)h(b)o(y)f Fj(MPI)p 960 1724 V 16 w(FINALIZE)p Fq(.)f(It)h(is)g(legal)h(to)f(call) h Fj(MPI)p 1636 1724 V 16 w(FINALIZED)75 1780 y Fq(b)q(efore)c Fj(MPI)p 296 1780 V 16 w(INIT)g Fq(and)g(after)g Fj(MPI)p 694 1780 V 15 w(FINALIZE)p Fq(.)-1030 b Fi(?)15 b Fh(\(Oct\))-117 1839 y Fi(>)g Fh(\(Oct\))75 1902 y Ff(3.8.1)49 b(Cla)o(ri\014cation)18 b(of)e(status)f(after)h(MPI)p 875 1902 15 2 v 18 w(Isend)75 2035 y Fm(Curren)o(t)e(Status:)j Fl(No)d(v)o(otes.)166 2138 y Fq(A)22 b Fj(status)i Fq(returned)e(b)o(y)g(a)g(call)h(to)e Fj(MPI)p 900 2138 14 2 v 16 w(W)l(AIT)p Fq(,)g Fj(MPI)p 1145 2138 V 16 w(TEST)p Fq(,)h(or)f(an)o(y)h(of)f(the)h(other)g(deriv)o (ed)75 2195 y(functions,)j(where)f(the)f Fj(request)h Fq(corresp)q(onds)g(to)e(a)h(send)g(call,)j(is)e(unde\014ned,)i(with)d (t)o(w)o(o)f(excep-)75 2251 y(tions:)36 b(The)24 b(error)f(status)f (\014eld)j(will)g(con)o(tain)e(v)m(alid)i(information)f(if)f(the)h(w)o (ait)f(or)g(test)f(call)j(re-)75 2308 y(turned)20 b(with)g Fd(MPI)p 411 2308 13 2 v 15 w(ERROR)p 563 2308 V 13 w(IN)p 617 2308 V 15 w(ST)m(A)m(TUS)p Fq(;)f(and)h(the)g(returned)g(status)f (can)h(b)q(e)g(querried)h(b)o(y)f(the)g(call)75 2364 y Fj(MPI)p 160 2364 14 2 v 16 w(TEST)p 290 2364 V 16 w(CANCELLED)p Fq(.)-690 b Fi(?)15 b Fh(\(Oct\))75 2486 y Ff(3.8.2)49 b(Cla)o(ri\014cation)18 b(of)e(MPI)p 627 2486 15 2 v 18 w(INTERCOMM)p 936 2486 V 20 w(CREA)l(TE)75 2619 y Fm(Curren)o(t)e(Status:)j Fl(P)o(assed)e(once.)1967 46 y Fh(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 9 11 9 10 bop 75 -100 a Fg(3.8.)34 b(DETERMINING)15 b(WHETHER)h Fj(MPI)e Fg(HAS)i(FINISHED)630 b Fq(9)75 45 y Fj(The)11 b(Problem:)44 b Fq(The)12 b(MPI)e(1.1)g(standard)h(sa)o(ys,)f(in)i(the) f(discussion)h(of)f Fj(MPI)p 1389 45 14 2 v 16 w(INTERCOMM)p 1679 45 V 16 w(CREA)l(TE)p Fq(,)75 102 y(b)q(oth)k(that)189 195 y(The)g(groups)g(m)o(ust)g(b)q(e)g(disjoin)o(t)75 289 y(and)g(that)189 383 y(The)g(leaders)h(ma)o(y)e(b)q(e)i(the)g(same) e(pro)q(cess.)75 477 y(T)l(o)d(further)h(m)o(uddy)g(the)g(w)o(aters,)e (the)i(reason)f(giv)o(en)i(for)e(\\The)g(groups)g(m)o(ust)h(b)q(e)g (disjoin)o(t")g(is)g(based)g(on)75 533 y(concerns)18 b(ab)q(out)g(the)g(implmen)o(tation)g(of)g Fj(MPI)p 922 533 V 15 w(INTERCOMM)p 1211 533 V 17 w(CREA)l(TE)h Fq(that)e(are)g(not) h(applicable)75 590 y(for)d(the)g(case)g(where)h(the)f(leaders)h(are)f (the)g(same)g(pro)q(cess.)75 710 y Fj(The)h(Fix:)45 b Fq(Delete)15 b(the)h(text:)189 804 y(\(the)f(t)o(w)o(o)e(leaders)j (could)h(b)q(e)e(the)h(same)f(pro)q(cess\))75 897 y(from)f(the)i (discussion)h(of)d Fj(MPI)p 610 897 V 16 w(INTERCOMM)p 900 897 V 17 w(CREA)l(TE)p Fq(.)166 954 y(Replace)j(the)e(text:)189 1048 y(All)f(in)o(ter-comm)o(unicator)g(constructors)e(are)h(blo)q(c)o (king)h(and)g(require)g(that)e(the)i(lo)q(cal)g(and)189 1104 y(remote)g(groups)h(b)q(e)h(disjoin)o(t)g(in)g(order)f(to)f(a)o(v) o(oid)h(deadlo)q(c)o(k.)75 1198 y(with)189 1292 y(All)f(in)o(ter-comm)o (unicator)g(constructors)e(are)h(blo)q(c)o(king)h(and)g(require)g(that) e(the)i(lo)q(cal)g(and)189 1348 y(remote)g(groups)h(b)q(e)h(disjoin)o (t.)289 1454 y Fe(A)n(dvic)n(e)e(to)h(users.)40 b Fq(The)14 b(groups)g(m)o(ust)f(b)q(e)i(disjoin)o(t)g(for)e(sev)o(eral)h(reasons.) 19 b(Primar-)289 1511 y(ily)l(,)12 b(this)f(is)g(the)f(in)o(ten)o(t)h (of)f(the)g(in)o(tercomm)o(unicators)h({)f(to)g(pro)o(vide)h(a)f(comm)o (unicator)289 1567 y(for)17 b(comm)o(unication)i(b)q(et)o(w)o(een)g (disjoin)o(t)f(groups.)29 b(This)19 b(is)f(re\015ected)h(in)g(the)g (de\014-)289 1624 y(nition)e(of)f Fj(MPI)p 559 1624 V 15 w(INTERCOMM)p 848 1624 V 17 w(MERGE)p Fq(,)g(whic)o(h)h(allo)o(ws)g (the)f(user)g(to)f(con)o(trol)h(the)289 1680 y(ranking)k(of)g(the)g (pro)q(cesses)g(in)h(the)g(created)f(in)o(tracomm)o(unicator;)h(this)g (ranking)289 1737 y(mak)o(es)c(little)i(sense)f(if)g(the)g(groups)g (are)f(not)h(disjoin)o(t.)28 b(In)18 b(addition,)h(the)f(natural)289 1793 y(extension)c(of)g(collectiv)o(e)i(op)q(erations)e(to)f(in)o (tercomm)o(unicators)h(\(b)q(eing)h(considered)289 1850 y(for)i(MPI-2\))h(mak)o(es)f(the)h(most)g(sense)g(when)h(the)f(groups)g (are)f(disjoin)o(t.)30 b(It)18 b(is)g(the)289 1906 y(in)o(ten)o(t)i(of) g(the)g(MPI)g(de\014nition)i(not)e(to)f(preclude)j(future)f (extensions.)35 b(\()p Fe(End)20 b(of)289 1963 y(advic)n(e)c(to)g (users.)p Fq(\))75 2084 y Ff(3.8.3)49 b(Cla)o(ri\014cation)18 b(of)e(Binding)i(of)f(MPI)p 854 2084 15 2 v 18 w(T)l(yp)q(e)p 971 2084 V 18 w(size)75 2217 y Fm(Curren)o(t)d(Status:)j Fl(P)o(assed)e(once.)166 2321 y Fq(This)i(clari\014cation)h(is)f (needed)h(in)f(the)g(MPI-1)f(description)i(of)e Fj(MPI)p 1369 2321 14 2 v 16 w(T)l(yp)q(e)p 1477 2321 V 17 w(size)p Fq(,)h(since)g(the)g(issue)75 2378 y(rep)q(eatedly)f(arises.)k(It)c(is) f(a)g(clari\014cation)i(of)e(the)g(binding.)189 2484 y Fe(A)n(dvic)n(e)20 b(to)j(users.)76 b Fq(The)21 b(binding)i(of)e Fj(MPI)p 1004 2484 V 16 w(T)l(yp)q(e)p 1112 2484 V 17 w(size)p Fq(,)i(sp)q(eci\014cally)l(,)i(whether)d(the)f(output)189 2540 y(argumen)o(t)e(in)i(C)g(is)f(of)g(t)o(yp)q(e)h Fk(int)f Fq(as)g(in)h(the)f(curren)o(t)h(standard)f(or)f(should)j(b)q (e)f(c)o(hanged)g(to)189 2597 y(something)14 b(else,)i(has)e(b)q(een)i (extensiv)o(ely)f(discussed)h(b)o(y)f(the)g(MPI)f(F)l(orum.)19 b(The)c(\014nal)g(decision)189 2653 y(is)g(to)g(lea)o(v)o(e)g(it)h(an)f Fk(int)p Fq(.)k(\()p Fe(End)d(of)g(advic)n(e)g(to)h(users.)p Fq(\))-32 46 y Fh(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 10 12 10 11 bop 75 -100 a Fq(10)952 b Fg(CHAPTER)16 b(3.)29 b(MISCELLANY)17 b(F)o(OR)e(1.2)75 45 y Ff(3.8.4)49 b(Cla)o (ri\014cation)18 b(of)e(MPI)p 627 45 15 2 v 18 w(REDUCE)75 131 y Fq(The)f(curren)o(t)h(text)e(on)h(p.)20 b(115)15 b(lines)i(25-28)d(from)g(MPI-1.1)h(\(June)h(12,)e(1995\))f(sa)o(ys:)166 187 y(The)j Fj(datat)o(yp)q(e)i Fq(argumen)o(t)e(of)f Fj(MPI)p 783 187 14 2 v 16 w(REDUCE)i Fq(m)o(ust)f(b)q(e)g(compatible)i (with)e Fj(op)p Fq(.)23 b(Prede\014ned)18 b(op-)75 244 y(erators)h(w)o(ork)g(only)i(with)f(the)g Fj(MPI)g Fq(t)o(yp)q(es)g (listed)h(in)g(Sec.)35 b(4.9.2)18 b(and)j(Sec.)35 b(4.9.3.)d (User-de\014ned)75 300 y(op)q(erators)14 b(ma)o(y)h(op)q(erate)g(on)g (general,)g(deriv)o(ed)h(datat)o(yp)q(es.)166 357 y(This)g(text)e(is)i (c)o(hanged)g(to:)166 413 y(The)g Fj(datat)o(yp)q(e)i Fq(argumen)o(t)e(of)f Fj(MPI)p 783 413 V 16 w(REDUCE)i Fq(m)o(ust)f(b)q(e)g(compatible)i(with)e Fj(op)p Fq(.)23 b(Prede\014ned)18 b(op-)75 470 y(erators)c(w)o(ork)h(only)h(with)f(the) h Fj(MPI)f Fq(t)o(yp)q(es)g(listed)i(in)f(Sec.)21 b(4.9.2)14 b(and)h(Sec.)21 b(4.9.3.)e(F)l(urthermore,)c(the)75 526 y Fj(datat)o(yp)q(e)20 b Fq(and)e Fj(op)h Fq(giv)o(en)f(for)g (prede\014ned)i(op)q(erators)d(m)o(ust)h(b)q(e)h(the)f(same)g(on)g(all) h(pro)q(cesses.)30 b(User-)75 583 y(de\014ned)15 b(op)q(erators)e(ma)o (y)g(op)q(erate)g(on)h(general,)g(deriv)o(ed)h(datat)o(yp)q(es)e (without)g(the)h(restrictions)g(ab)q(o)o(v)o(e)75 639 y(for)h(prede\014ned)h(op)q(erators.)-681 b Fi(>)15 b Fh(\(Oct\))166 695 y Fq(Note)10 b(that)f(it)i(is)f(p)q(ossible)i(for)e (users)g(to)f(supply)j(di\013eren)o(t)e(user-de\014ned)i(op)q(erations) e(to)g Fj(MPI)p 1750 695 V 16 w(REDUCE)75 752 y Fq(in)23 b(eac)o(h)e(pro)q(cess.)40 b Fj(MPI)21 b Fq(do)q(es)h(not)f(de\014ne)i (whic)o(h)g(op)q(erations)e(are)h(used)g(in)h(this)f(case)f(on)h(whic)o (h)75 808 y(op)q(erands.)189 915 y Fe(A)n(dvic)n(e)e(to)h(users.)71 b Fq(Users)20 b(should)i(mak)o(e)d(no)i(assumptions)f(ab)q(out)g(ho)o (w)g Fj(MPI)p 1638 915 V 16 w(RECUCE)h Fq(is)189 971 y(implemen)o(ted.)29 b(Safest)17 b(is)h(to)f(ensure)i(that)e(the)g (same)h(function)g(is)g(passed)g(to)f Fj(MPI)p 1685 971 V 16 w(REDUCE)189 1028 y Fq(b)o(y)e(eac)o(h)g(pro)q(cess.)20 b(\()p Fe(End)c(of)g(advic)n(e)g(to)h(users.)p Fq(\))-117 1089 y Fi(?)e Fh(\(Oct\))75 1221 y Fn(3.9)59 b(New)20 b(Datat)n(yp)r(es)75 1324 y Ff(3.9.1)49 b(Simple)17 b(Struct)d(T)l(yp)q (es)75 1457 y Fm(Curren)o(t)g(Status:)j Fl(Not)d(v)o(oted)g(on)f(y)o (et.)75 1655 y Fj(MPI)p 160 1655 V 16 w(TYPE)p 293 1655 V 17 w(SIMPLE)p 469 1655 V 15 w(STRUCT\(count,)k(a)o(rra)o(y)p 908 1655 V 14 w(of)p 959 1655 V 16 w(blo)q(cklengths,)h(a)o(rra)o(y)p 1334 1655 V 14 w(of)p 1385 1655 V 16 w(t)o(yp)q(es,)e(newt)o(yp)q(e\)) 117 1732 y Fl(IN)155 b Fj(count)482 b Fl(n)o(um)o(b)q(er)13 b(of)h(blo)q(c)o(ks)g(\(in)o(teger\))117 1807 y(IN)155 b Fj(a)o(rra)o(y)p 416 1807 V 15 w(of)p 468 1807 V 16 w(blo)q(cklengths)191 b Fl(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h (eac)o(h)g(blo)q(c)o(k)g(\(arra)o(y)g(of)f(in)o(teger\))117 1882 y(IN)155 b Fj(a)o(rra)o(y)p 416 1882 V 15 w(of)p 468 1882 V 16 w(t)o(yp)q(es)327 b Fl(t)o(yp)q(e)15 b(of)e(elemen)o(ts)h (in)f(eac)o(h)h(blo)q(c)o(k)g(\(arra)o(y)g(of)f(handle\))117 1958 y(OUT)108 b Fj(newt)o(yp)q(e)433 b Fl(new)15 b(datat)o(yp)q(e)f (\(handle\))166 2082 y Fq(Eac)o(h)c(t)o(yp)q(e)h(in)g(the)g Fj(a)o(rra)o(y)p 592 2082 V 14 w(of)p 643 2082 V 16 w(t)o(yp)q(es)h Fq(list)g(m)o(ust)e(b)q(e)h(a)f Fe(p)n(ortable)h Fq(datat)o(yp)q(e:)17 b(I.e.,)10 b(a)h(basic)g(datat)o(yp)q(e,)f(or)75 2138 y(a)g(datat)o(yp)q(e)f(constructed)i(from)e(a)h(p)q(ortable)h(datat)o (yp)q(e)e(using)i(one)f(of)g(the)g(constructors)g Fj(MPI)p 1681 2138 V 15 w(TYPE)p 1813 2138 V 17 w(CONTIGUOUS,)75 2195 y(MPI)p 160 2195 V 16 w(TYPE)p 293 2195 V 17 w(VECTOR,)16 b(MPI)p 600 2195 V 15 w(TYPE)p 732 2195 V 17 w(INDEXED,)f Fq(or)g Fj(MPI)p 1110 2195 V 16 w(TYPE)p 1243 2195 V 16 w(SIMPLE)p 1418 2195 V 16 w(STRUCT)p Fq(.)166 2251 y(This)22 b(t)o(yp)q(e)g(constructor)f(is)i(similar)f(to)g Fj(MPI)p 988 2251 V 15 w(TYPE)p 1120 2251 V 17 w(STRUCT)p Fq(,)g(except)h(that)e(the)h(user)g(do)q(es)75 2308 y(not)e(supply)h (an)f(arra)o(y)e(of)i(displacemen)o(ts.)35 b(Instead,)22 b(the)e(successiv)o(e)h(blo)q(c)o(ks)f(are)g(assumed)g(to)f(b)q(e)75 2364 y(con)o(tiguous.)g(P)o(adding)11 b(spaces)h(are)e(added,)j (according)e(to)g(the)g(default)h(alignmen)o(t)g(rules)g(for)e (structure)75 2421 y(comp)q(onen)o(ts.)21 b(This)c(facilitates)f(the)g (construction)g(of)f(datat)o(yp)q(es)g(for)g(structures,)g(as)h(the)f (user)h(need)75 2477 y(not)f(compute)g(displacemen)o(ts.)-743 b Fi(>)15 b Fh(\(Oct\))166 2534 y Fq(This)h(can)f(probably)h(b)q(e)g (more)e(elegan)o(tly)i(expressed)g(in)g(terms)f(of)g(exten)o(ts.)-117 2536 y Fi(?)g Fh(\(Oct\))75 2640 y Fb(Example)j(3.1)k Fq(The)14 b(follo)o(wing)g(co)q(de)f(declares)h(a)f(structure)g(t)o(yp) q(e)g(and)h(creates)f(a)g(datat)o(yp)q(e)f(for)h(that)75 2696 y(structure)i(t)o(yp)q(e.)1967 46 y Fh(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 11 13 11 12 bop 75 -100 a Fg(3.9.)34 b(NEW)15 b(D)o(A)l(T)l(A)l(TYPES)1245 b Fq(11)75 45 y Fk(struct)23 b(record)g({)147 102 y(char)g(name;)147 158 y(double)g(position[3];)147 214 y(float)47 b(mass;)147 271 y(})75 384 y(MPI_Datatype)22 b(record_type;)75 440 y(MPI_datatype)g(types[3])h(=)h({MPI_CHAR,)e(MPI_DOUBLE,)h(MPI_FLOAT};) 75 497 y(int)238 b(lengths[3])23 b(=)g({1,)h(3,)f(1};)75 610 y(MPI_Type_simple_struct\()e(3,)i(lengths,)g(types,)g (record_type\);)189 704 y Fe(R)n(ationale.)76 b Fq(The)21 b(co)q(de)h(ab)q(o)o(v)o(e)f(is)h(m)o(uc)o(h)f(simpler)h(and)g(more)e (natural)i(than)f(the)g(curren)o(t)189 760 y(approac)o(h,)14 b(that)h(uses)g Fj(MPI)p 677 760 14 2 v 16 w(TYPE)p 810 760 V 17 w(STRUCT)p Fq(,)g(and)h(requires)g(to)e(compute)h(displacemen) o(ts.)189 834 y(It)i(also)g(pro)o(vides)h(a)f(de\014nition)i(of)e(the)h (record)f(t)o(yp)q(e)h(that)e(is)i(arc)o(hitecture)g(indep)q(enden)o(t) i(\(as-)189 890 y(suming)c(that)e(compilers)j(don't)e(\014ddle)i(with)e (the)h(order)f(of)g(\014elds,)h(whic)o(h)g(is)g(true)f(in)i(C\))d({)i (this)189 947 y(facilitates)f(transfer)e(of)h(records)g(using)h(RMA,)f (an)g(\014ts)g(w)o(ell)h(with)g(v)m(arious)f(prop)q(osals)h(for)e (third)189 1003 y(part)o(y)h(transfers.)19 b(\()p Fe(End)d(of)g(r)n (ationale.)p Fq(\))189 1097 y Fe(A)n(dvic)n(e)h(to)h(users.)50 b Fq(Datat)o(yp)q(es)16 b(constructed)h(using)h Fj(MPI)p 1231 1097 V 16 w(TYPE)p 1364 1097 V 16 w(SIMPLE)p 1539 1097 V 16 w(STRUCT)g Fq(should)189 1154 y(b)q(e)i(used)g(only)g(for)f (structures)g(that)g(use)h(the)g(default)g(alignmen)o(t)g(rules.)34 b(They)20 b(should)g(not)189 1210 y(b)q(e)c(used)h(if)f(alignmen)o(t)g (rules)h(ha)o(v)o(e)f(b)q(een)h(c)o(hanged,)f(using)g(compiler)h (options)f(or)g(compilation)189 1267 y(pragmas.)j(\()p Fe(End)c(of)i(advic)n(e)f(to)g(users.)p Fq(\))189 1361 y Fe(A)n(dvic)n(e)g(to)h(implementors.)47 b Fq(The)16 b(datat)o(yp)q(e)g(constructor)f Fj(MPI)p 1330 1361 V 16 w(TYPE)p 1463 1361 V 17 w(SIMPLE)p 1639 1361 V 15 w(STRUCT)i Fq(is)189 1417 y(most)e(easily)i(implemen)o(ted)h(on)e (systems)g(that)f(use)h Fe(natur)n(al)g Fq(data)g(alignmen)o(t)h (rules.)23 b(On)17 b(suc)o(h)189 1474 y(systems,)12 b(eac)o(h)h(basic)g (datat)o(yp)q(e)f(has)g(an)h(exten)o(t)f(and)h(an)g(alignmen)o(t;)h (the)e(alignmen)o(t)i(of)e(a)g(com-)189 1530 y(p)q(ound)j(t)o(yp)q(e)g (is)h(the)f(strictest)f(alignmen)o(t)i(of)e(a)g(comp)q(onen)o(t.)20 b(P)o(adding)c(is)f(added)g(so)g(that)f(eac)o(h)189 1587 y(comp)q(onen)o(t)f(is)h(aligned)g(at)f(its)g(natural)g(alignmen)o(t.) 20 b(W)l(e)13 b(can)g(then)h(deriv)o(e)g(ob)o(vious)f(de\014nitions)189 1643 y(for)h(the)h(exten)o(t)g(and)h(alignmen)o(t)g(of)e(p)q(ortable)i (MPI)f(datat)o(yp)q(es:)189 1700 y(The)g(exten)o(t)f(and)h(alignmen)o (t)g(of)f(a)h(basic)g(datat)o(yp)q(e)f(is)i(the)e(exten)o(t)h(and)g (alignmen)o(t)g(of)f(the)h(cor-)189 1756 y(resp)q(onding)h(language)f (t)o(yp)q(e.)189 1812 y(The)10 b(alignmen)o(t)h(of)f(a)f(datat)o(yp)q (e)h(built)i(with)e(one)g(of)g(the)g(constructors)g Fj(MPI)p 1487 1812 V 15 w(TYPE)p 1619 1812 V 17 w(CONTIGUOUS,)189 1869 y(MPI)p 274 1869 V 15 w(TYPE)p 406 1869 V 17 w(VECTOR,)26 b Fq(and)e Fj(PI)p 780 1869 V 16 w(TYPE)p 913 1869 V 17 w(INDEXED)h Fq(is)g(the)f(alignmen)o(t)h(of)f(the)h(comp)q(onen)o(t) 189 1925 y(datat)o(yp)q(e;)14 b(the)h(exten)o(t)g(of)g(the)g(comp)q (ound)h(datat)o(yp)q(e)f(is)g(as)g(de\014ned)i(in)f(MPI1.)189 1982 y(The)h(alignmen)o(t)g(of)f(a)g(datat)o(yp)q(e)h(built)h(b)o(y)e (a)h(call)g(to)f Fj(MPI)p 1221 1982 V 16 w(TYPE)p 1354 1982 V 17 w(SIMPLE)p 1530 1982 V 16 w(STRUCT\(n,)h(lens,)189 2038 y(t)o(yp)q(es,)i(newt)o(yp)q(e\))h Fq(is)e(max)643 2045 y Fa(i)672 2038 y Fk(alignment)p Fq(\()p Fk(types)p Fq([)p Fk(i)p Fq(]\);)e(the)i(exten)o(t)g(is)g(computed)h(b)o(y)f(la)o (ying)g(out)189 2095 y(the)h(comp)q(onen)o(ts)g(with)h(minimal)h (padding)f(added)g(so)f(that)g(eac)o(h)h(comp)q(onen)o(t)f(starts)f(at) h(its)189 2151 y(alignmen)o(t.)189 2225 y(Some)g(compilers)h(treat)f (the)g(\014rst)g(comp)q(onen)o(t)g(of)g(a)g(structure)g(di\013eren)o (tly)l(.)33 b(E.g.,)18 b(the)i(IBM)189 2281 y(compiler)d(aligns)h (doubles)f(at)f(w)o(ord)g(b)q(oundaries,)i(but)f(align)g(a)f(structure) g(that)g(starts)f(with)i(a)189 2338 y(double)j(comp)q(onen)o(t)f(at)f (doublew)o(ord)i(b)q(oundaries.)32 b(The)19 b(de\014nition)i(of)d(the)h (alignmen)o(t)h(of)e(a)189 2394 y(datat)o(yp)q(e)h(built)i(with)f Fj(MPI)p 690 2394 V 16 w(TYPE)p 823 2394 V 16 w(SIMPLE)p 998 2394 V 16 w(STRUCT)h Fq(is)f(c)o(hanged,)h(accordingly)l(.)34 b(\()p Fe(End)20 b(of)189 2450 y(advic)n(e)c(to)g(implementors.)p Fq(\))75 2571 y Ff(3.9.2)49 b(Contiguous)17 b(Struct)e(T)l(yp)q(es)75 2704 y Fm(Curren)o(t)f(Status:)j Fl(F)m(ailed)c(6-7-10)f(at)i(Septem)o (b)q(er)g(meeting.)-32 46 y Fh(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 12 14 12 13 bop 75 -100 a Fq(12)947 b Fg(CHAPTER)15 b(3.)35 b(MISCELLANY)17 b(F)o(OR)e(1.2)166 45 y Fq(Data)j(structures)i(ma)o(y)f (con)o(tain)h(padding)g(holes,)i(added)e(for)f(alignmen)o(t.)34 b(The)20 b(de\014nition)h(of)75 102 y(an)f(MPI)g(datat)o(yp)q(e,)g (done)g(using)g(the)g Fj(MPI)p 865 102 14 2 v 16 w(TYPE)p 998 102 V 17 w(STRUCT)g Fq(constructor,)g(do)q(es)g(not)g(distinguish) 75 158 y(b)q(et)o(w)o(een)15 b(holes)g(that)f(con)o(tains)g(no)h (signi\014can)o(t)g(data,)f(and)h(are)f(there)h(only)g(for)f(alignmen)o (t)h(purp)q(oses,)75 214 y(and)k(b)q(et)o(w)o(een)g(holes)g(that)f(ma)o (y)f(con)o(tain)i(signi\014can)o(t)g(data)f(and)h(w)o(ere)f(purp)q (osefully)j(left)e(out)f(b)o(y)g(a)75 271 y(user)k(that)e(wishes)i(not) f(to)g(o)o(v)o(erwrite)g(some)g(of)g(the)g(information)h(in)g(a)f (structure.)38 b(As)22 b(a)f(result,)75 327 y(MPI)g(cannot)g(o)o(v)o (erwrite)f(these)h(holes,)i(this)f(prev)o(en)o(ts)e(the)i(ob)o(vious)f (optimization)h(of)e(shipping)j(a)75 384 y(data)13 b(structure)h(as)g (a)f(con)o(tiguous)h(blo)q(c)o(k)h(of)f(data,)f(padding)i(holes)f (included.)22 b(The)15 b(t)o(yp)q(e)f(constructor)75 440 y Fj(MPI)p 160 440 V 16 w(TYPE)p 293 440 V 17 w(CONTIGUOUS)p 598 440 V 18 w(STRUCT)d Fq(marks)f(an)o(y)g(suc)o(h)h(holes)g(as)g (\\don)o(t-care")e(lo)q(cations,)j(allo)o(wing)75 497 y(the)j(implemen)o(tation)i(to)d(o)o(v)o(erwrite)h(them.)75 648 y Fj(MPI)p 160 648 V 16 w(TYPE)p 293 648 V 17 w(CONTIGUOUS)p 598 648 V 18 w(STRUCT\(count,)j(a)o(rra)o(y)p 1041 648 V 14 w(of)p 1092 648 V 16 w(blo)q(cklengths,)h(a)o(rra)o(y)p 1468 648 V 14 w(of)p 1519 648 V 16 w(displacements,)g(a)o(r-)75 704 y(ra)o(y)p 136 704 V 15 w(of)p 188 704 V 16 w(t)o(yp)q(es,)e(newt)o (yp)q(e\))117 781 y Fl(IN)155 b Fj(count)482 b Fl(n)o(um)o(b)q(er)13 b(of)h(blo)q(c)o(ks)g(\(in)o(teger\))117 856 y(IN)155 b Fj(a)o(rra)o(y)p 416 856 V 15 w(of)p 468 856 V 16 w(blo)q(cklengths) 191 b Fl(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(eac)o(h)g(blo)q (c)o(k)g(\(arra)o(y)g(of)f(in)o(teger\))117 932 y(IN)155 b Fj(a)o(rra)o(y)p 416 932 V 15 w(of)p 468 932 V 16 w(displacements)164 b Fl(b)o(yte)14 b(displacemen)o(t)g(of)f(eac)o(h)h(blo)q(c)o(k)g (\(arra)o(y)g(of)f(in)o(teger\))117 1007 y(IN)155 b Fj(a)o(rra)o(y)p 416 1007 V 15 w(of)p 468 1007 V 16 w(t)o(yp)q(es)327 b Fl(t)o(yp)q(e)15 b(of)e(elemen)o(ts)h(in)f(eac)o(h)h(blo)q(c)o(k)g (\(arra)o(y)g(of)f(handle\))117 1082 y(OUT)108 b Fj(newt)o(yp)q(e)433 b Fl(new)15 b(datat)o(yp)q(e)f(\(handle\))166 1206 y Fq(The)c(seman)o(tics)h(of)e Fj(MPI)p 587 1206 V 16 w(TYPE)p 720 1206 V 17 w(CONTIGUOUS)p 1025 1206 V 18 w(STRUCT)i Fq(are)f(iden)o(tical)i(to)d Fj(MPI)p 1611 1206 V 16 w(TYPE)p 1744 1206 V 17 w(STRUCT)p Fq(,)75 1263 y(except)15 b(that,)e(if)h(a)g(receiv)o(e)h(call)g(uses)g(a)e(datat)o(yp)q(e)h (constructed)g(with)h(this)f(constructor,)f(then)i(the)f(in-)75 1319 y(ternal)f(holes)h(in)f(the)g(structure)g(can)g(b)q(e)g(o)o(v)o (erwritten.)18 b(The)13 b(same)g(rule)h(\(namely)l(,)f(that)f(holes)i (b)q(et)o(w)o(een)75 1376 y(comp)q(onen)o(ts)c(can)g(b)q(e)h(o)o(v)o (erwritten\))e(applies)j(also)e(to)g(datat)o(yp)q(es)f(constructed)h (using)h Fj(MPI)p 1635 1376 V 16 w(TYPE)p 1768 1376 V 17 w(SIMPLE)p 1944 1376 V 16 w(STRUCT)p Fq(.)166 1479 y Fm(Discussion:)29 b Fl(It)10 b(is)f(reasonable)g(to)g(assume)g(that)h (holes)f(in)g(la)o(y)o(outs)f(de\014ned)j(using)e Fd(MPI)p 1576 1479 13 2 v 14 w(TYPE)p 1698 1479 V 14 w(SIMPLE)p 1857 1479 V 15 w(STRUCT)75 1536 y Fl(con)o(tain)14 b(no)h(signi\014can) o(t)f(information,)e(since)k(this)f(datat)o(yp)q(e)g(constructor)h(is)f (exp)q(ected)i(to)e(b)q(e)g(used)h(for)f(sp)q(ec-)75 1592 y(ifying)f(the)h(la)o(y)o(out)f(of)h(structures.)24 b(Users)16 b(will)e(b)q(e)i(required)g(to)f(use)h Fd(MPI)p 1271 1592 V 14 w(TYPE)p 1393 1592 V 14 w(STRUCT)f Fl(when)g(holes)h (hold)75 1649 y(signi\014can)o(t)d(information.)189 1802 y Fe(A)n(dvic)n(e)e(to)i(users.)38 b Fq(Users)11 b(should)i(arrange)d (the)i(\014elds)g(in)g(a)f(C)h(struct)e(t)o(yp)q(e)i(in)g(descending)h (order)189 1859 y(of)i(their)i(size.)24 b(This)17 b(arrangemen)o(t)e (has)h(t)o(w)o(o)f(adv)m(an)o(tages:)21 b(\014rstly)c(it)f(ma)o(y)g (reduce)h(the)f(size)h(of)189 1915 y(the)i(struct)f(b)o(y)h (eliminating)i(padding)f(b)q(et)o(w)o(een)f(elemen)o(ts)h(with)f (di\013eren)o(t)g(alignmen)o(ts)h(\(this)189 1972 y(is)c(generally)i (true)e(for)g(C)g(programs\).)22 b(Secondly)l(,)c(it)e(ma)o(y)g (increase)h(the)g(n)o(um)o(b)q(er)f(of)g(cases)h(for)189 2028 y(whic)o(h)d(the)f(implemen)o(tation)h(can)g(p)q(erform)f(blo)q(c) o(k)g(copies)i(b)o(y)e(ensuring)h(that)e(the)h(\014rst)g(elemen)o(t)189 2085 y(of)19 b(the)g(structure)g(has)h(the)f(most)g(stringen)o(t)g (alignmen)o(t)h(requiremen)o(ts.)33 b(\()p Fe(End)19 b(of)i(advic)n(e)f(to)189 2141 y(users.)p Fq(\))189 2247 y Fe(A)n(dvic)n(e)15 b(to)i(implementors.)39 b Fq(An)16 b(implemen)o(tation)g(ma)o(y)f(ignore)g(the)h(constructor)189 2304 y Fj(MPI)p 274 2304 14 2 v 15 w(TYPE)p 406 2304 V 17 w(CONTIGUOUS)p 711 2304 V 18 w(STRUCT)p Fq(,)21 b(and)f(handle)i(it)e(exactly)h(as)f Fj(MPI)p 1539 2304 V 16 w(TYPE)p 1672 2304 V 17 w(STRUCT)p Fq(.)189 2360 y(Th)o(us,)f(there)g(is)h(\(almost\))d(no)i(o)o(v)o(erhead)g(in)g(supp) q(orting)h(this)f(constructor,)g(without)g(taking)189 2417 y(adv)m(an)o(tage)f(of)h(it.)32 b(One)20 b(p)q(ossible)h(w)o(a)o (y)d(of)h(taking)g(adv)m(an)o(tage)g(of)g(it,)h(is)g(to)e(adopt)h(a)g (canoni-)189 2473 y(cal)d(\\wire")g(represen)o(tation)g(for)g (structures.)22 b(Supp)q(ose)17 b(that)f(the)g(compiler)h(uses,)g(b)o (y)f(default,)189 2530 y(natural)g(alignmen)o(t)i(rules)f(for)f (structures:)23 b(eac)o(h)16 b(basic)i(comp)q(onen)o(t)e(is)i(aligned)g (at)e(a)g(natural)189 2586 y(b)q(oundary)l(.)k(Supp)q(ose)14 b(that)f(one)h(adopts)f(natural)h(alignmen)o(t)g(rules)h(for)e (messages)g(on)g(the)h(wire,)189 2642 y(in)19 b(a)f(homogeneous)g(en)o (vironmen)o(t:)27 b(messages)18 b(are)g(padded)h(so)f(that)g(eac)o(h)g (basic)h(elemen)o(t)g(is)189 2699 y(aligned)d(at)f(a)g(natural)g(b)q (oundary)l(.)20 b(Then)1967 46 y Fh(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 13 15 13 14 bop 75 -100 a Fg(3.9.)34 b(NEW)15 b(D)o(A)l(T)l(A)l(TYPES)1245 b Fq(13)189 45 y(\(1\))14 b(A)i(receiv)o(er)g(can)f(alw)o(a)o(ys)g (deco)q(de)h(an)g(incoming)h(message)d(and)i(retriev)o(e)g(correctly)f (the)h(ba-)189 102 y(sic)k(comp)q(onen)o(ts.)31 b(This)20 b(b)q(ecause)g(the)g(basic)g(datat)o(yp)q(e)e(of)h(the)g(elemen)o(t)h (indicates)h(at)e(what)189 158 y(b)q(oundary)c(it)h(is)f(aligned.)189 214 y(\(2\))f(A)h(sender)g(can)h(blo)q(c)o(k)f(cop)o(y)g(a)g(structure) g(on)o(to)f(the)h(\\wire")g(\(most)f(lik)o(ely)l(,)j(on)o(to)d(a)g (message)189 271 y(bu\013er\),)19 b(when)g(the)g(structure)g(is)g (naturally)h(aligned,)h Fe(r)n(elative)e(to)h(the)g(start)g(of)f(the)h (sending)189 327 y(bu\013er)p Fq(.)26 b(This)18 b(optimization)g(do)q (es)g(not)e(require)j(that)d(the)h(sending)i(datat)o(yp)q(e)e(b)q(e)h (built)g(using)189 384 y Fj(MPI)p 274 384 14 2 v 15 w(TYPE)p 406 384 V 17 w(CONTIGUOUS)p 711 384 V 18 w(STRUCT)p Fq(:)c(The)g(MPI)f (library)i(can)e(scan)h(datat)o(yp)q(es,)f(when)h(they)189 440 y(are)h(committed,)f(and)i(recognize)g(when)g(a)e(datat)o(yp)q(e)h (\(or)f(a)h(datat)o(yp)q(e)g(fragmen)o(t\))f(is)h(naturally)189 497 y(aligned.)24 b(Blo)q(c)o(k)17 b(cop)o(ying)g(can)g(b)q(e)g(used)g (ev)o(en)f(if)h(the)g(holes)g(in)g(the)f(datat)o(yp)q(e)g(con)o(tain)g (signi\014-)189 553 y(can)o(t)e(information.)189 610 y(\(3\))h(A)h(receiv)o(er)h(can)f(blo)q(c)o(k)h(cop)o(y)f(an)g (incoming)i(message)d(on)o(to)h(the)g(receiv)o(e)h(bu\013er)f(when)h (the)189 666 y(receiving)e(structure)f(is)h(naturally)g(aligned)h Fe(r)n(elative)f(to)g(the)h(start)g(of)f(the)h(r)n(e)n(c)n(eiving)d (bu\013er)i Fq(and)189 723 y(holes)10 b(can)h(b)q(e)g(ignored.)18 b(I.e.,)11 b(when)g(the)f(receiving)i(datat)o(yp)q(e)d(is)i(built)g (using)g Fj(MPI)p 1597 723 V 16 w(TYPE)p 1730 723 V 17 w(CONTIGUOUS)p 2035 723 V 18 w(STRUCT)189 779 y Fq(or)i Fj(MPI)p 328 779 V 16 w(TYPE)p 461 779 V 17 w(SIMPLE)p 637 779 V 16 w(STRUCT)h Fq(and)g(is)h(naturally)f(aligned,)i(relativ)o (e)e(to)g(its)g(\014rst)f(p)q(osition.)189 853 y(The)k(same)h(wire)g (represen)o(tation)f(ma)o(y)g(b)q(e)h(used)g(in)h(a)e(heterogenous)h (en)o(vironmen)o(t,)g(in)g(cases)189 909 y(where)h(data)e(con)o(v)o (ersion)i(is)g(done)g(at)f(the)h(destination;)i(the)e(destination)g (has)g(to)f(kno)o(w,)g(not)189 966 y(only)c(the)h(basic)g(datat)o(yp)q (es)e(used)i(b)o(y)f(the)h(source,)f(but)g(also)h(the)f(alignmen)o(t)h (rules)g(used)g(b)o(y)f(the)189 1022 y(source.)189 1096 y(Example:)34 b(Consider)23 b(a)f(C)g(structure)g Fk(struct)h({char)g (a[2];)g(float)g(b[2];)h(double)f(c})p Fq(;)189 1153 y(assume)16 b(that)g(\015oats)g(are)g(4)h(b)o(ytes)f(and)h(doubles)h (are)e(8)g(b)o(ytes,)h(and)g(structures)f(are)g(naturally)189 1209 y(aligned.)36 b(The)20 b(memory)f(la)o(y)o(out)h(is)44 b Fk([c][c]xx[ffff][ffff]xxxx[dddd)o(dddd])p Fq(,)18 b(and)i(the)189 1266 y(structure)15 b(is)g(aligned)i(to)d(an)i(8)e(b)o (yte)h(b)q(oundary)l(.)189 1340 y(Case)c(1:)18 b(The)12 b(en)o(tire)g(structure)f(is)h(comm)o(unicated.)20 b(Sender)12 b(constructs)f(a)h(datat)o(yp)q(e)f(for)g(struc-)189 1396 y(ture)f(using)h Fj(MPI)p 479 1396 V 15 w(TYPE)p 611 1396 V 17 w(STRUCT,)g(MPI)p 910 1396 V 15 w(TYPE)p 1042 1396 V 17 w(SIMPLE)p 1218 1396 V 16 w(STRUCT)g Fq(or)f Fj(MPI)p 1554 1396 V 15 w(TYPE)p 1686 1396 V 17 w(CONTIGUOUS)p 1991 1396 V 18 w(STRUCT)p Fq(;)189 1452 y(receiv)o(er)16 b(constructs)g(a)g(datat)o(yp)q(e)g(for)f(this)i(structure,)e(using)i Fj(MPI)p 1379 1452 V 16 w(CONTIGUOUS)p 1683 1452 V 18 w(STRUCT)189 1509 y Fq(or)12 b Fj(MPI)p 327 1509 V 16 w(TYPE)p 460 1509 V 16 w(SIMPLE)p 635 1509 V 16 w(STRUCT)p Fq(.)h(Then)g(the)g(la)o(y)o(outs)f(in)h(the)g(sender)g(memory)l(,)g (on)f(the)h(wire,)189 1565 y(and)i(in)h(the)f(receiv)o(er)h(memory)f (are)g(iden)o(tical,)h(and)g(data)e(can)i(b)q(e)g(simply)g(copied.)189 1639 y(Case)10 b(2:)17 b(comm)o(unication)12 b(in)o(v)o(olv)o(es)f (only)h(the)f(second)g(of)f(the)h(t)o(w)o(o)f(c)o(haracters,)g(and)h (the)g(remain-)189 1696 y(der)f(of)g(the)g(structure.)18 b(Receiv)o(er)11 b(constructs)f(the)g(datat)o(yp)q(e)g(using)h Fj(MPI)p 1441 1696 V 16 w(CONTIGUOUS)p 1745 1696 V 18 w(STRUCT)p Fq(.)189 1752 y(The)19 b(memory)f(la)o(y)o(out)g(is)h Fk([c]xx[ffff][ffff]xxxx[dddddd)o(dd])p Fq(,)d(and)j(the)g(wire)g(la)o (y)o(out)f(is)189 1809 y Fk([c]xxx[ffff][ffff]xxxx[)o(dddddddd)o(])p Fq(.)e(The)c(datat)o(yp)q(e)f(is)h(not)g(naturally)g(aligned,)h(so)f (that)189 1865 y(blo)q(c)o(k)k(cop)o(ying)g(cannot)f(b)q(e)h(used)g(at) f(the)h(sender)g(or)f(the)g(receiv)o(er.)22 b(Ho)o(w)o(ev)o(er,)14 b(a)h(smart)f(imple-)189 1922 y(men)o(tation)h(ma)o(y)g(realize)i(that) e(the)h(fragmen)o(t)f(starting)g(from)g(the)h(\015oat)f(is)i(naturally) f(aligned,)189 1978 y(relativ)o(e)e(to)e(the)i(start)e(of)h(the)g (bu\013er,)h(so)f(that)f(blo)q(c)o(k)j(cop)o(ying)e(can)h(b)q(e)g(used) g(for)f(this)g(fragmen)o(t.)189 2052 y(Case)f(3:)19 b(comm)o(unication) 14 b(in)o(v)o(olv)o(es)g(only)f(the)g(t)o(w)o(o)f(\015oats)h(and)g(the) g(double.)21 b(The)13 b(memory)g(la)o(y-)189 2108 y(out)c(is)i Fk([ffff][ffff]xxxx[dddddddd])o Fq(.)16 b(The)10 b(la)o(y)o(out)g(on)g (the)g(wire)g(is)h Fk([ffff][ffff][dddddddd])p Fq(.)189 2165 y(The)j(datat)o(yp)q(e)f(is)i(not)f(naturally)g(aligned,)i (relativ)o(e)e(to)g(its)g(start,)e(so)i(that)f(blo)q(c)o(k)i(cop)o (ying)g(can-)189 2221 y(not)f(b)q(e)i(used.)21 b(\()p Fe(End)15 b(of)i(advic)n(e)f(to)g(implementors.)p Fq(\))166 2360 y Fm(Alternativ)o(es)o(:)134 2432 y Fl(1.)22 b(No)14 b(new)h(function)f Fd(MPI)p 581 2432 13 2 v 14 w(TYPE)p 703 2432 V 15 w(CONTIGUOUS)p 985 2432 V 12 w(STRUCT)p Fl(:)g(instead,)g(holes)h(can)g(b)q(e)g(o)o(v)o(erwritten)g(only)189 2482 y(for)f(datat)o(yp)q(es)i(built)e(with)g Fd(MPI)p 717 2482 V 15 w(TYPE)p 840 2482 V 14 w(SIMPLE)p 999 2482 V 15 w(STRUCT)p Fl(.)f(The)j(argumen)o(t)e(is)g(that,)h(at)g(least)g (in)f(the)189 2532 y(scenario)f(I)g(presen)o(ted,)i(hole)d(o)o(v)o (erwriting)h(is)g(tak)o(en)g(adv)n(an)o(tage)f(of)g(only)g(for)h (structures)i(with)e(a)f(natural)189 2581 y(alignmen)o(t.)j(But,)f (then,)g(their)h(la)o(y)o(out)d(can)i(b)q(e)h(sp)q(eci\014ed)g(using)f Fd(MPI)p 1308 2581 V 14 w(TYPE)p 1430 2581 V 15 w(SIMPLE)p 1590 2581 V 14 w(STRUCT)p Fl(.)134 2654 y(2.)22 b(Both)d(sender)h(and)f (receiv)o(er)h(are)g(required)f(to)g(use)h Fd(MPI)p 1126 2654 V 14 w(TYPE)p 1248 2654 V 14 w(CONTIGUOUS)p 1529 2654 V 13 w(STRUCT)p Fl(,)e(and)g(the)189 2704 y(datat)o(yp)q(es)c(on)f (b)q(oth)h(ends)g(ha)o(v)o(e)g(to)f(b)q(e)h(structurally)g(equiv)n (alen)o(t,)f(i.e.,)f(de\014ned)j(b)o(y)e(the)h(same)f(sequence)-32 46 y Fh(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 14 16 14 15 bop 75 -100 a Fq(14)947 b Fg(CHAPTER)15 b(3.)35 b(MISCELLANY)17 b(F)o(OR)e(1.2)189 45 y Fl(of)e(de\014nitions.)18 b(There)d(are)f(sev)o(eral)h(p)q(ossible)f(v)n(arian)o(ts)f(for)g(suc)o (h)i(prop)q(osal)189 107 y(\(a\))20 b(This)g(construct)i(can)e(b)q(e)h (only)e(used)i(in)f(a)g(homogeneous)f(en)o(vironmen)o(t.)36 b(But,)22 b(then,)g(w)o(e)e(lose)189 157 y(p)q(ortabilit)o(y)11 b(to)h(heterogeneous)j(en)o(vironmen)o(t,)c(and)i(w)o(e)g(ac)o(hiev)o (e)f(little)g(that)h(could)f(not)h(b)q(e)g(ac)o(hiev)o(ed)g(b)o(y)189 207 y(using)g Fd(MPI)p 374 207 13 2 v 15 w(BYTE)p Fl(.)189 269 y(\(b\))18 b(This)g(construct)i(can)e(b)q(e)g(used)h(p)q(ortably)m (,)f(with)g(enhanced)h(p)q(erformance)f(obtained)g(in)f(a)h(homo-)189 319 y(geneous)i(en)o(vironmen)o(t.)36 b(E.g.,)20 b(a)g(homogeneous)f (implem)o(en)o(tation)e(will)i(use)h(blo)q(c)o(k)g(cop)o(ying,)g(while) 189 369 y(a)d(heterogeneous)j(implemen)o(tatio)o(n)15 b(will)i(mo)o(v)o(e)f(elemen)o(t)h(b)o(y)h(elemen)o(t)f(\(at)h(least,)g (for)g(heterogeneous)189 419 y(comm)o(unicators\).)j(This)16 b(assumes)g(that,)g(in)f(a)h(homogeneous)e(en)o(vironmen)o(t,)h (di\013eren)o(t)i(pro)q(cesses)i(are)189 469 y(compiled)8 b(with)i(the)g(same)f(data)h(alignmen)o(t)d(options.)17 b(Also,)10 b(this)g(runs)g(coun)o(ter)h(the)g(datat)o(yp)q(e)f(matc)o (hing)189 518 y(rules)k(in)g(MPI,)f(where)i(only)e(signature)i(matc)o (hing)d(is)h(required.)189 581 y(\(c\))d Fd(MPI)p 326 581 V 14 w(TYPE)p 448 581 V 14 w(CONTIGUOUS)p 729 581 V 13 w(STRUCT)f Fl(can)g(b)q(e)i(matc)o(hed)d(with)h(other)i(datat)o (yp)q(es,)f(as)g(long)e(as)i(signa-)189 630 y(tures)g(matc)o(h;)f (impro)o(v)o(ed)f(p)q(erformance)h(is)g(ac)o(hiev)o(ed)h(when)f(b)q (oth)h(ends)g(use)g Fd(MPI)p 1463 630 V 14 w(TYPE)p 1585 630 V 15 w(CONTIGUOUS)p 1867 630 V 12 w(STRUCT)p Fl(.)189 680 y(But,)i(then,)h(the)g(wire)g(proto)q(col)f(m)o(ust)f(b)q(e)i(so)f (that)h(individual)d(elemen)o(ts)i(can)h(b)q(e)f(extracted,)i(in)e (case)h(the)189 730 y(receiv)o(er)k(do)q(es)f(not)f(use)i Fd(MPI)p 664 730 V 14 w(TYPE)p 786 730 V 14 w(CONTIGUOUS)p 1067 730 V 13 w(STRUCT)p Fl(.)d(In)i(this)f(case,)i(asking)e(the)h (sender)h(to)189 780 y(use)g Fd(MPI)p 340 780 V 14 w(TYPE)p 462 780 V 15 w(CONTIGUOUS)p 744 780 V 12 w(STRUCT)f Fl(do)q(es)h(not)g (pro)o(vide)f(additional)f(function)h(o)o(v)o(er)h(the)g(main)189 830 y(prop)q(osal.)g(The)d(di\013erence)h(is)f(only)e(whether)j(the)f (use)g(asserts)i(that)d(the)h(sending)g(datat)o(yp)q(e)f(has)h(a)f (nat-)189 879 y(ural)i(la)o(y)o(out)g(\(using)h Fd(MPI)p 608 879 V 14 w(TYPE)p 730 879 V 14 w(CONTIGUOUS)p 1011 879 V 13 w(STRUCT)p Fl(\),)f(or)h(whether)h(MPI)f(disco)o(v)o(ers)h (that)f(fact)189 929 y(b)o(y)d(itself.)75 1123 y Ff(3.9.3)49 b(Convenient)17 b(F)o(o)o(rm)d(of)j(MPI)p 731 1123 15 2 v 18 w(T)l(yp)q(e)p 848 1123 V 18 w(indexed)p 1018 1123 V 18 w(blo)q(ck)75 1256 y Fm(Curren)o(t)d(Status:)j Fl(P)o(assed)e(once.)166 1360 y Fq(This)d(prop)q(osal)g(is)g(to)f(allo) o(w)h(constan)o(t)f(blo)q(c)o(ksize)i(and)f(arbitrary)f(displacemen)o (ts)i(in)f(the)g(in)o(terface)75 1416 y(to)e Fj(MPI)p 211 1416 14 2 v 15 w(TYPE)p 343 1416 V 17 w(INDEXED)p Fq(.)g(The)g(routine)h(no)o(w)f(tak)o(es)f Fj(a)o(rra)o(y)p 1114 1416 V 14 w(of)p 1165 1416 V 16 w(blo)q(cklengths)k Fq(and)d Fj(a)o(rra)o(y)p 1605 1416 V 15 w(of)p 1657 1416 V 16 w(displacements)75 1473 y Fq(as)17 b(argumen)o(ts.)27 b(There)18 b(are)f(man)o(y)h(co)q(des)g(using)g(indirect)i(addressing)e (arising)g(from)f(unstructured)75 1529 y(grids)f(where)h(the)f(blo)q(c) o(ksize)i(is)e(alw)o(a)o(ys)f(1)h(\(gather/scatter\).)k(W)l(e)c(prop)q (ose)g(this)h(in)o(terface,)f(allo)o(wing)75 1586 y(for)f(constan)o(t)f (blo)q(c)o(ksize)j(and)e(arbitrary)g(displacemen)o(ts.)75 1737 y Fj(MPI)p 160 1737 V 16 w(TYPE)p 293 1737 V 17 w(INDEXED)p 505 1737 V 16 w(BLOCK\()g(count,)i(blo)q(cklength,)h(a)o (rra)o(y)p 1180 1737 V 14 w(of)p 1231 1737 V 16 w(displacements,)g (oldt)o(yp)q(e,)f(newt)o(yp)q(e\))117 1870 y Fl(IN)155 b Fj(count)482 b Fl(length)14 b(of)f(arra)o(y)h(of)f(displacemen)o(ts) 117 1945 y(IN)155 b Fj(blo)q(cklength)371 b Fl(size)15 b(of)e(blo)q(c)o(k)117 2021 y(IN)155 b Fj(a)o(rra)o(y)p 416 2021 V 15 w(of)p 468 2021 V 16 w(displacements)164 b Fl(arra)o(y)14 b(of)f(displacemen)o(ts)117 2096 y(IN)155 b Fj(oldt)o(yp)q(e)450 b Fl(old)13 b(datat)o(yp)q(e)117 2171 y(OUT)108 b Fj(newt)o(yp)q(e)433 b Fl(new)15 b(datat)o(yp)q(e)75 2295 y Fk(int)23 b(MPI)p 245 2295 15 2 v 17 w(Type)p 358 2295 V 17 w(Indexed)p 543 2295 V 16 w(Block\()g(int)h(count,)f(int) g(blocklength,)393 2352 y(int*)g(array)p 635 2352 V 17 w(of)p 700 2352 V 17 w(displacements,)f(MPI)p 1147 2352 V 17 w(Datatype)g(oldtype,)393 2408 y(MPI)p 468 2408 V 17 w(Datatype*)h(newtype)f(\))75 2495 y(MPI)p 150 2495 V 17 w(TYPE)p 263 2495 V 16 w(INDEXED)p 447 2495 V 17 w(BLOCK\(COUNT,)g(BLOCKLENGTH,)g(ARRAY)p 1204 2495 V 17 w(OF)p 1269 2495 V 16 w(DISPLACEMENTS,)g(OLDTYPE,)393 2551 y(NEWTYPE,)h(IERROR\))170 2608 y(INTEGER)g(COUNT,)g(INTEGER)g (BLOCKLENGTH,)g(ARRAY)p 1153 2608 V 16 w(OF)p 1217 2608 V 17 w(DISPLACEMENT\(*\),)f(OLDTYPE,)170 2664 y(NEWTYPE,)h(IERROR)-117 2700 y Fi(>)15 b Fh(\(Oct\))1967 46 y(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 15 17 15 16 bop 75 -100 a Fg(3.10.)34 b(MINOR)16 b(CORRECTIONS)1109 b Fq(15)75 45 y Fk(int)23 b(MPI::Datatype::Indexed)p 701 45 15 2 v 15 w(Block\()g(int)g(count,)g(int)h(blocklength,)393 102 y(const)f(int)h(array)p 755 102 V 16 w(of)p 819 102 V 17 w(displacements[],)393 158 y(const)f(MPI::Datatype&)f(oldtype)h (\))1875 198 y Fi(?)16 b Fh(\(Oct\))75 331 y Fn(3.10)59 b(Mino)n(r)21 b(Co)n(rrections)75 480 y Fm(Curren)o(t)14 b(Status:)j Fl(P)o(assed)e(once.)166 584 y Fq(The)i(follo)o(wing)i (corrections)e(to)g Fj(MPI-1.1)f Fq(are)h(\(all)h(page)f(and)g(line)i (n)o(um)o(b)q(ers)f(are)f(for)g(the)g(June)75 640 y(12,)d(1995)g(v)o (ersion)i(without)f(c)o(hangebars\):)143 746 y Fi(\017)23 b Fq(P)o(age)14 b(19,)g(lines)j(1-2)e(reads)189 803 y(for)f(\(64)g (bit\))i(C)f(in)o(tegers)g(declared)i(to)d(b)q(e)i(of)f(t)o(yp)q(e)g Fj(longlong)g(int)189 859 y Fq(but)g(should)h(read)189 916 y(for)e(\(64)g(bit\))i(C)f(in)o(tegers)g(declared)i(to)d(b)q(e)i (of)f(t)o(yp)q(e)g Fj(long)g(long)g(int)189 1088 y Fm(Discussion)o(:) 189 1156 y Fl(Just)f(to)g(mak)o(e)f(it)g(correct)j(C)d(so)h(it)g(is)g (long)f(long.)1875 1212 y Fi(>)j Fh(\(Oct\))143 1297 y Fi(\017)23 b Fq(P)o(age)14 b(41,)g(lines)j(16-18)d(reads)189 1354 y(A)19 b Fb(empt)o(y)g Fq(status)g(is)h(a)f(status)g(whic)o(h)i (is)f(set)f(to)g(return)h Fj(tag)g(=)g(MPI)p 1460 1354 14 2 v 16 w(ANY)p 1568 1354 V 17 w(T)l(A)o(G)p Fq(,)f Fj(source)i(=)189 1410 y(MPI)p 274 1410 V 15 w(ANY)p 381 1410 V 18 w(SOURCE)p Fq(,)11 b(and)g(is)h(also)e(in)o(ternally)j (con\014gured)e(so)g(that)f(calls)h(to)g Fj(MPI)p 1601 1410 V 15 w(GET)p 1704 1410 V 17 w(COUNT)189 1467 y Fq(and)k Fj(MPI)p 362 1467 V 16 w(GET)p 466 1467 V 17 w(ELEMENTS)g Fq(return)g Fj(count)i(=)e(0)p Fq(.)189 1523 y(but)g(should)h(read)189 1580 y(A)j Fb(empt)o(y)g Fq(status)g(is)h(a)f(status)g(whic)o(h)i(is)f (set)f(to)g(return)h Fj(tag)g(=)g(MPI)p 1460 1580 V 16 w(ANY)p 1568 1580 V 17 w(T)l(A)o(G)p Fq(,)f Fj(source)i(=)189 1636 y(MPI)p 274 1636 V 15 w(ANY)p 381 1636 V 18 w(SOURCE)p Fq(,)c Fj(erro)o(r)e(=)i(MPI)p 842 1636 V 16 w(SUCCESS)p Fq(,)h(and)f(is)g(also)g(in)o(ternally)h(con\014gured)f(so)g(that)189 1693 y(calls)f(to)e Fj(MPI)p 430 1693 V 16 w(GET)p 534 1693 V 17 w(COUNT)i Fq(and)f Fj(MPI)p 893 1693 V 16 w(GET)p 997 1693 V 17 w(ELEMENTS)g Fq(return)g Fj(count)i(=)e(0)p Fq(.)189 1865 y Fm(Discussion)o(:)189 1933 y Fl(Filling)d(in)h(the)h (error)h(\014eld)f(got)g(forgotten.)1875 1989 y Fi(?)i Fh(\(Oct\))143 2074 y Fi(\017)23 b Fq(P)o(age)14 b(71,)g(line)j(10)e (reads)189 2131 y(and)g(do)g(not)g(a\013ect)f(the)i(the)f(con)o(ten)o (t)g(of)f(a)h(message)189 2187 y(but)g(should)h(read)189 2244 y(and)f(do)g(not)g(a\013ect)f(the)i(con)o(ten)o(t)e(of)h(a)g (message)189 2416 y Fm(Discussion)o(:)189 2484 y Fl(This)e(is)h(a)g (trivial)e(t)o(yp)q(o)i(of)f(ha)o(ving)g(the)h(2)g(times.)143 2626 y Fi(\017)23 b Fq(P)o(age)14 b(74,)g(lines)j(39-45)d(read)189 2682 y(A)i(datat)o(yp)q(e)f(ma)o(y)g(sp)q(ecify)i(o)o(v)o(erlapping)g (en)o(tries.)22 b(The)16 b(use)g(of)g(suc)o(h)g(a)f(datat)o(yp)q(e)h (in)g(a)g(receiv)o(e)-32 46 y Fh(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 16 18 16 17 bop 75 -100 a Fq(16)947 b Fg(CHAPTER)15 b(3.)35 b(MISCELLANY)17 b(F)o(OR)e(1.2)189 45 y Fq(op)q(eration)f(is)h (erroneous.)k(\(This)14 b(is)h(erroneous)f(ev)o(en)g(if)h(the)f(actual) g(message)g(receiv)o(ed)h(is)g(short)189 102 y(enough)g(not)g(to)g (write)g(an)o(y)g(en)o(try)g(more)f(than)h(once.\))189 174 y(A)i(datat)o(yp)q(e)g(ma)o(y)g(sp)q(ecify)i(o)o(v)o(erlapping)g (en)o(tries.)27 b(If)18 b(suc)o(h)g(a)g(datat)o(yp)q(e)f(is)h(used)g (in)h(a)e(receiv)o(e)189 231 y(op)q(eration,)d(that)h(is,)g(if)g(some)g (part)f(of)h(the)f(receiv)o(e)i(bu\013er)f(is)h(written)f(more)f(than)h (once)g(b)o(y)g(the)189 287 y(receiv)o(e)h(op)q(eration,)f(then)g(the)h (call)g(is)g(erroneous.)189 359 y(The)i(\014rst)f(part)g(w)o(as)h(an)f Fj(MPI-1.1)g Fq(addition.)29 b(The)18 b(second)g(part)f(o)o(v)o(erlaps) h(with)g(it.)28 b(The)18 b(old)189 416 y(text)c(will)j(b)q(e)f(remo)o (v)o(ed)f(so)g(it)g(no)o(w)g(reads)189 472 y(A)h(datat)o(yp)q(e)f(ma)o (y)g(sp)q(ecify)i(o)o(v)o(erlapping)g(en)o(tries.)22 b(The)16 b(use)g(of)g(suc)o(h)g(a)f(datat)o(yp)q(e)h(in)g(a)g(receiv)o (e)189 529 y(op)q(eration)e(is)h(erroneous.)k(\(This)14 b(is)h(erroneous)f(ev)o(en)g(if)h(the)f(actual)g(message)g(receiv)o(ed) h(is)g(short)189 585 y(enough)g(not)g(to)g(write)g(an)o(y)g(en)o(try)g (more)f(than)h(once.\))189 698 y Fm(Discussion)o(:)189 764 y Fl(The)f(in)o(ten)o(t)h(is)f(to)g(remo)o(v)o(e)f(duplicate)h (text)h(whic)o(h)f(seems)h(to)f(ha)o(v)o(e)g(b)q(een)h(left.)k(No)14 b(c)o(hange)h(in)f(meaning)189 814 y(is)f(in)o(tended.)143 950 y Fi(\017)23 b Fq(P)o(age)14 b(90,)g(line)j(3)e(reads)189 1006 y(MPI)p 281 1006 14 2 v 16 w(P)o(ac)o(k)p 393 1006 V 16 w(size\(coun)o(t,)g(MPI)p 724 1006 V 16 w(CHAR,)h(&k2\);)189 1063 y(but)f(should)h(read)189 1119 y(MPI)p 281 1119 V 16 w(P)o(ac)o(k)p 393 1119 V 16 w(size\(coun)o(t,)f(MPI)p 724 1119 V 16 w(CHAR,)h(comm,)e(&k2\);)189 1289 y Fm(Discussion)o(:)189 1355 y Fl(This)f(is)h(a)g(minor)e(\014x)i(to)f(an)h(example)f(for)g(a)h (missing)e(argumen)o(t.)143 1490 y Fi(\017)23 b Fq(P)o(age)14 b(90,)g(line)j(10)e(reads)189 1547 y(MPI)p 281 1547 V 16 w(P)o(ac)o(k\(c)o(hr,)f(coun)o(t,)g(MPI)p 726 1547 V 17 w(CHAR,)h(&lbuf,)h(k,)f(&p)q(osition,)h(comm\);)189 1603 y(but)f(should)h(read)189 1660 y(MPI)p 281 1660 V 16 w(P)o(ac)o(k\(c)o(hr,)e(coun)o(t,)g(MPI)p 726 1660 V 17 w(CHAR,)h(lbuf,)h(k,)f(&p)q(osition,)h(comm\);)189 1829 y Fm(Discussion)o(:)189 1895 y Fl(This)e(is)h(a)f(minor)f(\014x)i (to)g(an)f(example)g(for)g(an)g(argumen)o(t)g(that)h(is)g(declared)g (to)g(b)q(e)g(a)g(p)q(oin)o(ter)g(and)f(then)189 1945 y(passed)h(with)e(&.)143 2081 y Fi(\017)23 b Fq(P)o(age)14 b(109,)g(lines)j(26/27)d(and)h(page)g(110,)f(lines)j(28/29)d(reads)189 2137 y(The)h Fk(j)p Fq(th)g(blo)q(c)o(k)g(of)g(data)f(sen)o(t)h(from)f (eac)o(h)h(pro)q(cess)g(is)h(receiv)o(ed)g(b)o(y)f(ev)o(ery)f(pro)q (cess)i(and)f(placed)189 2194 y(in)h(the)f Fk(j)p Fq(th)g(blo)q(c)o(k)h (of)f(the)g(bu\013er)g Fj(recvbuf)p Fq(.)189 2250 y(but)g(should)h (read)189 2307 y(The)g(blo)q(c)o(k)i(of)e(data)g(sen)o(t)g(from)g(the)g Fk(j)p Fq(th)h(pro)q(cess)g(is)g(receiv)o(ed)g(b)o(y)g(ev)o(ery)f(pro)q (cess)h(and)g(placed)189 2363 y(in)f(the)f Fk(j)p Fq(th)g(blo)q(c)o(k)h (of)f(the)g(bu\013er)g Fj(recvbuf)p Fq(.)189 2533 y Fm(Discussion)o(:) 189 2598 y Fl(This)e(is)g(related)h(to)f(a)f(subtle)i(p)q(oin)o(t)f(in) g(usage.)18 b(The)13 b(sendbuf)h(is)f(a)g(single)g(item)f(whereas)i (the)g(recvbuf)g(is)189 2648 y(an)f(\\arra)o(y")g(of)h(items.)1967 46 y Fh(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 17 19 17 18 bop 75 -100 a Fg(3.10.)34 b(MINOR)16 b(CORRECTIONS)1109 b Fq(17)143 45 y Fi(\017)23 b Fq(P)o(age)14 b(117,)g(lines)j(22/23)d (reads)189 102 y(MPI)h(pro)o(vides)g(sev)o(en)h(suc)o(h)f(prede\014ned) i(datat)o(yp)q(es.)189 158 y(but)e(should)h(read)189 214 y(MPI)f(pro)o(vides)g(nine)i(suc)o(h)e(prede\014ned)i(datat)o(yp)q (es.)189 442 y Fm(Discussion)o(:)189 509 y Fl(This)c(is)h(an)g(ob)o (vious)f(mistak)o(e.)k(There)e(are)f(nine)g(listed)g(on)f(line)h (28-40.)143 647 y Fi(\017)23 b Fq(P)o(age)14 b(122,)g(lines)j(35-36)d (read)204 704 y Fj(MPI)p 289 704 14 2 v 16 w(OP)p 367 704 V 16 w(FREE\()i(op\))230 893 y Fl(IN)156 b Fj(op)541 b Fl(op)q(eration)14 b(\(handle\))189 1035 y Fq(but)h(should)h(read)204 1091 y Fj(MPI)p 289 1091 V 16 w(OP)p 367 1091 V 16 w(FREE\()g(op\))230 1280 y Fl(INOUT)63 b Fj(op)541 b Fl(op)q(eration)14 b(\(handle\))189 1463 y Fm(Discussion)o(:)189 1530 y Fl(This)j(app)q(ears)g(to)g(ha)o(v) o(e)g(b)q(een)h(a)f(mistak)o(e.)26 b(This)17 b(function)g(frees)h(the)g (function)e(p)q(oin)o(ter)i(and)f(sets)h(its)189 1580 y(v)n(alue)10 b(to)h(MPI)p 424 1580 13 2 v 15 w(OP)p 499 1580 V 15 w(NULL.)g(Th)o(us,)g(it)g(is)g(b)q(oth)g(input)g(and)f (output)i(as)f(are)g(the)g(free)h(routines)g(for)e(Groups,)189 1629 y(Comms,)g(etc.)143 1768 y Fi(\017)23 b Fq(P)o(age)14 b(125,)g(line)j(1)e(reads)189 1824 y(CALL)h(MPI)p 420 1824 14 2 v 16 w(ALLREDUCE\(sum,)f(c,)g(n,)g(MPI)p 1040 1824 V 17 w(REAL,)g(MPI)p 1300 1824 V 17 w(SUM,)g(0,)f(comm,)g(ierr\)) 189 1881 y(but)h(should)h(read)189 1937 y(CALL)g(MPI)p 420 1937 V 16 w(ALLREDUCE\(sum,)f(c,)g(n,)g(MPI)p 1040 1937 V 17 w(REAL,)g(MPI)p 1300 1937 V 17 w(SUM,)g(comm,)f(ierr\))189 2108 y Fm(Discussion)o(:)189 2175 y Fl(The)g(extra)h(argumen)o(t)e(app) q(ears)j(to)e(b)q(e)h(left)f(o)o(v)o(er)g(from)f(the)i(previous)f (example)f(with)h(MPI)p 1670 2175 13 2 v 16 w(REDUCE.)189 2225 y(This)f(is)h(a)g(mistak)o(e.)143 2363 y Fi(\017)23 b Fq(P)o(age)14 b(194,)g(line)j(48)e(reads)189 2420 y Fj(MPI)p 274 2420 14 2 v 15 w(ERRHANDLER)p 582 2420 V 18 w(CREA)l(TE\(FUNCTION,)h(HANDLER,)g(IERROR\))189 2476 y Fq(but)f(should)h(read)189 2533 y Fj(MPI)p 274 2533 V 15 w(ERRHANDLER)p 582 2533 V 18 w(CREA)l(TE\(FUNCTION,)g(ERRHANDLER,) h(IERROR\))189 2704 y Fm(Discussion)o(:)-32 46 y Fh(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 18 20 18 19 bop 75 -100 a Fq(18)947 b Fg(CHAPTER)15 b(3.)35 b(MISCELLANY)17 b(F)o(OR)e(1.2)189 45 y Fl(This)e(is)h(a)g(small)d(t)o (yp)q(o)j(to)g(matc)o(h)f(all)f(the)j(other)f(names)f(for)h(this)g (argumen)o(t.)143 186 y Fi(\017)23 b Fq(P)o(age)14 b(196,)g(lines)j (1-2)e(reads)204 243 y Fj(MPI)p 289 243 14 2 v 16 w(ERRHANDLER)p 598 243 V 17 w(FREE\()h(errhandler)f(\))230 433 y Fl(IN)156 b Fj(errhandler)397 b Fl(MPI)14 b(error)h(handler)f(\(handle\))189 576 y Fq(but)h(should)h(read)204 633 y Fj(MPI)p 289 633 V 16 w(ERRHANDLER)p 598 633 V 17 w(FREE\()g(errhandler)f(\))230 823 y Fl(INOUT)63 b Fj(errhandler)397 b Fl(MPI)14 b(error)h(handler)f (\(handle\))189 1007 y Fm(Discussion)o(:)189 1075 y Fl(This)c(app)q (ears)h(to)f(ha)o(v)o(e)h(b)q(een)g(a)f(mistak)o(e.)16 b(This)10 b(function)g(frees)i(the)f(function)f(p)q(oin)o(ter)g(and)h (sets)g(its)g(v)n(alue)189 1125 y(to)h(MPI)p 321 1125 13 2 v 15 w(ERRHANDLER)p 636 1125 V 14 w(NULL.)g(Th)o(us,)h(it)f(is)g (b)q(oth)h(input)f(and)g(output)h(as)f(are)h(the)g(free)g(routines)g (for)189 1175 y(Groups,)g(Comms,)e(etc.)143 1316 y Fi(\017)23 b Fq(P)o(age)14 b(203,)g(line)j(1)e(reads)189 1372 y Fj(MPI)p 274 1372 14 2 v 15 w(PCONTROL\(level\))189 1429 y Fq(but)g(should)h(read)189 1485 y Fj(MPI)p 274 1485 V 15 w(PCONTROL\(LEVEL\))189 1657 y Fm(Discussion)o(:)189 1726 y Fl(This)d(is)h(a)g(minor)e(t)o(yp)q(o.)143 1867 y Fi(\017)23 b Fq(P)o(age)14 b(210,)g(line)j(44)e(reads)189 1923 y Fd(MPI)p 266 1923 13 2 v 14 w(PENDING)189 1980 y Fq(but)g(should)h(read)189 2036 y Fd(MPI)p 266 2036 V 14 w(ERR)p 359 2036 V 14 w(PENDING)189 2209 y Fm(Discussion)o(:)189 2277 y Fl(This)d(is)h(a)g(t)o(yp)q(o.)k(See)c(page)g(197,)f(line)g(20)h (for)f(the)i(original)d(de\014nition.)143 2418 y Fi(\017)23 b Fq(P)o(age)14 b(211,)g(line)j(44)e(reads)189 2475 y Fd(MPI)p 266 2475 V 14 w(DOUBLE)p 445 2475 V 14 w(COMPLEX)189 2531 y Fq(but)g(should)h(b)q(e)g(mo)o(v)o(ed)f(to)f(P)o(age)h(212,)f (line)j(22)d(since)j(it)e(is)h(an)f(optional)h(F)l(ortran)e(datat)o(yp) q(e.)189 2647 y Fm(Discussion)o(:)1967 46 y Fh(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 19 21 19 20 bop 75 -100 a Fg(3.10.)34 b(MINOR)16 b(CORRECTIONS)1109 b Fq(19)189 45 y Fl(On)20 b(page)h(19,)f(lines)h(2-3,)f(it)g(clearly)h (states)g(this)f(is)h(an)f(optional)f(F)m(ortran)h(datat)o(yp)q(e.)37 b(It)21 b(is)f(listed)189 95 y(incorrectly)14 b(in)g(the)g(app)q (endix.)1875 150 y Fi(>)i Fh(\(Oct\))143 236 y Fi(\017)23 b Fq(P)o(age)14 b(212,)g(add)h(new)h(lines)h(of)d(text)h(at)g(line)i (22)d(and)i(line)g(25)f(to)g(read:)189 292 y(etc.)189 349 y(Th)o(us,)f(the)i(text)e(will)j(no)o(w)e(read:)189 518 y Fk(/*)23 b(optional)g(datatypes)g(\(Fortran\))f(*/)189 574 y(MPI_INTEGER1)189 631 y(MPI_INTEGER2)189 687 y(MPI_INTEGER4)189 744 y(MPI_REAL2)189 800 y(MPI_REAL4)189 857 y(MPI_REAL8)189 913 y(etc.)189 1026 y(/*)h(optional)g(datatypes)g(\(C\))g(*/)189 1082 y(MPI_LONG_LONG_INT)189 1139 y(etc.)189 1292 y Fm(Discussion)o(:) 189 1360 y Fl(On)15 b(page)g(19,)f(line)g(6,)h(concludes)h(the)f(list)f (of)h(optional)e(datat)o(yp)q(es)j(with)e(etc.)22 b(It)15 b(w)o(as)g(not)g(in)o(tended)g(to)189 1410 y(b)q(e)f(a)g(complete)g (list)g(of)f(all)g(p)q(ossible)i(ones.)20 b(Some)13 b(ha)o(v)o(e)h(in)o (terpreted)i(the)e(app)q(endix)h(to)f(b)q(e)h(a)f(complete)189 1460 y(list)f(since)h(only)f(the)h(ones)h(explictly)e(on)g(page)h(19)f (are)h(listed.)k(Adding)13 b(etc.,)h(to)g(the)g(F)m(ortran)f(and)h(C)f (list)189 1510 y(in)g(the)i(app)q(endix)e(will)g(mak)o(e)f(it)i(clear)g (these)h(are)g(only)e(some)g(of)g(the)i(p)q(ossible)f(optional)e(datat) o(yp)q(es.)1875 1565 y Fi(?)k Fh(\(Oct\))143 1651 y Fi(\017)23 b Fq(P)o(age)14 b(213,)g(line)j(28.)i(The)d(follo)o(wing)g(text)e (should)i(b)q(e)g(added:)189 1707 y(/*)9 b(Prede\014ned)j(functions)f (in)g(C)e(and)i(F)l(ortran)e(*/)g(MPI)p 1137 1707 14 2 v 17 w(NULL)p 1278 1707 V 17 w(COPY)p 1428 1707 V 17 w(FN)g(MPI)p 1607 1707 V 17 w(NULL)p 1748 1707 V 17 w(DELETE)p 1954 1707 V 16 w(FN)189 1764 y(MPI)p 281 1764 V 16 w(DUP)p 397 1764 V 16 w(FN)189 1880 y Fm(Discussion)o(:)189 1948 y Fl(These)15 b(are)f(missing)e(and)i(de\014ned)h(on)e(pages)i(169,)d (line)i(17)f(and)h(p.)k(170,)13 b(line)g(4.)143 2089 y Fi(\017)23 b Fq(P)o(age)14 b(220,)g(lines)j(19-20)d(reads)189 2146 y Fj(int)i(double)g(MPI)p 479 2146 V 16 w(Wtime\(void\))189 2202 y(int)g(double)g(MPI)p 479 2202 V 16 w(Wtick\(void\))189 2258 y Fq(but)f(should)h(read)189 2315 y Fj(double)g(MPI)p 413 2315 V 16 w(Wtime\(void\))189 2371 y(double)g(MPI)p 413 2371 V 16 w(Wtick\(void\))189 2544 y Fm(Discussion)o(:)189 2612 y Fl(This)d(w)o(as)h(a)g(mistak)o(e)e(caused)j(b)o(y)f(the)g (macros.)-32 46 y Fh(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 20 22 20 21 bop 75 -100 a Fq(20)947 b Fg(CHAPTER)15 b(3.)35 b(MISCELLANY)17 b(F)o(OR)e(1.2)143 45 y Fi(\017)23 b Fq(P)o(age)14 b(222,)g(line)j(34)e(reads)189 102 y Fj(INTEGER)f (REQUEST,)g(COUNT,)g(D)o(A)l(T)l(A)l(TYPE,)g(DEST,)g(T)l(A)o(G,)f (COMM,)g(REQUEST,)i(IERROR)189 158 y Fq(but)g(should)h(read)189 214 y Fj(INTEGER)f(COUNT,)h(D)o(A)l(T)l(A)l(TYPE,)g(DEST,)f(T)l(A)o(G,) g(COMM,)g(REQUEST,)h(IERROR)189 387 y Fm(Discussion)o(:)189 455 y Fl(An)e(extra)g(argumen)o(t)f(got)g(added.)19 b(It)14 b(needs)h(to)f(b)q(e)g(remo)o(v)o(ed.)143 596 y Fi(\017)23 b Fq(P)o(age)14 b(222,)g(line)j(38)e(reads)189 653 y Fj(INTEGER)f(REQUEST,)g(COUNT,)g(D)o(A)l(T)l(A)l(TYPE,)g(DEST,)g(T)l(A) o(G,)f(COMM,)g(REQUEST,)i(IERROR)189 709 y Fq(but)g(should)h(read)189 766 y Fj(INTEGER)f(COUNT,)h(D)o(A)l(T)l(A)l(TYPE,)g(DEST,)f(T)l(A)o(G,) g(COMM,)g(REQUEST,)h(IERROR)189 938 y Fm(Discussion)o(:)189 1006 y Fl(An)e(extra)g(argumen)o(t)f(got)g(added.)19 b(It)14 b(needs)h(to)f(b)q(e)g(remo)o(v)o(ed.)143 1147 y Fi(\017)23 b Fq(P)o(age)14 b(227,)g(lines)j(19-20)d(reads)189 1204 y Fj(MPI)p 274 1204 14 2 v 15 w(INTERCOMM)p 563 1204 V 17 w(MERGE\(INTERCOMM,)i(HIGH,)f(INTRA)o(COMM,)g(IERROR\))189 1260 y(INTEGER)g(INTERCOMM,)h(INTRA)o(COMM,)f(IERROR)189 1317 y Fq(but)g(should)h(read)189 1373 y Fj(MPI)p 274 1373 V 15 w(INTERCOMM)p 563 1373 V 17 w(MERGE\(INTERCOMM,)g(HIGH,)f (NEWINTRA)o(COMM,)g(IERROR\))189 1430 y(INTEGER)g(INTERCOMM,)h (NEWINTRA)o(COMM,)f(IERROR)189 1602 y Fm(Discussion)o(:)189 1670 y Fl(This)g(mak)o(es)e(it)i(matc)o(h)f(the)i(de\014nition)e(in)h (the)g(text)h(on)f(P)o(age)g(159,)f(lines)h(34-36.)20 b(It)15 b(only)g(c)o(hanges)g(the)189 1720 y(name)d(of)i(the)g(argumen) o(t.)143 1861 y Fi(\017)23 b Fq(P)o(age)14 b(228,)g(line)j(46)e(reads) 189 1918 y Fj(MPI)p 274 1918 V 15 w(ERRHANDLER)p 582 1918 V 18 w(CREA)l(TE\(FUNCTION,)h(HANDLER,)g(IERROR\))189 1974 y Fq(but)f(should)h(read)189 2031 y Fj(MPI)p 274 2031 V 15 w(ERRHANDLER)p 582 2031 V 18 w(CREA)l(TE\(FUNCTION,)g (ERRHANDLER,)h(IERROR\))189 2203 y Fm(Discussion)o(:)189 2271 y Fl(This)c(c)o(hange)i(mak)o(es)d(the)j(app)q(endix)f(consistan)o (t)g(with)g(c)o(hange)g(on)g(p.)k(194,)12 b(line)i(48.)143 2412 y Fi(\017)23 b Fq(P)o(age)14 b(229,)g(line)j(33)e(reads)189 2469 y Fj(MPI)p 274 2469 V 15 w(PCONTROL\(level\))189 2525 y Fq(but)g(should)h(read)189 2582 y Fj(MPI)p 274 2582 V 15 w(PCONTROL\(LEVEL\))1967 46 y Fh(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 21 23 21 22 bop 75 -100 a Fg(3.10.)34 b(MINOR)16 b(CORRECTIONS)1109 b Fq(21)189 45 y Fm(Discussion)o(:)189 114 y Fl(This)13 b(is)h(to)g(mak)o(e)e(the)j(language)e(binding)g(the)h(same)f(as)h(the) h(c)o(hange)f(to)g(P)o(age)f(203,)g(line)g(1.)-32 46 y Fh(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From mpi-comm-human@mcs.anl.gov Fri Sep 20 08:44:04 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id IAA27997; Fri, 20 Sep 1996 08:43:34 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id XAA28152 for mpi-comm-out; Thu, 19 Sep 1996 23:26:32 -0500 Received: from mcs.anl.gov (donner.mcs.anl.gov [140.221.5.134]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id XAA28135; Thu, 19 Sep 1996 23:26:02 -0500 Message-Id: <199609200426.XAA28135@antares.mcs.anl.gov> To: mpi-comm@antares.mcs.anl.gov, mpi-core@antares.mcs.anl.gov Subject: Misc-1.2 Date: Thu, 19 Sep 1996 23:26:00 -0500 From: Rusty Lusk Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk %!PS-Adobe-2.0 %%Creator: dvips 5.528 Copyright 1986, 1994 Radical Eye Software %%Title: temp.dvi %%CreationDate: Thu Sep 19 16:46:18 1996 %%Pages: 23 %%PageOrder: Ascend %%BoundingBox: 0 0 612 792 %%EndComments %DVIPSCommandLine: dvips -o temp.ps temp %DVIPSParameters: dpi=300, comments removed %DVIPSSource: TeX output 1996.09.19:1646 %%BeginProcSet: tex.pro /TeXDict 250 dict def TeXDict begin /N{def}def /B{bind def}N /S{exch}N /X{S N}B /TR{translate}N /isls false N /vsize 11 72 mul N /hsize 8.5 72 mul N /landplus90{false}def /@rigin{isls{[0 landplus90{1 -1}{-1 1} ifelse 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale isls{landplus90{VResolution 72 div vsize mul 0 exch}{Resolution -72 div hsize mul 0}ifelse TR}if Resolution VResolution vsize -72 div 1 add mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@landscape{/isls true N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder}N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail}B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{ pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get}B /ch-xoff{128 ch-data dup length 3 sub get sub}B /ch-yoff{ch-data dup length 2 sub get 127 sub}B /ch-dx{ch-data dup length 1 sub get}B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ ch-image}imagemask restore}B /D{/cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D }B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto /V matrix currentmatrix dup 1 get dup mul exch 0 get dup mul add .99 lt{/QV}{/RV}ifelse load def pop pop}N /eop{SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if pop /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put}for 65781.76 div /vsize X 65781.76 div /hsize X}N /p{show}N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V {}B /RV statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval dup(Display)eq exch 0 4 getinterval(NeXT)eq or}{pop false} ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /QV{gsave transform round exch round exch itransform moveto rulex 0 rlineto 0 ruley neg rlineto rulex neg 0 rlineto fill grestore}B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail} B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{ 3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{ 3 2 roll p a}B /bos{/SS save N}B /eos{SS restore}B end %%EndProcSet TeXDict begin 40258431 52099146 1000 300 300 (/tmp_mnt/Net/antireo/antireo6/lusk/mpi2/report2/temp.dvi) @start /Fa 1 106 df<030007800780030000000000000000007F807F80038003800380 038003800380038003800380038003800380FFFCFFFC0E187D9714>105 D E /Fb 12 122 df<387CFEFEFE7C3807077C860F>46 D<00E00001E0000FE000FFE000 F3E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E000 03E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E000FFFF80 FFFF80111D7C9C1A>49 D<01FC0007FF000E0F801E0FC03F07E03F07E03F07E03F07E01E 0FC0000FC0000F80001F0001FC0001FC00000F800007C00003E00003F00003F83803F87C 03F8FE03F8FE03F8FE03F0FC03F07807E03C0FC01FFF8003FC00151D7E9C1A>51 D69 D<07FC001FFF003F0F803F07 C03F03E03F03E00C03E00003E0007FE007FBE01F03E03C03E07C03E0F803E0F803E0F803 E0FC05E07E0DE03FF8FE0FE07E17147F9319>97 D<01FE0007FF800F83C01E01E03E00F0 7C00F07C00F8FC00F8FFFFF8FFFFF8FC0000FC0000FC00007C00007C00003E00181E0018 0F807007FFE000FF8015147F9318>101 D108 DI112 D<01800180018003800380038007800F803F80FFFCFFFC0F800F800F800F80 0F800F800F800F800F800F800F860F860F860F860F8607CC03F801F00F1D7F9C14>116 D120 DI E /Fc 4 103 df<70F8FCFC74040404080810102040060E7C840D>59 D<00F1800389C00707800E03801C03803C0380380700780700780700780700F00E00F00E 00F00E00F00E10F01C20F01C20703C20705C40308C400F078014147E9318>97 D<07803F8007000700070007000E000E000E000E001C001C001CF01D0C3A0E3C0E380F38 0F700F700F700F700FE01EE01EE01EE01CE03CE038607060E031C01F0010207E9F14>I< 00007C0000CE00019E00039E00030C000700000700000700000700000E00000E00000E00 00FFF0000E00000E00001C00001C00001C00001C00001C00003800003800003800003800 00380000700000700000700000700000700000E00000E00000E00000E00000C00001C000 318000798000F300006200003C000017297E9F16>102 D E /Fd 33 122 df<00E001C0038007000E000E001C001C003800380038007000700070007000E0 00E000E000E000E000E000E000E000E000E000E000E00070007000700070003800380038 001C001C000E000E000700038001C000E00B2A7E9E10>40 DI44 D<001C0000003E0000003E0000 002E0000006700000067000000E7800000C7800000C3800001C3C0000183C0000181C000 0381E0000381E0000700F0000700F0000600F0000E0078000FFFF8000FFFF8001C003C00 1C003C0018003C0038001E0038001E0070001F0070000F0070000F00E0000780191D7F9C 1C>65 DI<003FC000FFF003C0F007 80300F00001E00003C00003C0000780000780000780000F00000F00000F00000F00000F0 0000F00000F00000F00000F000007800007800007800003C00003C00001E00000F000807 801803C07800FFF0003F80151F7D9D1B>III<003F8001FFF003C0F807 80380F00181E00003C00003C0000780000780000780000F00000F00000F00000F00000F0 0000F00000F007F8F007F8F000387800387800387800383C00383C00381E00380F003807 803803C0F801FFF0003F80151F7D9D1C>71 D73 D76 DII<003F000001FFE00003FFF00007C0F8000F807C001E 001E003E001F003C000F00780007807800078078000780F00003C0F00003C0F00003C0F0 0003C0F00003C0F00003C0F00003C0F00003C0F80007C078000780780007807C000F803C 000F003E001F001F003E000F807C0007C0F80003FFF00001FFE000003F00001A1F7E9D1F >II<003F000001FFE00003FFF000 07C0F8000F807C001F003E003E001F003C000F00780007807800078078000780F00003C0 F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C078000780 78000780780E07803C0F0F003E079F001E03DE000F83FC0007C1F80003FFF00001FFF800 003F780000003C0000003E0000001F0000000F801A237E9D1F>II<03F8000FFE001C0F00380700700300600000E00000E000 00E00000E00000F000007800007F00003FE0001FFC0007FE0001FF00001F800007800003 C00003C00001C00001C00001C00001C0C00180E00380F007007C0E001FFC0007F000121F 7E9D17>III<78000E007C001E003C003C001E0038000F0070000F00F0000781E0 0003C1C00001C3C00001E7800000F70000007E0000003E0000003C0000003C0000007E00 000077000000E7800001E3800003C1C0000381E0000700F0000F00F8000E0078001C003C 003C003E0078001F0070000F00F0000F80191D7F9C1C>88 DI<0FC03FF07FF87038401C001C001C00FC0FFC3FFC781CE01CE01CE01CF07C7FFC 7FDC3F1C0E127E9114>97 DI<000E000E000E000E000E000E000E000E000E000E000E0F8E1FEE3FFE7C3E70 0E700EE00EE00EE00EE00EE00EE00EF00E701E7C3E3FFE1FEE0F8E0F1D7E9C15>100 D<07C01FE03FF078787018601CFFFCFFFCFFFCE000E000E000700070043C1C3FFC1FF807 E00E127E9112>I108 D110 D112 D<1C001C001C001C001C001C00FFE0FFE01C001C001C001C001C001C00 1C001C001C001C001C001C001C201FF00FF007C00C187F970F>116 DI<7003807807003C0E001C1C000E1C0007380003F00001E00001C00001 E00003F0000738000E18000E1C001C0E00380700700380F003C01212809113>120 DI E /Fe 24 119 df<00003F03E00000C386700001878CF00003879CF00003031860000700380000 070038000007003800000E003800000E007000000E007000000E00700000FFFFFF80001C 007000001C00E000001C00E000001C00E000001C00E000003800E000003801C000003801 C000003801C000003801C000007001C00000700380000070038000007003800000700380 00006003800000E007000000E007000000E007000000E007000000C006000001C00E0000 01C00E000031860C0000798F180000F31E100000620C6000003C07C000002429829F1C> 11 D<70F8F8F0E005057B840E>46 D<00000200000006000000060000000E0000001E00 00001E0000003F0000002F0000004F0000004F0000008F0000010F0000010F0000020F00 00020F0000040F00000C0F0000080F0000100F0000100F0000200F80003FFF8000400780 00C007800080078001000780010007800200078002000780060007801E000F80FF807FF8 1D207E9F22>65 D<01FFFFFE001E001C001E000C001E0004001E0004003C0004003C0004 003C0004003C00040078080800780800007808000078180000F0300000FFF00000F03000 00F0300001E0200001E0200001E0200001E0001003C0002003C0002003C0004003C00040 078000800780018007800100078007000F001F00FFFFFE001F1F7D9E1F>69 D<01FFFF00001E03C0001E00E0001E0070001E0078003C0078003C0078003C0078003C00 78007800F0007800F0007801E0007801C000F0070000F01E0000FFF00000F0380001E01C 0001E01E0001E00E0001E00F0003C01E0003C01E0003C01E0003C01E0007803C0007803C 0807803C0807803C100F801C10FFF00C20000007C01D207D9E21>82 D<00F1800389C00707800E03801C03803C0380380700780700780700780700F00E00F00E 00F00E00F00E20F01C40F01C40703C40705C40308C800F070013147C9317>97 D<07803F8007000700070007000E000E000E000E001C001C001CF01D0C3A0E3C0E380F38 0F700F700F700F700FE01EE01EE01EE01CE03CE038607060E031C01F0010207B9F15>I< 007E0001C1000300800E07801E07801C07003C0200780000780000780000F00000F00000 F00000F00000F0000070010070020030040018380007C00011147C9315>I<0000780003 F80000700000700000700000700000E00000E00000E00000E00001C00001C000F1C00389 C00707800E03801C03803C0380380700780700780700780700F00E00F00E00F00E00F00E 20F01C40F01C40703C40705C40308C800F070015207C9F17>I<007C01C207010E011C01 3C013802780C7BF07C00F000F000F000F0007000700170023804183807C010147C9315> I<00007800019C00033C00033C000718000700000700000E00000E00000E00000E00000E 0001FFE0001C00001C00001C00001C000038000038000038000038000038000070000070 0000700000700000700000700000E00000E00000E00000E00000C00001C00001C0000180 003180007B0000F300006600003C00001629829F0E>I<003C6000E27001C1E00380E007 00E00F00E00E01C01E01C01E01C01E01C03C03803C03803C03803C03803C07003C07001C 0F001C17000C2E0003CE00000E00000E00001C00001C00301C00783800F0700060E0003F 8000141D7E9315>I<01E0000FE00001C00001C00001C00001C000038000038000038000 038000070000070000071E000763000E81800F01C00E01C00E01C01C03801C03801C0380 1C0380380700380700380700380E10700E20700C20701C20700C40E00CC060070014207D 9F17>I<00C001E001E001C000000000000000000000000000000E003300230043804300 470087000E000E000E001C001C001C003840388030807080310033001C000B1F7C9E0E> I<03C01FC0038003800380038007000700070007000E000E000E000E001C001C001C001C 0038003800380038007000700070007100E200E200E200E200640038000A207C9F0C> 108 D<1C0F80F0002630C318004740640C004780680E004700700E004700700E008E00E0 1C000E00E01C000E00E01C000E00E01C001C01C038001C01C038001C01C038001C01C070 8038038071003803806100380380E10038038062007007006600300300380021147C9325 >I<1C0F802630C04740604780604700704700708E00E00E00E00E00E00E00E01C01C01C 01C01C01C01C03843803883803083807083803107003303001C016147C931A>I<007C00 01C3000301800E01C01E01C01C01E03C01E07801E07801E07801E0F003C0F003C0F003C0 F00780F00700700F00700E0030180018700007C00013147C9317>I<01C1E00262180474 1C04781C04701E04701E08E01E00E01E00E01E00E01E01C03C01C03C01C03C01C0380380 780380700380E003C1C0072380071E000700000700000E00000E00000E00000E00001C00 001C0000FFC000171D809317>I<1C1E002661004783804787804707804703008E00000E 00000E00000E00001C00001C00001C00001C000038000038000038000038000070000030 000011147C9313>114 D<00FC030206010C030C070C060C000F800FF007F803FC003E00 0E700EF00CF00CE008401020601F8010147D9313>I<018001C003800380038003800700 0700FFF007000E000E000E000E001C001C001C001C003800380038003820704070407080 708031001E000C1C7C9B0F>I<0E00C03300E02301C04381C04301C04701C08703800E03 800E03800E03801C07001C07001C07001C07101C0E20180E20180E201C1E200C264007C3 8014147C9318>I<0E03803307802307C04383C04301C04700C08700800E00800E00800E 00801C01001C01001C01001C02001C02001C04001C04001C08000E300003C00012147C93 15>I E /Ff 50 123 df<000078007C7800FC7801FC7803C00003800007800007800007 8000078000078000078000078000078000FFFC78FFFC78FFFC7807807807807807807807 807807807807807807807807807807807807807807807807807807807807807807807807 8078078078078078078078152480A31A>12 D45 DI<00C001C007C0FFC0FFC0FBC003C003C003C003C003C003C0 03C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C0 03C0FFFFFFFFFFFF10227CA118>49 D<03F0000FFC001FFE003C1F003007807007C06003 C0E003E0C001E04001E04001E00001E00001E00001E00003C00003C0000780000780000F 00001E00003C0000780000F00001E00001C0000380000700000E00001C00003800007000 00FFFFE0FFFFE0FFFFE013227EA118>I<01F00007FC001FFF003E0F003807807003C020 03C02003C00003C00003C00003C0000780000780000F00001E0003FC0003F80003FE0000 0F000007800003C00003C00001E00001E00001E00001E00001E08001E0C003C0E003C070 07803C0F801FFF000FFC0003F00013237EA118>I<001F00001F00002F00002F00006F00 00EF0000CF0001CF0001CF00038F00038F00078F00070F000F0F000E0F001E0F003C0F00 3C0F00780F00780F00F00F00FFFFF8FFFFF8FFFFF8000F00000F00000F00000F00000F00 000F00000F00000F00000F0015217FA018>I55 D<01F00007FC000FFE001E0F003C07803C0780 7803C07803C07803C07803C07803C03803803C07801E0F000F1E0007FC0003F8000FFE00 1E0F003C07807803C07803C0F001E0F001E0F001E0F001E0F001E0F001E07803C07803C0 3C07803E0F801FFF0007FC0001F00013237EA118>I<01F00007FC000FFE001E0F003C07 003807807803807803C0F003C0F001C0F001E0F001E0F001E0F001E0F001E0F001E07003 E07803E07803E03C07E03E1FE01FFDE00FF1E003E3C00003C00003C00003800007800007 00000F00101E00183C003FF8001FF00007C00013237EA118>I<001F0000001F0000003F 8000003F8000003B8000007BC0000073C0000071C00000F1E00000F1E00000E0E00001E0 F00001E0F00001C0F00003C0780003C078000380780007803C0007803C0007003C000F00 1E000F001E000FFFFE001FFFFF001FFFFF001C000F003C0007803C000780380007807800 03C0780003C0700003C0F00001E0F00001E0E00001E01B237EA220>65 DI<000FF000007FFC0000FFFF0001F01F0003C00700078000000F0000001E0000 003E0000003C0000003C000000780000007800000078000000F0000000F0000000F00000 00F0000000F0000000F0000000F0000000F0000000F00000007800000078000000780000 003C0000003C0000003E0000001E0000000F0000000780008003C0038001F00F8000FFFF 00007FFC00000FF00019257DA31F>IIII<000FF000003FFE0000 FFFF8001F80F8003E00380078000000F0000001E0000001E0000003C0000003C00000078 0000007800000078000000F0000000F0000000F0000000F0000000F0000000F0000000F0 00FFC0F000FFC0F000FFC0780003C0780003C0780003C03C0003C03C0003C01E0003C01E 0003C00F0003C0078003C003E003C001F807C000FFFFC0003FFF00000FF8001A257DA321 >I73 D76 DII<001FC000007FF00001FFFC0003F07E 0007C01F000F800F801F0007C01E0003C03C0001E03C0001E0780000F0780000F0780000 F070000070F0000078F0000078F0000078F0000078F0000078F0000078F0000078F00000 78F0000078780000F0780000F0780000F07C0001F03C0001E03E0003E01E0003C01F0007 C00F800F8007C01F0003F07E0001FFFC00007FF000001FC0001D257DA324>II< FFFC00FFFF80FFFFC0F003E0F000F0F00078F00038F0003CF0003CF0003CF0003CF0003C F00038F00078F000F0F003E0FFFFC0FFFF80FFFE00F01E00F00F00F00700F00780F00380 F003C0F001E0F001E0F000F0F000F0F00078F00038F0003CF0001EF0001EF0000F18237B A21F>82 D<00FE0003FFC007FFE00F81E01E00603C00003C000078000078000078000078 00007800007C00003C00003F00001FC0000FFC0007FF0001FF80003FC00007E00001F000 00F00000F8000078000078000078000078000078000078C000F0E000F0F801E07E07C03F FF800FFF0001FC0015257EA31B>III90 D<07E01FF83FFC381E201E000F000F000F000F00FF07FF1FFF3E0F780FF00FF00FF00FF0 0FF83F7FFF3FEF1F8F10167E9517>97 DI<01FC0007FF000FFF801F03803C0180 780000780000700000F00000F00000F00000F00000F00000F00000780000780000780000 3C00401F03C00FFFC007FF8001FC0012167E9516>I<0003C00003C00003C00003C00003 C00003C00003C00003C00003C00003C00003C00003C00003C003E3C00FFBC01FFFC03F0F C03C07C07803C07803C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C07803 C07803C03C07C03E0FC01FFFC00FFBC003E3C012237EA219>I<03F00007FC001FFE003E 0F003C0780780380780380F001C0FFFFC0FFFFC0FFFFC0F00000F00000F0000070000078 00007800003C00801F07800FFF8007FF0001F80012167E9516>I<003F00FF01FF03C003 8007800780078007800780078007800780FFF8FFF8FFF807800780078007800780078007 80078007800780078007800780078007800780078007800780102380A20F>I<01F07807 FFF80FFFF81F1F001E0F003C07803C07803C07803C07803C07801E0F001F1F000FFE001F FC0019F0003800003800003C00001FFE001FFFC01FFFE03FFFF07801F07800F8F00078F0 0078F00078F000787800F03E03E01FFFC00FFF8001FC0015217F9518>I 105 D107 DIII<01FC0007FF000FFF801F07C03C01E07800F078 00F0700070F00078F00078F00078F00078F00078F000787800F07800F07C01F03E03E01F 07C00FFF8007FF0001FC0015167F9518>II114 D<07F01FFC3FFE3C0E7806780078007C003F003FF01FF80FFC01FE001F000F000F000FC0 0FF81EFFFE3FFC0FF010167F9513>I<0F000F000F000F000F000F00FFF8FFF8FFF80F00 0F000F000F000F000F000F000F000F000F000F000F000F000F000F080F1C07FC07F803E0 0E1C7F9B12>III<7801F07C01E03E03C01E07C00F 0780078F0007DE0003FC0001FC0000F80000700000F80001FC0003DC00039E00078F000F 07801E07801E03C03C01E07800F0F800F81516809516>120 DI<7FFFC07FFFC07FFFC0000F80000F00 001E00003E00007C0000780000F00001F00003E00003C0000780000F80001F00001E0000 3C00007C0000FFFFC0FFFFC0FFFFC012167F9515>I E /Fg 32 90 df45 D<3078F8787005057C840D>I<007E0001C3000301800701 C00E00C00E00E01C00E01C00E03C01E03801E07801E07801E07801E07801E07801E0F003 C0F003C0F003C0F003C0F003C0F00380F00780E00780E00700E00700E00E00600E00701C 003038003870000FC000131F7C9D17>48 D<000C001C00FC0F3800380038003800380038 00700070007000700070007000E000E000E000E000E000E001C001C001C001C001C001C0 038003C0FFFE0F1E7C9D17>I<003F8000C1E00100F00200780400780400780F007C0F80 7C0F807C0F00780600780000F80000F00001E00001C0000380000700000E00001C000038 0000600000C0000180000300200600200800401000403FFFC07FFF80FFFF80161E7E9D17 >I<007F000183C00201E00400F00700F00F00F00F01F00F01F00001E00001E00003C000 0380000700000E0000F800000E000007000007800007C00003C00007C03007C07807C0F8 07C0F807C0F00780800F00400E00201C0018780007E000141F7D9D17>I<01803001FFE0 03FFC003FF0003FC00020000020000020000040000040000040000047C00058700060380 0C01800801C00001C00001E00001E00001E00001E07003C0F803C0F003C0E00380800780 400700400E00201C0018700007C000141F7D9D17>53 D<2000003FFFE07FFFC07FFF8040 0100C0020080020080040000080000100000200000400000400000800001800003000003 00000700000600000E00000E00001E00001C00001C00003C00003C00003C000078000078 0000780000300000131F799D17>55 D<003F0000C1C00100600200600400300C00300C00 300C00300C00600E00600F80C00FC18007F60003FC0001FC0001FF00063F800C0F801007 C03003C06001C06000C0C000C0C000C0C000C0C00080C0010060030030040018180007E0 00141F7D9D17>I<007E0001C3000301800601C00E01C01C00C03C00E03C00E03C01E078 01E07801E07801E07801E07803E07803E03803C03807C01C0BC00C13C003E38000078000 0780000700000E00600E00F01C00F01800E0300080600041C0003F0000131F7C9D17>I< 0000100000001800000038000000380000007800000078000000FC000001BC0000013C00 00033C0000023C0000063C0000043E0000081E0000081E0000101E0000101E0000201E00 00200F0000400F0000400F0000FFFF0000800F0001000F80010007800200078002000780 04000780040007800C0007C03E0007C0FF807FFC1E207E9F22>65 D<0001F808000E061800380138007000F801E0007803C0007007800030078000300F0000 301F0000301E0000303E0000203C0000007C0000007C0000007C0000007C000000F80000 00F8000000F8000000F8000000F80000007800004078000080780000803C0000803C0001 001C0002000E00020006000C000300100001C0E000003F00001D217B9F21>67 D<07FFFF00007C01E0003C00F0003C00780078003C0078003C0078001E0078001E007800 1E0078001F00F0001F00F0001F00F0001F00F0001F00F0001F00F0001F01E0001E01E000 3E01E0003E01E0003E01E0003C01E0007C03C0007803C000F003C000F003C001E003C003 C003C0078007800F0007803C00FFFFE000201F7E9E23>I<07FFFFF8007C0078003C0038 003C001800780018007800080078000800780008007800080078080800F0100000F01000 00F0100000F0300000FFF00000F0700001E0200001E0200001E0200001E0200001E00008 01E0001003C0001003C0001003C0002003C0002003C0006003C000C0078001C0078007C0 FFFFFF801D1F7E9E1F>I<07FFFFF8007C0078003C0038003C0018007800180078000800 78000800780008007800080078000800F0100000F0100000F0100000F0300000F0700000 FFF00001E0600001E0200001E0200001E0200001E0200001E0000003C0000003C0000003 C0000003C0000003C0000003C000000780000007C00000FFFE00001D1F7E9E1E>I<0001 FC04000F030C003C009C0070007C00E0003C01C0003803800018078000180F0000181F00 00181E0000183E0000103C0000007C0000007C0000007C0000007C000000F8000000F800 0000F8007FFCF80003E0780001E0780001E0780003C0780003C03C0003C03C0003C01C00 03C00E0007C007000B800380118001E06080003F80001E217B9F24>I<07FFC7FFC0007C 00F800003C007800003C007800007800F000007800F000007800F000007800F000007800 F000007800F00000F001E00000F001E00000F001E00000F001E00000FFFFE00000F001E0 0001E003C00001E003C00001E003C00001E003C00001E003C00001E003C00003C0078000 03C007800003C007800003C007800003C007800003C007800007800F000007C00F8000FF F8FFF800221F7E9E22>I<07FFE0007C00003C00003C0000780000780000780000780000 780000780000F00000F00000F00000F00000F00000F00001E00001E00001E00001E00001 E00001E00003C00003C00003C00003C00003C00003C00007800007C000FFFC00131F7F9E 10>I<07FFE07FE0007C001F00003C000C00003C00180000780010000078004000007800 800000780100000078020000007804000000F008000000F010000000F060000000F0F000 0000F1F0000000F278000001E478000001E878000001F03C000001E03C000001E01E0000 01E01E000003C00F000003C00F000003C00F000003C007800003C007800003C003C00007 8003C00007C007E000FFFC3FFC00231F7E9E23>75 D<07FFF000007E0000003C0000003C 000000780000007800000078000000780000007800000078000000F0000000F0000000F0 000000F0000000F0000000F0000001E0000001E0000001E0000001E0000001E0008001E0 010003C0010003C0010003C0030003C0020003C0060003C0060007801E0007807C00FFFF FC00191F7E9E1C>I<07FC0000FFC0007C0000F800003C00017800003C00017800004E00 02F000004E0002F000004E0004F000004E0004F000004E0008F000004E0008F000008700 11E00000870011E00000870021E00000870021E00000870041E00000838041E000010380 83C00001038083C00001038103C00001038203C0000101C203C0000101C403C0000201C4 0780000201C80780000201C80780000201D00780000200F00780000600E00780000600E0 0F00000F00C00F8000FFE0C1FFF8002A1F7E9E2A>I<07FC01FFC0003E003E00003E0018 00003E001800004F001000004F001000004780100000478010000043C010000043C01000 0083C020000081E020000081E020000080F020000080F020000080782000010078400001 007C400001003C400001003C400001001E400001001E400002000F800002000F80000200 0F800002000780000200078000060003800006000300000F00010000FFE0010000221F7E 9E22>I<0003F800001E0E000038070000E0038001C001C003C001E0078000E00F0000F0 0F0000F01E0000F01E0000F83E0000F83C0000F87C0000F87C0000F87C0000F87C0000F8 F80001F0F80001F0F80001F0F80001F0F80003E0780003E0780003C0780007C07C000780 3C000F003C001E001E001C000E0038000700F00003C3C00000FE00001D217B9F23>I<07 FFFF00007C03C0003C01E0003C00F0007800F0007800F8007800F8007800F8007800F800 7800F000F001F000F001E000F003C000F0078000F00F0000FFF80001E0000001E0000001 E0000001E0000001E0000001E0000003C0000003C0000003C0000003C0000003C0000003 C000000780000007C00000FFFC00001D1F7E9E1F>I<07FFFC00007C0700003C03C0003C 01E0007801E0007801F0007801F0007801F0007801F0007801E000F003E000F003C000F0 078000F00F0000F03C0000FFF00001E0300001E0380001E01C0001E01C0001E01C0001E0 1E0003C03E0003C03E0003C03E0003C03E0003C03E0003C03E0207803E0407C01F04FFFC 0F18000003E01F207E9E21>82 D<003F040060CC01803C03801C03001C0700180600080E 00080E00080E00080E00000F00000F80000FE00007FE0003FF8001FFC0007FE00007E000 01E00000E00000F00000F04000E04000E04000E04000E06000C0600180E00380F80300C6 0C0081F80016217D9F19>I<3FFFFFF03C0780F03007803060078030400F0010400F0010 C00F0010800F0010800F0010800F0010001E0000001E0000001E0000001E0000001E0000 001E0000003C0000003C0000003C0000003C0000003C0000003C00000078000000780000 0078000000780000007800000078000000F0000001F800007FFFE0001C1F7A9E21>IIII< 03FFC0FFC0007F007E00003E003800001E003000001E002000000F004000000F00800000 0F81000000078200000007C600000003C400000003E800000001F000000001F000000000 F000000000F800000000F8000000017C000000023C000000043C0000000C1E000000081E 000000101F000000200F000000400F800000C0078000008007C000010003C000070003E0 001F8007E000FFE01FFE00221F7F9E22>II E /Fh 15 117 df<020408103020604040C0C0C0C0C0C0C0C0404060203010 080402071A7F920C>40 D<8040201018080C0404060606060606060604040C0818102040 80071A7E920C>I<1F00318060C04040C060C060C060C060C060C060C060C060404060C0 31801F000B107F8F0F>48 D<0C003C00CC000C000C000C000C000C000C000C000C000C00 0C000C000C00FF8009107E8F0F>I<1F00618040C08060C0600060006000C00180030006 000C00102020207FC0FFC00B107F8F0F>I<1F00218060C060C000C0008001800F000080 00400060C060C060804060801F000B107F8F0F>I<0300030007000F000B001300330023 004300C300FFE003000300030003001FE00B107F8F0F>I<20803F002C00200020002000 2F0030802040006000600060C06080C061801F000B107F8F0F>I<0780184030C060C060 00C000CF00F080E040C060C060C060406060C030801F000B107F8F0F>I<40007FE07FC0 8080808001000200040004000C0008000800180018001800180018000B117E900F>I<1F 00318060C060C060C071803F000F00338061C0C060C060C060404060801F000B107F8F0F >I<1F00318060C0C040C060C060C06040E021E01E600060004060C0608043003E000B10 7F8F0F>I<03E0000C1800100400200200600300400100C00180C00180C00180C00180C0 01806003006003003006003006000C180003E00011117E9017>79 D<1F8030C06000C000C000C000C000C000604030801F000A0B7F8A0E>99 D<10103030FE3030303030323232321C070F7F8E0C>116 D E /Fi 3 64 df<03C00FF01FF83FFC7FFE7FFEFFFFFFFFFFFFFFFFFFFFFFFF7FFE7FFE3FFC1FF8 0FF003C010127D9317>15 D62 D<00040000000C0000000C0000 000C0000000C0000000C0000000C0000000C0000000C0000000C0000000C0000000C0000 000C0000000C0000000C0000000C0000000C0000000C0000000C0000000C0000000C0000 000C0000000C0000000C0000000C0000000C0000FFFFFFE0FFFFFFE01B1C7C9B23>I E /Fj 60 123 df<00F8F001F8F003F8F00780F00700F00F00F00F00F00F00F00F00F00F 00F00F00F00F00F0FFF8F0FFF8F0FFF8F00F00F00F00F00F00F00F00F00F00F00F00F00F 00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F01420809F 18>13 D<007000E001C00380078007000E001E001E003C003C003C007800780078007800 7000F000F000F000F000F000F000F000F000F000F000F000F00070007800780078007800 3C003C003C001E001E000E0007000780038001C000E000700C2E7EA112>40 DI<7878787818303060 60E0050A7D830D>44 DII<03F000 07F8001FFE001E1E003C0F00380700780780780780700380F003C0F003C0F003C0F003C0 F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0780780780780 7807803807003C0F001E1E001FFE000FFC0003F00012207E9E17>48 D<00C001C00FC0FFC0FFC0F3C003C003C003C003C003C003C003C003C003C003C003C003 C003C003C003C003C003C003C003C003C003C003C0FFFEFFFEFFFE0F1F7C9E17>I58 D61 D<001F0000001F0000003F8000003B8000003B8000007BC0000073 C0000071C00000F1E00000E1E00000E0E00001E0F00001E0F00001C0F00003C0780003C0 78000380780007803C0007803C0007003C000FFFFE000FFFFE000FFFFE001E000F001E00 0F003C000F803C0007803C000780780007C0780003C0780003C0F00003E01B207F9F1E> 65 DI<001F C000FFF801FFFC03E03C07800C0F00001E00003E00003C00007C00007800007800007800 00F00000F00000F00000F00000F00000F00000F00000F000007800007800007800007C00 003C00003E00001E00000F000207800E03E03E01FFFC00FFF0001FC017227DA01D>IIII<001FE000FFF801FFFE03E03E07800E0F 00001E00003E00003C00007C0000780000780000780000F00000F00000F00000F00000F0 0000F00000F003FEF003FE7803FE78001E78001E7C001E3C001E3E001E1E001E0F001E07 801E03E03E01FFFE00FFF8001FC017227DA01E>III75 DI II<003F0000 00FFC00003FFF00007E1F8000F807C001F003E001E001E003C000F003C000F0078000780 7800078078000780F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0 F00003C0F00003C0F80007C07800078078000780780007803C000F003C000F001E001E00 1F003E000F807C0007E1F80003FFF00000FFC000003F00001A227DA021>II<003F000000FFC00003FF F00007E1F8000F807C001F003E001E001E003C000F003C000F0078000780780007807800 0780F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F000 03C0F00003C07800078078000780781E07803C0F0F003C0F8F001E079E001F03FE000F83 FC0007E1F80003FFF00000FFF800003F780000007C0000003E0000001E0000001F000000 0F801A277DA021>II<01FC0007FF800FFFC01F03C03C00C03C00007800007800007800007800007800 007C00003C00003F00001FE0000FFC0007FE0001FF00003F800007C00003C00003E00001 E00001E00001E00001E00001E00001C0C003C0F007C0FC0F807FFF001FFE0003F8001322 7EA019>IIIII<780007807C000F003E001F001E001E000F003C000F807C0007807800 03C0F00003E1F00001F1E00000F3C000007FC000007F8000003F0000001F0000001E0000 003F0000007F8000007FC00000F3C00001F1E00001E0F00003C0F80007C0780007803C00 0F003E001F001E001E000F003C000F807C0007C0780003C0F00003E01B207F9F1E>III<07E03FF87FFC701E401F000F000F00 0F003F07FF1FFF7E0FF80FF00FF00FF00FF83F7FFF3FEF1F8F10147E9316>97 DI<03F00FFC 1FFE3E0E3C0278007800F000F000F000F000F000F000780078003C013E0F1FFF0FFE03F0 10147E9314>I<0007800007800007800007800007800007800007800007800007800007 8000078000078007C7800FF7801FFF803E1F807C0780780780F80780F00780F00780F007 80F00780F00780F00780F00780780780780F803E1F801FFF800FF78007C78011207E9F17 >I<03F0000FFC001FFE003E1F003C0700780700700380FFFF80FFFF80FFFF80F00000F0 0000F000007000007800003C01003E07001FFF0007FE0001F80011147F9314>I<007E01 FE03FE078007000F000F000F000F000F000F000F00FFF0FFF0FFF00F000F000F000F000F 000F000F000F000F000F000F000F000F000F000F000F000F000F20809F0E>I<03E0F00F FFF01FFFF03E3E003C1E00780F00780F00780F00780F00780F003C1E003E3E001FFC003F F80033E0003000003800003FFE003FFF801FFFC03FFFE07803F0F000F0F000F0F000F0F8 01F07E07E03FFFC00FFF0003FC00141E7F9317>III107 DIII<01F80007FE001FFF803F0FC03C03C07801E07801E0F000 F0F000F0F000F0F000F0F000F0F000F07801E07801E03C03C03F0FC01FFF8007FE0001F8 0014147F9317>II<03C7800FF780 1FFF803E1F807C0F80780780780780F00780F00780F00780F00780F00780F00780F80780 7807807C0F803E1F801FFF800FF78007C780000780000780000780000780000780000780 000780000780000780111D7E9317>II<07F01FFC3FFC780C78 00780078007C003FC01FF00FF803F8007C003C003CC03CF07CFFF87FF00FC00E147F9311 >I<1E001E001E001E001E001E00FFF0FFF0FFF01E001E001E001E001E001E001E001E00 1E001E001E001E001E001E201FF00FF007C00C1A7F9910>II< F003C0F003C07803807807807807803C0F003C0F003C0F001E0E001E1E001E1E000F1C00 0F3C000F3C0007380007380007B80003F00003F00001E00012147F9315>II<7801E07C03C03E07801E0F000F0F00079E0003FC0003F80001F8 0000F00001F00001F80003FC00079E000F0F000E0F001E07803C03C07801E0F801F01414 809315>II<7FFF7FFF7FFF003E00 3C007800F800F001E003E007C007800F001F001E003C007C00FFFFFFFFFFFF10147F9314 >I E /Fk 73 126 df<030600078F00078F00078F00078F00078F00078F007FFFC0FFFF E0FFFFE07FFFC00F1E000F1E000F1E000F1E000F1E000F1E007FFFC0FFFFE0FFFFE07FFF C01E3C001E3C001E3C001E3C001E3C001E3C000C1800131C7E9B18>35 D<01C00007E0000FF0000E70001C38001C38001C38001C38001C73F01C73F01CE3F00FE3 800FC7000F87000F07001F0E003F0E007B8E0073DC00E1DC00E0F800E0F800E07070E078 7070FC707FFFE03FCFE00F03C0141C7F9B18>38 D<007000F001E003C007800F001E001C 00380038007000700070007000E000E000E000E000E000E000E000E00070007000700070 00380038001C001E000F00078003C001F000F000700C24799F18>40 D<6000F00078003C001E000F000780038001C001C000E000E000E000E000700070007000 70007000700070007000E000E000E000E001C001C0038007800F001E003C007800F00060 000C247C9F18>I<01C00001C00001C00001C000C1C180F1C780F9CF807FFF001FFC0007 F00007F0001FFC007FFF00F9CF80F1C780C1C18001C00001C00001C00001C00011147D97 18>I<1C3E7E7F3F1F070E1E7CF860080C788518>44 D<3078FCFC78300606778518>46 D<000300000780000780000F80000F00001F00001E00001E00003E00003C00007C000078 0000780000F80000F00001F00001E00003E00003C00003C00007C0000780000F80000F00 000F00001F00001E00003E00003C00003C00007C0000780000F80000F00000F000006000 0011247D9F18>I<01800380038007800F803F80FF80FB80438003800380038003800380 038003800380038003800380038003800380038003807FFCFFFE7FFC0F1C7B9B18>49 D<03F0000FFE003FFF007C0F807003C0E001C0F000E0F000E06000E00000E00000E00001 C00001C00003C0000780000F00001E00003C0000780000F00001E00007C0000F80001E00 E03C00E07FFFE0FFFFE07FFFE0131C7E9B18>I<07F8001FFE003FFF007807807803C078 01C03001C00001C00003C0000380000F0003FF0003FE0003FF000007800003C00001C000 00E00000E00000E0F000E0F000E0F001C0F003C07C07803FFF001FFE0003F800131C7E9B 18>I<001F00003F0000770000770000E70001E70001C7000387000787000707000E0700 1E07003C0700380700780700F00700FFFFF8FFFFF8FFFFF8000700000700000700000700 000700000700007FF000FFF8007FF0151C7F9B18>I<03F8000FFE001FFF003E0F803803 807001C07001C07001C07001C03803803C07801FFF0007FC000FFE001F1F003C07807001 C0F001E0E000E0E000E0E000E0E000E07001C07803C03E0F801FFF000FFE0003F800131C 7E9B18>56 D<3078FCFC783000000000000000003078FCFC78300614779318>58 D<183C7E7E3C180000000000000000183C7E7E3E1E0E1C3C78F060071A789318>I<7FFF C0FFFFE0FFFFE0FFFFE0000000000000000000000000FFFFE0FFFFE0FFFFE07FFFC0130C 7E9318>61 D<0FF0003FFC007FFF00700F00F00380F00380600780000F00003E00007C00 01F00001E00003C00003C00003C00003C00003C000038000000000000000000000000000 00000003800007C00007C00007C000038000111C7D9B18>63 D<00700000F80000F80000 D80000D80001DC0001DC0001DC00018C00038E00038E00038E00038E0003060007070007 07000707000707000FFF800FFF800FFF800E03800E03801C01C01C01C07F07F0FF8FF87F 07F0151C7F9B18>65 DI<00F8E003FEE007 FFE00F07E01E03E03C01E03800E07000E07000E0700000E00000E00000E00000E00000E0 0000E00000E00000E000007000007000E07000E03800E03C00E01E01C00F07C007FF8003 FE0000F800131C7E9B18>I<7FF800FFFE007FFF001C0F801C03C01C03C01C01E01C00E0 1C00E01C00F01C00701C00701C00701C00701C00701C00701C00701C00701C00F01C00E0 1C00E01C01E01C01C01C03C01C0F807FFF00FFFE007FF800141C7F9B18>III<01F1C0 03FDC00FFFC01F0FC01C03C03803C03801C07001C07001C0700000E00000E00000E00000 E00000E00000E00FF0E01FF0E00FF07001C07001C07003C03803C03803C01C07C01F0FC0 0FFFC003FDC001F1C0141C7E9B18>I<7F07F0FF8FF87F07F01C01C01C01C01C01C01C01 C01C01C01C01C01C01C01C01C01C01C01FFFC01FFFC01FFFC01C01C01C01C01C01C01C01 C01C01C01C01C01C01C01C01C01C01C01C01C07F07F0FF8FF87F07F0151C7F9B18>I<7F FF00FFFF807FFF0001C00001C00001C00001C00001C00001C00001C00001C00001C00001 C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001 C0007FFF00FFFF807FFF00111C7D9B18>I<7F07F0FF87F87F07F01C03C01C07801C0700 1C0E001C1E001C3C001C38001C70001CF0001DF0001DF0001FB8001FB8001F1C001E1C00 1C0E001C0E001C07001C07001C03801C03801C01C07F03F0FF87F87F03F0151C7F9B18> 75 D<7FE000FFE0007FE0000E00000E00000E00000E00000E00000E00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E00000E00700E00700E0070 0E00700E00707FFFF0FFFFF07FFFF0141C7F9B18>II<7E07F0FF0FF87F07F01D81C01D81C01D81C01DC1C01CC1C01CC1C01CE1C01C E1C01CE1C01C61C01C71C01C71C01C31C01C39C01C39C01C39C01C19C01C19C01C1DC01C 0DC01C0DC01C0DC07F07C0FF87C07F03C0151C7F9B18>I<0FF8003FFE007FFF00780F00 700700F00780E00380E00380E00380E00380E00380E00380E00380E00380E00380E00380 E00380E00380E00380E00380E00380E00380F00780700700780F007FFF003FFE000FF800 111C7D9B18>II<0FF8003FFE007FFF0078 0F00700700F00780E00380E00380E00380E00380E00380E00380E00380E00380E00380E0 0380E00380E00380E00380E00380E1E380E1E380F0E78070F700787F007FFF003FFE000F FC00001C00001E00000E00000F0000070000070011227D9B18>I<7FF800FFFE007FFF00 1C0F801C03801C03C01C01C01C01C01C01C01C03C01C03801C0F801FFF001FFE001FFE00 1C0F001C07001C03801C03801C03801C03801C03801C039C1C039C1C039C7F01F8FF81F8 7F00F0161C7F9B18>I<03F3801FFF803FFF807C0F80700780E00380E00380E00380E000 007000007800003F00001FF00007FE0000FF00000F800003C00001C00000E00000E06000 E0E000E0E001E0F001C0F80780FFFF80FFFE00E7F800131C7E9B18>I<7FFFF8FFFFF8FF FFF8E07038E07038E07038E0703800700000700000700000700000700000700000700000 700000700000700000700000700000700000700000700000700000700000700007FF0007 FF0007FF00151C7F9B18>IIII<7F8FE0 7F9FE07F8FE00E07000F0700070E00078E00039C0003DC0001F80001F80000F00000F000 00700000F00000F80001F80001DC00039E00038E00070F000707000E07800E03801E03C0 7F07F0FF8FF87F07F0151C7F9B18>II<3F FFE07FFFE07FFFE07001C07003C0700780700700000F00001E00001C00003C0000780000 700000F00001E00001C00003C0000780000700000F00001E00E01C00E03C00E07800E070 00E0FFFFE0FFFFE0FFFFE0131C7E9B18>II93 D<7FFF00FFFF80FFFF807FFF0011047D7F18>95 D<1FE0003FF8007FFC00781E00300E00 00070000070000FF0007FF001FFF007F0700780700E00700E00700E00700F00F00781F00 3FFFF01FFBF007E1F014147D9318>97 D<7E0000FE00007E00000E00000E00000E00000E 00000E00000E3E000EFF800FFFC00FC1E00F80E00F00700E00700E00380E00380E00380E 00380E00380E00380F00700F00700F80E00FC1E00FFFC00EFF80063E00151C809B18>I< 01FE0007FF001FFF803E0780380300700000700000E00000E00000E00000E00000E00000 E000007000007001C03801C03E03C01FFF8007FF0001FC0012147D9318>I<001F80003F 80001F8000038000038000038000038000038003E3800FFB801FFF803C1F80380F807007 80700380E00380E00380E00380E00380E00380E00380700780700780380F803C1F801FFF F00FFBF803E3F0151C7E9B18>I<01F00007FC001FFE003E0F00380780700380700380E0 01C0E001C0FFFFC0FFFFC0FFFFC0E000007000007001C03801C03E03C01FFF8007FF0001 FC0012147D9318>I<001F80007FC000FFE000E1E001C0C001C00001C00001C0007FFFC0 FFFFC0FFFFC001C00001C00001C00001C00001C00001C00001C00001C00001C00001C000 01C00001C00001C00001C0007FFF007FFF007FFF00131C7F9B18>I<01E1F007FFF80FFF F81E1E301C0E003807003807003807003807003807001C0E001E1E001FFC001FF80039E0 003800001C00001FFE001FFFC03FFFE07801F0700070E00038E00038E00038E000387800 F07E03F01FFFC00FFF8001FC00151F7F9318>I<7E0000FE00007E00000E00000E00000E 00000E00000E00000E3E000EFF800FFFC00FC1C00F80E00F00E00E00E00E00E00E00E00E 00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E07FC3FCFFE7FE7FC3FC171C809B 18>I<03800007C00007C00007C0000380000000000000000000000000007FC000FFC000 7FC00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C000 01C00001C00001C000FFFF00FFFF80FFFF00111D7C9C18>I<0038007C007C007C003800 000000000000000FFC1FFC0FFC001C001C001C001C001C001C001C001C001C001C001C00 1C001C001C001C001C001C001C001C001C001C001C6038F078FFF07FE03F800E277E9C18 >II<7FE000FFE0007FE00000E00000E000 00E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E000 00E00000E00000E00000E00000E00000E00000E00000E0007FFFC0FFFFE07FFFC0131C7E 9B18>I<7CE0E000FFFBF8007FFFF8001F1F1C001E1E1C001E1E1C001C1C1C001C1C1C00 1C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C00 7F1F1F00FFBFBF807F1F1F001914819318>I<7E3E00FEFF807FFFC00FC1C00F80E00F00 E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E07FC3 FCFFE7FE7FC3FC1714809318>I<01F0000FFE001FFF003E0F803803807001C07001C0E0 00E0E000E0E000E0E000E0E000E0F001E07001C07803C03C07803E0F801FFF000FFE0001 F00013147E9318>I<7E3E00FEFF807FFFC00FC1E00F80E00F00700E00700E00380E0038 0E00380E00380E00380E00380F00700F00700F80E00FC1E00FFFC00EFF800E3E000E0000 0E00000E00000E00000E00000E00000E00007FC000FFE0007FC000151E809318>I<01E3 8007FB801FFF803E1F80380F80700780700780E00380E00380E00380E00380E00380E003 80700780700780380F803C1F801FFF800FFB8003E3800003800003800003800003800003 80000380000380003FF8003FF8003FF8151E7E9318>I<7F87E0FF9FF07FBFF803F87803 F03003E00003C00003C00003800003800003800003800003800003800003800003800003 80007FFE00FFFF007FFE0015147F9318>I<07F7003FFF007FFF00780F00E00700E00700 E007007C00007FE0001FFC0003FE00001F00600780E00380E00380F00380F80F00FFFF00 FFFC00E7F00011147D9318>I<0180000380000380000380000380007FFFC0FFFFC0FFFF C00380000380000380000380000380000380000380000380000380000380400380E00380 E00380E001C1C001FFC000FF80003E0013197F9818>I<7E07E0FE0FE07E07E00E00E00E 00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E01E00F 03E007FFFC03FFFE01FCFC1714809318>I<7F8FF0FF8FF87F8FF01E03C00E03800E0380 0E0380070700070700070700038E00038E00038E00038E0001DC0001DC0001DC0000F800 00F80000700015147F9318>II<7F8FF07F9FF07F8FF0070700078E00039E0001DC0001F80000F80000 700000F00000F80001DC00039E00038E000707000F07807F8FF0FF8FF87F8FF015147F93 18>I<7F8FF0FF8FF87F8FF00E01C00E03800E0380070380070700070700038700038600 038E0001CE0001CE0000CC0000CC0000DC00007800007800007800007000007000007000 00F00000E00079E0007BC0007F80003F00001E0000151E7F9318>I<3FFFF07FFFF07FFF F07001E07003C0700780000F00001E00003C0000F80001F00003C0000780000F00701E00 703C0070780070FFFFF0FFFFF0FFFFF014147F9318>I<0007E0001FE0007FE000780000 E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00001E0007F C000FF8000FF80007FC00001E00000E00000E00000E00000E00000E00000E00000E00000 E00000E00000E00000E000007800007FE0001FE00007E013247E9F18>I<7C0000FF0000 FFC00003C00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E000 00E00000F000007FC0003FE0003FE0007FC000F00000E00000E00000E00000E00000E000 00E00000E00000E00000E00000E00000E00003C000FFC000FF00007C000013247E9F18> 125 D E /Fl 70 123 df<007E1F0001C1B1800303E3C00703C3C00E03C1800E01C0000E 01C0000E01C0000E01C0000E01C0000E01C000FFFFFC000E01C0000E01C0000E01C0000E 01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E 01C0000E01C0000E01C0000E01C0007F87FC001A1D809C18>11 D<007E0001C180030180 0703C00E03C00E01800E00000E00000E00000E00000E0000FFFFC00E01C00E01C00E01C0 0E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0 0E01C07F87F8151D809C17>I<007FC001C1C00303C00703C00E01C00E01C00E01C00E01 C00E01C00E01C00E01C0FFFFC00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01 C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C07FCFF8151D809C17>I<60 60F0F0F8F86868080808080808101010102020404080800D0C7F9C15>34 D<00E0000001900000030800000308000007080000070800000708000007080000071000 0007100000072000000740000003C03FE003800F00038006000380040005C0040009C008 0010E0100030E010006070200060702000E0384000E03C4000E01C8000E00F0020E00700 20700780403009C0401830E18007C03E001B1F7E9D20>38 D<004000800100020006000C 000C0018001800300030007000600060006000E000E000E000E000E000E000E000E000E0 00E000E000E000600060006000700030003000180018000C000C00060002000100008000 400A2A7D9E10>40 D<800040002000100018000C000C0006000600030003000380018001 80018001C001C001C001C001C001C001C001C001C001C001C001C0018001800180038003 000300060006000C000C00180010002000400080000A2A7E9E10>I<60F0F07010101010 20204080040C7C830C>44 DI<60F0F06004047C830C>I<03C00C 301818300C300C700E60066006E007E007E007E007E007E007E007E007E007E007E007E0 07E00760066006700E300C300C18180C3007E0101D7E9B15>48 D<030007003F00C70007 000700070007000700070007000700070007000700070007000700070007000700070007 000700070007000F80FFF80D1C7C9B15>I<07C01830201C400C400EF00FF80FF807F807 7007000F000E000E001C001C00380070006000C00180030006010C01180110023FFE7FFE FFFE101C7E9B15>I<07E01830201C201C781E780E781E381E001C001C00180030006007 E00030001C001C000E000F000F700FF80FF80FF80FF00E401C201C183007E0101D7E9B15 >I<000C00000C00001C00003C00003C00005C0000DC00009C00011C00031C00021C0004 1C000C1C00081C00101C00301C00201C00401C00C01C00FFFFC0001C00001C00001C0000 1C00001C00001C00001C0001FFC0121C7F9B15>I<300C3FF83FF03FC020002000200020 002000200023E024302818301C200E000E000F000F000F600FF00FF00FF00F800E401E40 1C2038187007C0101D7E9B15>I<00F0030C06040C0E181E301E300C700070006000E3E0 E430E818F00CF00EE006E007E007E007E007E007600760077006300E300C18180C3003E0 101D7E9B15>I<4000007FFF807FFF007FFF004002008004008004008008000010000010 0000200000600000400000C00000C00001C0000180000180000380000380000380000380 00078000078000078000078000078000078000030000111D7E9B15>I<03E00C30100820 0C20066006600660067006780C3E083FB01FE007F007F818FC307E601E600FC007C003C0 03C003C00360026004300C1C1007E0101D7E9B15>I<03C00C301818300C700C600EE006 E006E007E007E007E007E0076007700F300F18170C2707C700060006000E300C780C7818 7010203030C00F80101D7E9B15>I<60F0F0600000000000000000000060F0F06004127C 910C>I<60F0F0600000000000000000000060F0F0701010101020204080041A7C910C>I< 7FFFFFC0FFFFFFE000000000000000000000000000000000000000000000000000000000 00000000FFFFFFE07FFFFFC01B0C7E8F20>61 D<0FE03038401CE00EF00EF00EF00E000C 001C0030006000C000800180010001000100010001000100000000000000000000000300 0780078003000F1D7E9C14>63 D<000600000006000000060000000F0000000F0000000F 00000017800000178000001780000023C0000023C0000023C0000041E0000041E0000041 E0000080F0000080F0000180F8000100780001FFF80003007C0002003C0002003C000600 3E0004001E0004001E000C001F001E001F00FF80FFF01C1D7F9C1F>65 DI<001F808000E061800180198007000780 0E0003801C0003801C00018038000180780000807800008070000080F0000000F0000000 F0000000F0000000F0000000F0000000F0000000F0000000700000807800008078000080 380000801C0001001C0001000E000200070004000180080000E03000001FC000191E7E9C 1E>IIII<001F808000E0618001801980070007800E000380 1C0003801C00018038000180780000807800008070000080F0000000F0000000F0000000 F0000000F0000000F0000000F000FFF0F0000F8070000780780007807800078038000780 1C0007801C0007800E00078007000B800180118000E06080001F80001C1E7E9C21>II I<1FFF00F800780078007800780078007800780078007800780078007800780078007800 780078007800787078F878F878F878F0F040E021C01F00101D7F9B15>I76 DII<003F800000E0E000 0380380007001C000E000E001C0007003C00078038000380780003C0780003C0700001C0 F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0700001C0 780003C0780003C0380003803C0007801C0007000E000E0007001C000380380000E0E000 003F80001B1E7E9C20>II82 D<07E0801C1980300580700380600180E00180E00080E00080E000 80F00000F800007C00007FC0003FF8001FFE0007FF0000FF80000F800007C00003C00001 C08001C08001C08001C0C00180C00180E00300D00200CC0C0083F800121E7E9C17>I<7F FFFFC0700F01C0600F00C0400F0040400F0040C00F0020800F0020800F0020800F002000 0F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F000000 0F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000001F800003 FFFC001B1C7F9B1E>II<0808101020204040404080 8080808080B0B0F8F8787830300D0C7A9C15>92 D<1FC000307000783800781C00301C00 001C00001C0001FC000F1C00381C00701C00601C00E01C40E01C40E01C40603C40304E80 1F870012127E9115>97 DI<07E00C 301878307870306000E000E000E000E000E000E00060007004300418080C3007C00E127E 9112>I<003F000007000007000007000007000007000007000007000007000007000007 0003E7000C1700180F00300700700700600700E00700E00700E00700E00700E00700E007 00600700700700300700180F000C370007C7E0131D7E9C17>I<03E00C301818300C700E 6006E006FFFEE000E000E000E00060007002300218040C1803E00F127F9112>I<00F801 8C071E061E0E0C0E000E000E000E000E000E00FFE00E000E000E000E000E000E000E000E 000E000E000E000E000E000E000E000E007FE00F1D809C0D>I<00038003C4C00C38C01C 3880181800381C00381C00381C00381C001818001C38000C300013C00010000030000018 00001FF8001FFF001FFF803003806001C0C000C0C000C0C000C06001803003001C0E0007 F800121C7F9215>II<18003C003C 0018000000000000000000000000000000FC001C001C001C001C001C001C001C001C001C 001C001C001C001C001C001C001C00FF80091D7F9C0C>I<00C001E001E000C000000000 000000000000000000000FE000E000E000E000E000E000E000E000E000E000E000E000E0 00E000E000E000E000E000E000E000E060E0F0C0F1C061803E000B25839C0D>III II<03F0000E1C0018060030 0300700380600180E001C0E001C0E001C0E001C0E001C0E001C060018070038030030018 06000E1C0003F00012127F9115>II<03C1000C3300180B 00300F00700700700700E00700E00700E00700E00700E00700E00700600700700700300F 00180F000C370007C700000700000700000700000700000700000700000700003FE0131A 7E9116>II<1F9030704030C010C010E010F8007F803FE00FF000F88038 8018C018C018E010D0608FC00D127F9110>I<04000400040004000C000C001C003C00FF E01C001C001C001C001C001C001C001C001C001C101C101C101C101C100C100E2003C00C 1A7F9910>IIII<7F8FF00F03800F0300 07020003840001C80001D80000F00000700000780000F800009C00010E00020E00060700 0403801E07C0FF0FF81512809116>II<7FFC7038603840 7040F040E041C003C0038007000F040E041C043C0C380870087038FFF80E127F9112>I E /Fm 17 119 df<78FCFCFCFC7800000000000078FCFCFCFC7806127D910D>58 D<00038000000380000007C0000007C0000007C000000FE000000FE000001FF000001BF0 00001BF0000031F8000031F8000061FC000060FC0000E0FE0000C07E0000C07E0001803F 0001FFFF0003FFFF8003001F8003001F8006000FC006000FC00E000FE00C0007E0FFC07F FEFFC07FFE1F1C7E9B24>65 D<001FE02000FFF8E003F80FE007C003E00F8001E01F0000 E03E0000E03E0000607E0000607C000060FC000000FC000000FC000000FC000000FC0000 00FC000000FC000000FC0000007C0000607E0000603E0000603E0000C01F0000C00F8001 8007C0030003F80E0000FFFC00001FE0001B1C7D9B22>67 DI< 07F8201FFEE03C07E07801E07000E0F000E0F00060F00060F80000FE0000FFE0007FFE00 3FFF003FFF800FFFC007FFE0007FE00003F00001F00000F0C000F0C000F0C000E0E000E0 F001C0FC03C0EFFF0083FC00141C7D9B1B>83 D<0FF8001C1E003E0F803E07803E07C01C 07C00007C0007FC007E7C01F07C03C07C07C07C0F807C0F807C0F807C0780BC03E13F80F E1F815127F9117>97 D<03FC000E0E001C1F003C1F00781F00780E00F80000F80000F800 00F80000F80000F800007800007801803C01801C03000E0E0003F80011127E9115>99 D<01FC000F07001C03803C01C07801C07801E0F801E0F801E0FFFFE0F80000F80000F800 007800007C00603C00601E00C00F038001FC0013127F9116>101 D<1E003F003F003F003F001E00000000000000000000000000FF00FF001F001F001F001F 001F001F001F001F001F001F001F001F001F001F00FFE0FFE00B1E7F9D0E>105 D108 D110 D<01FC000F07801C01C03C01E07800F07800F0F800F8F800F8F800F8F800F8F800F8F800 F87800F07800F03C01E01E03C00F078001FC0015127F9118>I114 D<1FD830786018E018E018F000FF807FE07FF01F F807FC007CC01CC01CE01CE018F830CFC00E127E9113>I<030003000300030007000700 0F000F003FFCFFFC1F001F001F001F001F001F001F001F001F001F0C1F0C1F0C1F0C0F08 079803F00E1A7F9913>II< FFC1FCFFC1FC1F00601F80E00F80C00FC0C007C18007C18003E30003E30001F60001F600 01FE0000FC0000FC0000780000780000300016127F9119>I E /Fn 53 123 df<00000F80003F8F80007F8F8000FF8F8001FF8F8003E0000003C0000007C000 0007C0000007C0000007C0000007C0000007C0000007C0000007C0000007C00000FFFF8F 80FFFF8F80FFFF8F8007C00F8007C00F8007C00F8007C00F8007C00F8007C00F8007C00F 8007C00F8007C00F8007C00F8007C00F8007C00F8007C00F8007C00F8007C00F8007C00F 8007C00F8007C00F8007C00F8007C00F8007C00F8007C00F8007C00F8007C00F80192B7F AA20>12 D45 DI<007F000001 FFC00007FFF0000FFFF8000FC1F8001F007C003F007E003E003E003C001E007C001F007C 001F007C001F0078000F00F8000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8 000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8000F8078 000F007C001F007C001F007C001F003E003E003E003E003F007E001F80FC000FC1F8000F FFF80007FFF00001FFC000007F000019297EA71E>48 D<00180000380000F80007F800FF F800FFF800F8F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000 F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000 F80000F80000F80000F80000F80000F80000F80000F80000F8007FFFF07FFFF07FFFF014 287BA71E>I<00FE0003FFC007FFE00FFFF01F03F83C00FC38007E78003E70003EF0001F F0001F60001F20001F00001F00001F00001F00003E00003E00007C00007C0000F80001F0 0001E00003C0000780000F00001E00003C0000780000F00001E00003C0000780000F0000 1E00003C00007FFFFF7FFFFF7FFFFF7FFFFF18287EA71E>I<007F000001FFC00007FFF0 000FFFF8001FC1F8003E007C003C003E0078003E0038003E0010003E0000003E0000003E 0000003C0000007C000000FC000001F8000007F00000FFE00000FFC00000FFE00000FFF0 000001FC0000007C0000003E0000001F0000001F0000000F8000000F8000000F8000000F 8000000F8040000F8060001F00F0001F00F8003F007E007E003F81FC001FFFF8000FFFF0 0003FFE000007F000019297EA71E>I<0003F0000007F0000005F000000DF000000DF000 001DF0000039F0000039F0000079F0000079F00000F1F00000F1F00001E1F00003E1F000 03E1F00007C1F00007C1F0000F81F0000F81F0001F01F0001F01F0003E01F0007C01F000 7C01F000F801F000FFFFFF80FFFFFF80FFFFFF80FFFFFF800001F0000001F0000001F000 0001F0000001F0000001F0000001F0000001F0000001F0000001F00019277EA61E>I<3F FFFC3FFFFC3FFFFC3FFFFC3E00003E00003E00003E00003E00003E00003E00003E00003E 00003E00003E3F003EFFC03FFFE03FFFF03FE1F83F807C3F003E3E003E00003E00001F00 001F00001F00001F00001F00001F00001F20001F60003E70003EF8007C7C00FC3F03F81F FFF00FFFE007FF8000FE0018287EA61E>I<000FF000003FFC0000FFFC0001FFFC0003F8 0C0007E000000FC000000F8000001F0000001E0000003E0000003C0000007C0000007C00 00007C3FE000F8FFF000F9FFF800FBFFFC00FF807E00FF003E00FE003F00FC001F00FC00 1F00FC000F80F8000F80F8000F80F8000F80F8000F8078000F807C000F807C000F807C00 0F003E001F003E001F001F003E001F807C000FC1FC0007FFF80003FFF00001FFC000007F 000019297EA71E>II<007F000001FFC000 07FFF0000FFFF8001FC1FC003F007E003E003E007E003F007C001F007C001F007C001F00 7C001F007C001F003E003E003E003E001F007C000FC1F80007FFF00003FFE00003FFE000 0FFFF8001FC1FC003F007E003E003E007C001F007C001F00F8000F80F8000F80F8000F80 F8000F80F8000F80F8000F807C001F007C001F007E003F003F007E001FC1FC000FFFF800 07FFF00003FFE000007F000019297EA71E>I<007F000001FFC00003FFE0000FFFF0000F C1F8001F007C003E007C007C003E007C001E007C001F00F8001F00F8001F00F8000F00F8 000F80F8000F80F8000F80F8000F80F8001F807C001F807C001F807E003F803E007F803F 00FF801FFFEF800FFFCF8007FF8F8003FE1F0000001F0000001F0000001E0000003E0000 003E0000007C0000007C000000F8001801F0001E07E0003FFFC0001FFF80000FFE000003 F8000019297EA71E>I<0001F000000003F800000003F800000007FC00000007BC000000 07BC0000000F3E0000000F1E0000000F1E0000001F1F0000001E1F0000001E0F0000003E 0F8000003C0F8000003C078000007C07C000007807C00000F803E00000F803E00000F003 E00001F001F00001F001F00001E001F00003E000F80003E000F80003C000F80007FFFFFC 0007FFFFFC000FFFFFFE000F80003E000F80003E001F00003F001F00001F001E00001F00 3E00000F803E00000F803C00000F807C000007C07C000007C078000007C0F8000003E0F8 000003E0232A7EA928>65 D<0001FF00000FFFE0003FFFF8007FFFF800FE01F801F80030 03F0001007C000000F8000001F8000001F0000003E0000003E0000007E0000007C000000 7C0000007C000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000 F8000000F8000000F80000007C0000007C0000007C0000007E0000003E0000003E000000 1F0000001F8000000F80000007C0000003F0000401F8001C00FE00FC007FFFFC003FFFF8 000FFFE00001FF001E2C7CAA26>67 DIII72 DI75 D77 DI<0001FC0000000FFF8000003FFFE000 007FFFF00001FE03FC0003F800FE0007E0003F0007C0001F000F80000F801F000007C01F 000007C03E000003E03E000003E07C000001F07C000001F07C000001F078000000F0F800 0000F8F8000000F8F8000000F8F8000000F8F8000000F8F8000000F8F8000000F8F80000 00F8F8000000F8F8000000F87C000001F07C000001F07C000001F07E000003F03E000003 E03F000007E01F000007C01F80000FC00FC0001F8007E0003F0007F0007F0003F800FE00 01FE03FC0000FFFFF800003FFFE000000FFF80000001FC0000252C7DAA2C>II82 D<007FC00001FFF80007FFFE000F FFFF001FC07F003F000F007E0006007C0000007C000000F8000000F8000000F8000000F8 000000F8000000FC0000007E0000007F0000003F8000001FF800000FFF800007FFE00003 FFF80000FFFC00000FFE000000FF0000003F0000001F8000000F8000000FC0000007C000 0007C0000007C0000007C0000007C0000007C000000F8060000F80F0001F00FC003F00FF 80FE007FFFFC001FFFF80007FFE00000FF80001A2C7DAA21>II86 DI<01FE000FFF803FFFC03FFFE03C03F03001F00001F800 00F80000F80000F80000F80000F8007FF807FFF81FFFF83FE0F87F00F8FC00F8F800F8F8 00F8F800F8FC01F87E07F87FFFF83FFFF81FFCF80FE0F8151B7E9A1D>97 DI<007FC001FFF007FFFC0F FFFC1FC07C1F00083E00007C00007C00007C0000F80000F80000F80000F80000F80000F8 0000F800007C00007C00007E00003E00001F000C1FC07C0FFFFC07FFFC01FFF0007F8016 1B7E9A1B>I<00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E 00003E00003E00003E00003E00003E00FC3E03FF3E07FFFE0FFFFE1FC1FE3F007E3E003E 7C003E7C003EFC003EF8003EF8003EF8003EF8003EF8003EF8003EF8003EFC003E7C003E 7C003E3E007E3F00FE1FC1FE0FFFFE07FFBE03FF3E00FC3E172A7EA91F>I<007E0003FF 8007FFC00FFFE01F83F03F00F03E00787C00787C003878003CFFFFFCFFFFFCFFFFFCFFFF FCF80000F80000F800007800007C00007C00003E00003F000C1FC07C0FFFFC07FFFC01FF F0007F80161B7E9A1B>I<001FC0007FC000FFC001FFC003F00003E00007C00007C00007 C00007C00007C00007C00007C00007C00007C000FFFE00FFFE00FFFE0007C00007C00007 C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007 C00007C00007C00007C00007C00007C00007C00007C00007C00007C000122A7FA912>I< 00F8078003FE7FC00FFFFFC01FFFFFC01F07C0003E03E0003E03E0007C01F0007C01F000 7C01F0007C01F0007C01F0007C01F0003E03E0003E03E0001F07C0001FFFC0003FFF8000 3BFE000038F8000078000000780000003C0000003FFFC0003FFFF8001FFFFC001FFFFE00 3FFFFF007C007F00F8001F80F8000F80F8000F80F8000F80FC001F807E003F003F80FE00 3FFFFE000FFFF80007FFF00000FF80001A287E9A1E>III108 DII<007F000001FFC00007FFF0000FFFF8001FC1 FC003F007E003E003E007C001F007C001F0078000F00F8000F80F8000F80F8000F80F800 0F80F8000F80F8000F80F8000F807C001F007C001F007E003F003E003E003F007E001FC1 FC000FFFF80007FFF00001FFC000007F0000191B7E9A1E>II114 D<03FC001FFF803FFFC07FFFC07C07C0F80080F80000F80000F80000FC00007F80007FF8 003FFE001FFF0007FF8000FFC0000FE00007E00003E00003E04003E0E007E0FC0FC0FFFF C07FFF801FFE0003F800131B7E9A17>I<07C00007C00007C00007C00007C00007C00007 C000FFFFC0FFFFC0FFFFC007C00007C00007C00007C00007C00007C00007C00007C00007 C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C04007E1C003 FFE003FFE001FF8000FC0013227FA116>IIII<7C000FC03E001F803F001F001F803E000F807C0007C0FC0003E0F80001F1F00001 FBE00000FFC000007FC000003F8000001F0000001F0000003F8000007FC00000FBC00000 F3E00001F1F00003E0F80007C07C000F807C000F803E001F001F003E000F807E000FC0FC 0007E01B1B809A1C>III E /Fo 15 122 df<07E00FF03FFC3FFC7FFEFFFFFFFFFFFFFFFFFF FFFFFF7FFE3FFC3FFC0FF007E01010788F21>46 D<000001E00000000003F0000000000F F0000000003FF000000000FFF00000000FFFF0000003FFFFF00000FFFFFFF00000FFFFFF F00000FFFFFFF00000FFF0FFF00000FC00FFF000000000FFF000000000FFF000000000FF F000000000FFF000000000FFF000000000FFF000000000FFF000000000FFF000000000FF F000000000FFF000000000FFF000000000FFF000000000FFF000000000FFF000000000FF F000000000FFF000000000FFF000000000FFF000000000FFF000000000FFF000000000FF F000000000FFF000000000FFF000000000FFF000000000FFF000000000FFF000000000FF F000000000FFF000000000FFF000000000FFF000000000FFF000000000FFF000000000FF F000000000FFF000000000FFF000000000FFF000000000FFF000000000FFF000000000FF F000000000FFF000000000FFF000000000FFF000000000FFF000000000FFF000000000FF F000000000FFF000000000FFF000000000FFF000000000FFF000000000FFF000000000FF F000003FFFFFFFFFC03FFFFFFFFFC03FFFFFFFFFC03FFFFFFFFFC03FFFFFFFFFC02A4478 C33B>49 D<0000FFE00000000FFFFE0000003FFFFFC00000FFFFFFF00003FFFFFFFC0007 FC01FFFE000FE0007FFF001FC0001FFF803F80000FFFC03FE00007FFE07FF80007FFE07F FC0003FFF0FFFC0003FFF0FFFE0001FFF8FFFE0001FFF8FFFE0001FFF8FFFE0000FFFCFF FE0000FFFC7FFC0000FFFC7FFC0000FFFC3FF80000FFFC1FF00000FFFC0FE00000FFFC00 000001FFFC00000001FFF800000001FFF800000001FFF000000003FFF000000003FFE000 000003FFE000000007FFC000000007FF800000000FFF800000001FFF000000001FFE0000 00003FFC000000007FF8000000007FE000000000FFC000000001FF8000000003FF000000 0007FC000000000FF8000000001FF0000000003FC0000000007F80007C0000FF00007C00 00FE00007C0001FC00007C0003F00000F80007E00000F8000FC00000F8001F800000F800 3F000001F8007C000003F800FFFFFFFFF801FFFFFFFFF003FFFFFFFFF007FFFFFFFFF00F FFFFFFFFF01FFFFFFFFFF03FFFFFFFFFF07FFFFFFFFFF07FFFFFFFFFF0FFFFFFFFFFE0FF FFFFFFFFE0FFFFFFFFFFE0FFFFFFFFFFE02E447AC33B>I77 D<0007FFFC000000007FFFFFC0000001FFFF FFF8000003FFFFFFFE000007FE001FFF000007FF0003FFC0000FFF8001FFE0000FFF8000 FFF0000FFF80007FF0000FFF80007FF8000FFF80007FF80007FF00003FFC0007FF00003F FC0003FE00003FFC0000F800003FFC00000000003FFC00000000003FFC00000000003FFC 00000000003FFC00000007FFFFFC000000FFFFFFFC000007FFFFFFFC00003FFFE03FFC00 00FFFE003FFC0003FFF0003FFC0007FFC0003FFC000FFF00003FFC001FFE00003FFC003F FC00003FFC007FF800003FFC007FF800003FFC00FFF000003FFC00FFF000003FFC00FFF0 00003FFC00FFF000003FFC00FFF000003FFC00FFF000007FFC007FF80000FFFC007FF800 01EFFC003FFC0003EFFC003FFF0007CFFF000FFFC03F8FFFF807FFFFFF07FFFC01FFFFFC 03FFFC007FFFF001FFFC0003FF80007FF8362E7DAD3A>97 D<00001FFFC0000000FFFFF8 000007FFFFFE00001FFFFFFF80007FFC00FFC000FFE001FFC001FFC003FFE003FF8003FF E007FF0003FFE00FFE0003FFE00FFE0003FFE01FFC0001FFC01FFC0001FFC03FFC0000FF 803FFC00003E007FF8000000007FF8000000007FF800000000FFF800000000FFF8000000 00FFF800000000FFF800000000FFF800000000FFF800000000FFF800000000FFF8000000 00FFF800000000FFF8000000007FF8000000007FF8000000007FFC000000003FFC000000 003FFC000000001FFC000000F81FFE000000F80FFE000000F80FFF000001F007FF800003 F003FFC00007E001FFE0000FC000FFF0001F80007FFE00FF00001FFFFFFE000007FFFFF8 000000FFFFE00000001FFE00002D2E7CAD35>99 D<00001FFE00000001FFFFE0000007FF FFF800001FFFFFFE00007FFC07FF0000FFE001FF8001FFC0007FC003FF80003FE007FF00 003FF00FFE00001FF01FFE00000FF81FFC00000FF83FFC00000FFC3FFC000007FC7FFC00 0007FC7FF8000007FC7FF8000007FE7FF8000007FEFFF8000007FEFFF8000007FEFFFFFF FFFFFEFFFFFFFFFFFEFFFFFFFFFFFEFFFFFFFFFFFCFFF800000000FFF800000000FFF800 000000FFF8000000007FF8000000007FF8000000007FFC000000003FFC000000003FFC00 0000003FFC0000001C1FFE0000003E0FFE0000003E07FF0000007E07FF000000FC03FF80 0001F801FFC00003F0007FF0001FE0003FFE00FFC0001FFFFFFF800007FFFFFE000000FF FFF80000000FFF80002F2E7DAD36>101 D<000000FFC000000007FFF80000003FFFFC00 0000FFFFFF000001FFC1FF000007FF03FF80000FFC03FF80000FF807FFC0001FF807FFC0 003FF007FFC0003FF007FFC0003FE003FF80007FE003FF80007FE001FF00007FE000FE00 007FE0003800007FE0000000007FE0000000007FE0000000007FE0000000007FE0000000 007FE0000000007FE0000000007FE0000000007FE0000000007FE0000000FFFFFFFE0000 FFFFFFFE0000FFFFFFFE0000FFFFFFFE0000FFFFFFFE0000007FF0000000007FF0000000 007FF0000000007FF0000000007FF0000000007FF0000000007FF0000000007FF0000000 007FF0000000007FF0000000007FF0000000007FF0000000007FF0000000007FF0000000 007FF0000000007FF0000000007FF0000000007FF0000000007FF0000000007FF0000000 007FF0000000007FF0000000007FF0000000007FF0000000007FF0000000007FF0000000 007FF0000000007FF0000000007FF0000000007FF0000000007FF0000000007FF0000000 007FF0000000007FF0000000007FF0000000007FF00000003FFFFFF800003FFFFFF80000 3FFFFFF800003FFFFFF800003FFFFFF800002A487DC724>I<00FC0001FE0003FF0007FF 800FFFC01FFFE01FFFE01FFFE01FFFE01FFFE01FFFE00FFFC007FF8003FF0001FE0000FC 00000000000000000000000000000000000000000000000000000000000000000000007F C0FFFFC0FFFFC0FFFFC0FFFFC0FFFFC003FFC001FFC001FFC001FFC001FFC001FFC001FF C001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FF C001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FF C001FFC001FFC001FFC001FFC0FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF18497CC820>105 D<007FC000FFFFC000FFFFC000FFFFC000FFFFC000FFFFC00003FFC00001FFC00001FFC0 0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 0001FFC00001FFC00001FFC00001FFC000FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF 8019487CC720>108 D<007FC001FFC00000FFFFC00FFFF80000FFFFC03FFFFE0000FFFF C0FFFFFF0000FFFFC1FC07FF8000FFFFC3E003FFC00003FFC7C001FFC00001FFCF0001FF E00001FFDE0000FFE00001FFDC0000FFE00001FFFC0000FFF00001FFF80000FFF00001FF F00000FFF00001FFF00000FFF00001FFF00000FFF00001FFE00000FFF00001FFE00000FF F00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FF E00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FF F00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FF E00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FF F00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FFE00000FFF00001FF E00000FFF00001FFE00000FFF000FFFFFFC07FFFFFE0FFFFFFC07FFFFFE0FFFFFFC07FFF FFE0FFFFFFC07FFFFFE0FFFFFFC07FFFFFE03B2E7CAD42>110 D<00000FFF0000000000 FFFFF000000007FFFFFE0000001FFFFFFF8000003FFC03FFC00000FFE0007FF00001FF80 001FF80003FF00000FFC0007FE000007FE000FFE000007FF000FFC000003FF001FFC0000 03FF803FFC000003FFC03FF8000001FFC03FF8000001FFC07FF8000001FFE07FF8000001 FFE07FF8000001FFE0FFF8000001FFF0FFF8000001FFF0FFF8000001FFF0FFF8000001FF F0FFF8000001FFF0FFF8000001FFF0FFF8000001FFF0FFF8000001FFF0FFF8000001FFF0 FFF8000001FFF07FF8000001FFE07FF8000001FFE07FF8000001FFE07FF8000001FFE03F FC000003FFC03FFC000003FFC01FFC000003FF801FFE000007FF800FFE000007FF0007FF 00000FFE0003FF80001FFC0001FFC0003FF80000FFE0007FF000007FFC03FFE000001FFF FFFF80000007FFFFFE00000000FFFFF0000000000FFF000000342E7DAD3B>I<00FF803F 8000FFFF80FFF000FFFF83FFFC00FFFF87FFFE00FFFF8FC3FF00FFFF8F07FF0003FF9E0F FF8001FFBC0FFF8001FFB80FFF8001FFF80FFF8001FFF00FFF8001FFF007FF0001FFF007 FF0001FFE003FE0001FFE000F80001FFE000000001FFE000000001FFC000000001FFC000 000001FFC000000001FFC000000001FFC000000001FFC000000001FFC000000001FFC000 000001FFC000000001FFC000000001FFC000000001FFC000000001FFC000000001FFC000 000001FFC000000001FFC000000001FFC000000001FFC000000001FFC000000001FFC000 000001FFC000000001FFC000000001FFC000000001FFC0000000FFFFFFE00000FFFFFFE0 0000FFFFFFE00000FFFFFFE00000FFFFFFE00000292E7CAD31>114 D<000FFF00E0007FFFF3E001FFFFFFE007FFFFFFE00FF800FFE01FC0001FE03F80000FE0 3F000007E07F000003E07F000003E0FF000003E0FF000003E0FF800003E0FFC0000000FF F0000000FFFE000000FFFFF800007FFFFFC0007FFFFFF0003FFFFFFC001FFFFFFF000FFF FFFF8007FFFFFFC003FFFFFFE000FFFFFFF0003FFFFFF00003FFFFF800001FFFF8000000 FFFC0000001FFC7800000FFCF8000007FCF8000003FCFC000003FCFC000003FCFE000003 F8FE000003F8FF000003F8FF800007F0FFC0000FF0FFF0001FE0FFFC00FFC0FFFFFFFF80 FC7FFFFE00F81FFFF800E003FF8000262E7CAD2F>I<7FFFFFC000FFFF807FFFFFC000FF FF807FFFFFC000FFFF807FFFFFC000FFFF807FFFFFC000FFFF8000FFF000000FE00000FF F800000FC00000FFF800000FC000007FFC00000F8000007FFC00001F8000003FFC00001F 0000003FFE00003F0000001FFE00003E0000001FFF00007E0000000FFF00007C0000000F FF8000FC00000007FF8000F800000007FFC001F800000003FFC001F000000003FFE003F0 00000003FFE003F000000001FFF003E000000001FFF007E000000000FFF007C000000000 FFF80FC0000000007FF80F80000000007FFC1F80000000003FFC1F00000000003FFE3F00 000000001FFE3E00000000001FFF7E00000000000FFF7C00000000000FFFFC0000000000 0FFFFC000000000007FFF8000000000007FFF8000000000003FFF0000000000003FFF000 0000000001FFE0000000000001FFE0000000000000FFC0000000000000FFC00000000000 007F800000000000007F800000000000003F000000000000003F000000000000003F0000 00000000003E000000000000007E000000000000007C00000000000000FC000000001F80 00F8000000003FC001F8000000007FE001F000000000FFF003F000000000FFF003E00000 0000FFF007E000000000FFF00FC000000000FFF01F8000000000FFF03F80000000007FE0 7F00000000007F43FE00000000003FFFF800000000001FFFF0000000000007FFC0000000 000001FE00000000000039427EAD3F>121 D E /Fp 8 117 df<0003FF8000001FFFF000 007FFFFE0000FE03FF0001F000FF8003C000FFC00780007FE00FF0007FF00FF8007FF01F FC007FF81FFE007FF81FFE007FF81FFE007FF81FFE007FF81FFE007FF80FFC007FF007F8 007FF003F0007FF0000000FFE0000000FFC0000001FF80000001FF00000003FE00000007 FC0000001FF000000FFFC000000FFF8000000FFFF800000003FE00000000FF800000007F E00000003FF00000003FF80000003FFC0000001FFC0000001FFE0000001FFE0200001FFF 1FC0001FFF3FE0001FFF7FF0001FFF7FF0001FFFFFF8001FFFFFF8001FFFFFF8001FFEFF F8001FFEFFF0001FFE7FF0003FFC7FE0003FFC3FC0003FF81F80007FF01FE000FFE007FC 03FFC003FFFFFF0001FFFFFE00003FFFF0000007FF800028397CB731>51 D<0000001FFF000030000001FFFFE000F000000FFFFFFC01F000007FFFFFFE03F00001FF FE007F87F00003FFE0000FCFF0000FFF000003FFF0001FFC000001FFF0003FF80000007F F0007FF00000003FF000FFC00000003FF001FFC00000001FF003FF800000000FF007FF00 0000000FF00FFF0000000007F00FFE0000000007F01FFE0000000003F01FFE0000000003 F03FFC0000000003F03FFC0000000001F03FFC0000000001F07FFC0000000001F07FF800 00000001F07FF80000000000007FF8000000000000FFF8000000000000FFF80000000000 00FFF8000000000000FFF8000000000000FFF8000000000000FFF8000000000000FFF800 0000000000FFF8000000000000FFF8000000000000FFF8000000000000FFF80000000000 007FF80000000000007FF80000000000007FF80000000000007FFC0000000000F03FFC00 00000000F03FFC0000000000F03FFC0000000000F01FFE0000000000F01FFE0000000001 E00FFE0000000001E00FFF0000000001E007FF0000000003C003FF8000000003C001FFC0 000000078000FFE00000000F00007FF00000001F00003FF80000003E00001FFC0000007C 00000FFF000001F8000003FFE00007F0000001FFFE003FC00000007FFFFFFF000000000F FFFFFC0000000001FFFFF000000000001FFF0000003C3D7BBB47>67 D<001FFF00000001FFFFF0000003FFFFFC000007F007FE00000FF801FF00001FFC00FF80 001FFC007FC0001FFC007FE0001FFC003FE0000FF8003FF0000FF8003FF00007F0003FF0 0001C0003FF0000000003FF0000000003FF0000000003FF0000000FFFFF000000FFFFFF0 00007FF83FF00001FF803FF00007FE003FF0000FF8003FF0001FF0003FF0003FE0003FF0 007FE0003FF0007FE0003FF000FFC0003FF000FFC0003FF000FFC0003FF000FFC0003FF0 00FFC0007FF0007FE0007FF0007FE000DFF0003FF0039FF8001FFC0F0FFFF007FFFE0FFF F001FFFC07FFF0003FE000FFF02C267DA530>97 D<0001FFC000000FFFF800003FFFFE00 00FF80FF0001FE003F8007FC001FC00FF8000FE00FF8000FF01FF00007F03FF00007F83F F00007F87FE00007F87FE00003FC7FE00003FC7FE00003FCFFE00003FCFFFFFFFFFCFFFF FFFFFCFFFFFFFFFCFFE0000000FFE0000000FFE0000000FFE00000007FE00000007FE000 00007FE00000003FE00000003FF000003C1FF000003C1FF000003C0FF800007807FC0000 F803FE0001F001FF0007E000FFC03FC0003FFFFF000007FFFC000000FFE00026267DA52D >101 D<00FF00000000FFFF00000000FFFF00000000FFFF00000000FFFF0000000007FF 0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF 0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF 0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF007FC00003FF 01FFF80003FF07FFFC0003FF0F03FE0003FF1C01FF0003FF3001FF8003FF6000FF8003FF E000FFC003FFC000FFC003FF8000FFC003FF8000FFC003FF8000FFC003FF0000FFC003FF 0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF 0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF 0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF0000FFC003FF 0000FFC003FF0000FFC003FF0000FFC0FFFFFC3FFFFFFFFFFC3FFFFFFFFFFC3FFFFFFFFF FC3FFFFF303C7CBB37>104 D<00FF01FF8000FFFF0FFFF000FFFF3FFFFC00FFFFFE03FF 00FFFFF000FF8003FFC0007FC003FF80003FE003FF00003FF003FF00001FF803FF00001F FC03FF00000FFC03FF00000FFE03FF00000FFE03FF000007FE03FF000007FF03FF000007 FF03FF000007FF03FF000007FF03FF000007FF03FF000007FF03FF000007FF03FF000007 FF03FF000007FF03FF000007FE03FF000007FE03FF00000FFE03FF00000FFC03FF00000F FC03FF00001FF803FF00001FF803FF00003FF003FF80003FE003FFC0007FC003FFF001FF 8003FFFC07FF0003FF3FFFFC0003FF0FFFF00003FF01FF000003FF0000000003FF000000 0003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF000000 0003FF0000000003FF0000000003FF0000000003FF0000000003FF00000000FFFFFC0000 00FFFFFC000000FFFFFC000000FFFFFC00000030377DA537>112 D<00FE03F000FFFE0FFE00FFFE1FFF00FFFE3C3F80FFFE707FC007FE60FFE003FEE0FFE0 03FEC0FFE003FFC0FFE003FF807FC003FF807FC003FF803F8003FF800E0003FF00000003 FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF 00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00 000003FF00000003FF00000003FF00000003FF00000003FF000000FFFFFE0000FFFFFE00 00FFFFFE0000FFFFFE000023267DA529>114 D<00078000000780000007800000078000 00078000000F8000000F8000000F8000000F8000001F8000001F8000003F8000003F8000 007F800000FF800001FF800007FF80001FFFFFF0FFFFFFF0FFFFFFF0FFFFFFF001FF8000 01FF800001FF800001FF800001FF800001FF800001FF800001FF800001FF800001FF8000 01FF800001FF800001FF800001FF800001FF800001FF800001FF800001FF800001FF8000 01FF803C01FF803C01FF803C01FF803C01FF803C01FF803C01FF803C01FF803C00FF8078 00FFC078007FC070003FE0E0001FFFC00007FF800001FF001E377EB626>116 D E /Fq 83 124 df<001F83E000F06E3001C078780380F8780300F03007007000070070 000700700007007000070070000700700007007000FFFFFF800700700007007000070070 000700700007007000070070000700700007007000070070000700700007007000070070 000700700007007000070070000700700007007000070070007FE3FF001D20809F1B>11 D<003F0000E0C001C0C00381E00701E00701E00700000700000700000700000700000700 00FFFFE00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700 E00700E00700E00700E00700E00700E00700E00700E07FC3FE1720809F19>I<003FE000 E0E001C1E00381E00700E00700E00700E00700E00700E00700E00700E00700E0FFFFE007 00E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E007 00E00700E00700E00700E00700E00700E07FE7FE1720809F19>I<001F81F80000F04F04 0001C07C06000380F80F000300F00F000700F00F00070070000007007000000700700000 070070000007007000000700700000FFFFFFFF0007007007000700700700070070070007 007007000700700700070070070007007007000700700700070070070007007007000700 700700070070070007007007000700700700070070070007007007000700700700070070 07007FE3FE3FF02420809F26>I<001F81FF0000F06F070001C07C0F000380F80F000300 F00700070070070007007007000700700700070070070007007007000700700700070070 0700FFFFFFFF000700700700070070070007007007000700700700070070070007007007 000700700700070070070007007007000700700700070070070007007007000700700700 070070070007007007000700700700070070070007007007007FE3FE3FF02420809F26> I<70F8F8F8F8F8F8F8707070707070707070702020202020000000000070F8F8F8700521 7CA00D>33 D<7038F87CFC7EFC7E743A0402040204020804080410081008201040200F0E 7E9F17>I<00780000008400000184000003020000070200000702000007020000070200 00070400000704000007080000070800000310000003A00FFC03C003E0038001C001C000 8001C0010003E0010004E0020008F00200187004003078080070380800701C1000F01E10 00F00E2000F0074000F003C0087003C0087801C010380670301C18386007E00F801E227E A023>38 D<70F8FCFC74040404080810102040060E7C9F0D>I<00200040008001000200 06000C000C00180018003000300030007000600060006000E000E000E000E000E000E000 E000E000E000E000E000E0006000600060007000300030003000180018000C000C000600 020001000080004000200B2E7DA112>I<800040002000100008000C0006000600030003 0001800180018001C000C000C000C000E000E000E000E000E000E000E000E000E000E000 E000E000C000C000C001C001800180018003000300060006000C00080010002000400080 000B2E7DA112>I<01800180018001800180C183F18F399C0FF003C003C00FF0399CF18F C1830180018001800180018010147DA117>I<0006000000060000000600000006000000 060000000600000006000000060000000600000006000000060000000600000006000000 06000000060000FFFFFFF0FFFFFFF0000600000006000000060000000600000006000000 060000000600000006000000060000000600000006000000060000000600000006000000 0600001C207D9A23>I<70F8FCFC74040404080810102040060E7C840D>II<70F8F8F87005057C840D>I<000100030003000600060006000C000C000C0018 0018001800300030003000600060006000C000C000C00180018001800300030003000600 060006000C000C000C00180018001800300030003000600060006000C000C000C000102D 7DA117>I<03F0000E1C001C0E00180600380700700380700380700380700380F003C0F0 03C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C070 03807003807003807807803807001806001C0E000E1C0003F000121F7E9D17>I<018003 800F80F38003800380038003800380038003800380038003800380038003800380038003 800380038003800380038003800380038007C0FFFE0F1E7C9D17>I<03F0000C1C00100E 00200700400780800780F007C0F803C0F803C0F803C02007C00007C0000780000780000F 00000E00001C0000380000700000600000C0000180000300000600400C00401800401000 803FFF807FFF80FFFF80121E7E9D17>I<03F0000C1C00100E00200F00780F8078078078 0780380F80000F80000F00000F00000E00001C0000380003F000003C00000E00000F0000 07800007800007C02007C0F807C0F807C0F807C0F00780400780400F00200E001C3C0003 F000121F7E9D17>I<000600000600000E00000E00001E00002E00002E00004E00008E00 008E00010E00020E00020E00040E00080E00080E00100E00200E00200E00400E00C00E00 FFFFF0000E00000E00000E00000E00000E00000E00000E0000FFE0141E7F9D17>I<1803 001FFE001FFC001FF8001FE00010000010000010000010000010000010000011F000161C 00180E001007001007800003800003800003C00003C00003C07003C0F003C0F003C0E003 80400380400700200600100E000C380003E000121F7E9D17>I<007C000182000701000E 03800C07801C0780380300380000780000700000700000F1F000F21C00F40600F80700F8 0380F80380F003C0F003C0F003C0F003C0F003C07003C07003C070038038038038070018 07000C0E00061C0001F000121F7E9D17>I<4000007FFFC07FFF807FFF80400100800200 80020080040000080000080000100000200000200000400000400000C00000C00001C000 018000038000038000038000038000078000078000078000078000078000078000078000 030000121F7D9D17>I<03F0000C0C001006003003002001806001806001806001807001 807803003E03003F06001FC8000FF00003F80007FC000C7E00103F00300F806003804001 C0C001C0C000C0C000C0C000C0C000806001802001001002000C0C0003F000121F7E9D17 >I<03F0000E18001C0C00380600380700700700700380F00380F00380F003C0F003C0F0 03C0F003C0F003C07007C07007C03807C0180BC00E13C003E3C000038000038000038000 0700300700780600780E00700C002018001070000FC000121F7E9D17>I<70F8F8F87000 00000000000000000070F8F8F87005147C930D>I<70F8F8F87000000000000000000000 70F0F8F878080808101010202040051D7C930D>I<7FFFFFE0FFFFFFF000000000000000 00000000000000000000000000000000000000000000000000FFFFFFF07FFFFFE01C0C7D 9023>61 D<0FC0307040384038E03CF03CF03C603C0038007000E000C001800180010003 000200020002000200020002000000000000000000000007000F800F800F8007000E207D 9F15>63 D<000100000003800000038000000380000007C0000007C0000007C0000009E0 000009E0000009E0000010F0000010F0000010F00000207800002078000020780000403C 0000403C0000403C0000801E0000801E0000FFFE0001000F0001000F0001000F00020007 800200078002000780040003C00E0003C01F0007E0FFC03FFE1F207F9F22>65 DI<000FC040007030C001C009C0 038005C0070003C00E0001C01E0000C01C0000C03C0000C07C0000407C00004078000040 F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000 780000007C0000407C0000403C0000401C0000401E0000800E0000800700010003800200 01C0040000703800000FC0001A217D9F21>IIII72 D I<0FFFC0007C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C 00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00203C 00F83C00F83C00F83C00F0380040780040700030E0000F800012207E9E17>I76 DII<001F800000F0 F00001C0380007801E000F000F000E0007001E0007803C0003C03C0003C07C0003E07800 01E0780001E0F80001F0F80001F0F80001F0F80001F0F80001F0F80001F0F80001F0F800 01F0F80001F0780001E07C0003E07C0003E03C0003C03C0003C01E0007800E0007000F00 0F0007801E0001C0380000F0F000001F80001C217D9F23>II82 D<07E0800C1980100780300380600180600180E00180E00080E00080E00080F000 00F000007800007F00003FF0001FFC000FFE0003FF00001F800007800003C00003C00001 C08001C08001C08001C08001C0C00180C00380E00300F00600CE0C0081F80012217D9F19 >I<7FFFFFE0780F01E0600F0060400F0020400F0020C00F0030800F0010800F0010800F 0010800F0010000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F 0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F 0000000F0000000F0000001F800007FFFE001C1F7E9E21>IIII<7FF83FF80FE00FC007C0 070003C0020001E0040001F00C0000F0080000781000007C1000003C2000003E4000001E 4000000F8000000F8000000780000003C0000007E0000005E0000009F0000018F8000010 780000207C0000603C0000401E0000801F0001800F0001000780020007C0070003C01F80 07E0FFE01FFE1F1F7F9E22>II< FEFEC0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0 C0C0C0C0C0C0C0FEFE072D7CA10D>91 D<080410082010201040204020804080408040B8 5CFC7EFC7E7C3E381C0F0E7B9F17>II<1FE00030 3000781800781C00300E00000E00000E00000E0000FE00078E001E0E00380E00780E00F0 0E10F00E10F00E10F01E10781E103867200F83C014147E9317>97 D<0E0000FE00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00 000E3E000EC3800F01C00F00E00E00E00E00700E00700E00780E00780E00780E00780E00 780E00780E00700E00700E00E00F00E00D01C00CC300083E0015207F9F19>I<03F80E0C 1C1E381E380C70007000F000F000F000F000F000F00070007000380138011C020E0C03F0 10147E9314>I<000380003F800003800003800003800003800003800003800003800003 8000038000038003E380061B801C0780380380380380700380700380F00380F00380F003 80F00380F00380F003807003807003803803803807801C07800E1B8003E3F815207E9F19 >I<03F0000E1C001C0E00380700380700700700700380F00380F00380FFFF80F00000F0 0000F000007000007000003800801800800C010007060001F80011147F9314>I<007C00 C6018F038F07060700070007000700070007000700FFF007000700070007000700070007 00070007000700070007000700070007000700070007007FF01020809F0E>I<0000E003 E3300E3C301C1C30380E00780F00780F00780F00780F00780F00380E001C1C001E380033 E0002000002000003000003000003FFE001FFF800FFFC03001E0600070C00030C00030C0 0030C000306000603000C01C038003FC00141F7F9417>I<0E0000FE00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E3E000E43000E81800F01C0 0F01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0 0E01C00E01C00E01C0FFE7FC16207F9F19>I<1C001E003E001E001C0000000000000000 00000000000E007E000E000E000E000E000E000E000E000E000E000E000E000E000E000E 000E000E000E00FFC00A1F809E0C>I<00E001F001F001F000E000000000000000000000 0000007007F000F000700070007000700070007000700070007000700070007000700070 00700070007000700070007000706070F060F0C061803F000C28829E0E>I<0E0000FE00 000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E0FF00E03 C00E03000E02000E04000E08000E10000E30000E70000EF8000F38000E1C000E1E000E0E 000E07000E07800E03800E03C00E03E0FFCFF815207F9F18>I<0E00FE000E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00 0E000E000E000E000E000E000E000E00FFE00B20809F0C>I<0E1F01F000FE618618000E 81C81C000F00F00E000F00F00E000E00E00E000E00E00E000E00E00E000E00E00E000E00 E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E0 0E000E00E00E000E00E00E00FFE7FE7FE023147F9326>I<0E3E00FE43000E81800F01C0 0F01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0 0E01C00E01C00E01C0FFE7FC16147F9319>I<01F800070E001C03803801C03801C07000 E07000E0F000F0F000F0F000F0F000F0F000F0F000F07000E07000E03801C03801C01C03 80070E0001F80014147F9317>I<0E3E00FEC3800F01C00F00E00E00E00E00F00E00700E 00780E00780E00780E00780E00780E00780E00700E00F00E00E00F01E00F01C00EC3000E 3E000E00000E00000E00000E00000E00000E00000E00000E0000FFE000151D7F9319>I< 03E0800619801C05803C0780380380780380700380F00380F00380F00380F00380F00380 F003807003807803803803803807801C0B800E138003E380000380000380000380000380 000380000380000380000380003FF8151D7E9318>I<0E78FE8C0F1E0F1E0F0C0E000E00 0E000E000E000E000E000E000E000E000E000E000E000E00FFE00F147F9312>I<1F9030 704030C010C010C010E00078007F803FE00FF00070803880188018C018C018E030D0608F 800D147E9312>I<020002000200060006000E000E003E00FFF80E000E000E000E000E00 0E000E000E000E000E000E000E080E080E080E080E080610031001E00D1C7F9B12>I<0E 01C0FE1FC00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E03C00603C0030DC001F1FC16147F9319>III<7FC3FC0F01E00701C007018003810001C20000E40000EC00007800003800003C 00007C00004E000087000107000303800201C00601E01E01E0FF07FE1714809318>II<3FFF380E200E201C40384078407000 E001E001C00380078007010E011E011C0338027006700EFFFE10147F9314>II E /Fr 46 122 df<00200040008001000300060004000C000C0018001800 3000300030007000600060006000E000E000E000E000E000E000E000E000E000E000E000 E000E000E0006000600060007000300030003000180018000C000C000400060003000100 0080004000200B327CA413>40 D<800040002000100018000C0004000600060003000300 01800180018001C000C000C000C000E000E000E000E000E000E000E000E000E000E000E0 00E000E000E000C000C000C001C0018001800180030003000600060004000C0018001000 2000400080000B327DA413>I<70F8FCFC7404040404080810102040060F7C840E>44 DI<70F8F8F87005057C840E>I<008003800F80F3800380038003 800380038003800380038003800380038003800380038003800380038003800380038003 8003800380038003800380038007C0FFFE0F217CA018>49 D<03F0000C1C001007002007 804003C04003C08003E0F003E0F801E0F801E0F801E02003E00003E00003C00003C00007 80000700000E00001C0000180000300000600000C0000180000100000200200400200800 201800603000403FFFC07FFFC0FFFFC013217EA018>I<000200000600000E00000E0000 1E00001E00002E00004E00004E00008E00008E00010E00020E00020E00040E00040E0008 0E00100E00100E00200E00200E00400E00800E00FFFFF8000E00000E00000E00000E0000 0E00000E00000E00001F0001FFF015217FA018>52 D<1000801E07001FFF001FFE001FF8 0013E00010000010000010000010000010000010000010F800130E001407001803801003 800001C00001C00001E00001E00001E00001E07001E0F001E0F001E0E001C08001C04003 C04003802007001006000C1C0003F00013227EA018>I<007E0001C1000300800601C00E 03C01C03C0180180380000380000780000700000700000F0F800F30C00F40600F40300F8 0380F801C0F001C0F001E0F001E0F001E0F001E0F001E07001E07001E07001E03801C038 01C01803801C03000C0600070C0001F00013227EA018>I<01F800060E00080300100180 2001802000C06000C06000C06000C07000C07801803E01003F02001FC4000FF80003F800 03FC00067F00083F80100F803007C06001C06000E0C000E0C00060C00060C00060C00060 6000406000C03000801803000E0E0003F00013227EA018>56 D<01F000060C000C060018 0700380380700380700380F001C0F001C0F001C0F001E0F001E0F001E0F001E0F001E070 01E07003E03803E01805E00C05E00619E003E1E00001C00001C00001C000038000038030 0300780700780600700C002018001030000FC00013227EA018>I<000180000001800000 0180000003C0000003C0000003C0000005E0000005E000000DF0000008F0000008F00000 10F800001078000010780000203C0000203C0000203C0000401E0000401E0000401E0000 800F0000800F0000FFFF000100078001000780030007C0020003C0020003C0040003E004 0001E0040001E00C0000F00C0000F03E0001F8FF800FFF20237EA225>65 D<0007E0100038183000E0063001C00170038000F0070000F00E0000701E0000701C0000 303C0000303C0000307C0000107800001078000010F8000000F8000000F8000000F80000 00F8000000F8000000F8000000F800000078000000780000107C0000103C0000103C0000 101C0000201E0000200E000040070000400380008001C0010000E0020000381C000007E0 001C247DA223>67 DIII72 DI77 DI80 D82 D<03F0200C0C601802603001E07000E0600060E00060E00060E00020 E00020E00020F00000F000007800007F00003FF0001FFE000FFF0003FF80003FC00007E0 0001E00000F00000F0000070800070800070800070800070C00060C00060E000C0F000C0 C80180C6070081FC0014247DA21B>I<7FFFFFF878078078600780184007800840078008 40078008C007800C80078004800780048007800480078004000780000007800000078000 000780000007800000078000000780000007800000078000000780000007800000078000 000780000007800000078000000780000007800000078000000780000007800000078000 000FC00003FFFF001E227EA123>I<0FE0001838003C0C003C0E00180700000700000700 00070000FF0007C7001E07003C0700780700700700F00708F00708F00708F00F08781708 3C23900FC1E015157E9418>97 D<0E0000FE00001E00000E00000E00000E00000E00000E 00000E00000E00000E00000E00000E00000E00000E1F000E61C00E80600F00300E00380E 003C0E001C0E001E0E001E0E001E0E001E0E001E0E001E0E001E0E001C0E003C0E00380F 00700C80600C41C0083F0017237FA21B>I<01FE000703000C07801C0780380300780000 700000F00000F00000F00000F00000F00000F00000F000007000007800403800401C0080 0C010007060001F80012157E9416>I<0000E0000FE00001E00000E00000E00000E00000 E00000E00000E00000E00000E00000E00000E00000E001F8E00704E00C02E01C01E03800 E07800E07000E0F000E0F000E0F000E0F000E0F000E0F000E0F000E07000E07800E03800 E01801E00C02E0070CF001F0FE17237EA21B>I<01FC000707000C03801C01C03801C078 01E07000E0F000E0FFFFE0F00000F00000F00000F00000F000007000007800203800201C 00400E008007030000FC0013157F9416>I<003C00C6018F038F030F0700070007000700 07000700070007000700FFF8070007000700070007000700070007000700070007000700 07000700070007000700070007807FF8102380A20F>I<00007001F198071E180E0E181C 07001C07003C07803C07803C07803C07801C07001C07000E0E000F1C0019F00010000010 00001800001800001FFE000FFFC00FFFE03800F0600030400018C00018C00018C0001860 00306000303800E00E038003FE0015217F9518>I<0E0000FE00001E00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E00000E1F800E60C00E80E0 0F00700F00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E0070 0E00700E00700E00700E00700E0070FFE7FF18237FA21B>I<1C001E003E001E001C0000 0000000000000000000000000000000E00FE001E000E000E000E000E000E000E000E000E 000E000E000E000E000E000E000E000E000E00FFC00A227FA10E>I<01C003E003E003E0 01C00000000000000000000000000000000001E00FE001E000E000E000E000E000E000E0 00E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E060E0 F0C0F18061803E000B2C82A10F>I<0E0000FE00001E00000E00000E00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E03FC0E01F00E01C00E01800E0200 0E04000E08000E10000E38000EF8000F1C000E1E000E0E000E07000E07800E03C00E01C0 0E01E00E00F00E00F8FFE3FE17237FA21A>I<0E1FC07F00FE60E183801E807201C00F00 3C00E00F003C00E00E003800E00E003800E00E003800E00E003800E00E003800E00E0038 00E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800 E00E003800E00E003800E0FFE3FF8FFE27157F942A>109 D<0E1F80FE60C01E80E00F00 700F00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00 700E00700E00700E00700E0070FFE7FF18157F941B>I<01FC000707000C01801800C038 00E0700070700070F00078F00078F00078F00078F00078F00078F000787000707800F038 00E01C01C00E038007070001FC0015157F9418>I<0E1F00FE61C00E80600F00700E0038 0E003C0E001C0E001E0E001E0E001E0E001E0E001E0E001E0E001E0E003C0E003C0E0038 0F00700E80E00E41C00E3F000E00000E00000E00000E00000E00000E00000E00000E0000 0E0000FFE000171F7F941B>I<0E3CFE461E8F0F0F0F060F000E000E000E000E000E000E 000E000E000E000E000E000E000E000F00FFF010157F9413>114 D<0F8830786018C018C008C008E008F0007F803FE00FF001F8003C801C800C800CC00CC0 08E018D0308FC00E157E9413>I<02000200020002000600060006000E001E003E00FFF8 0E000E000E000E000E000E000E000E000E000E000E000E040E040E040E040E040E040708 030801F00E1F7F9E13>I<0E0070FE07F01E00F00E00700E00700E00700E00700E00700E 00700E00700E00700E00700E00700E00700E00700E00700E00F00E00F006017003827800 FC7F18157F941B>I119 D121 D E /Fs 20 118 df45 D68 D73 D77 D80 D<007F802001FFE02007C078600F001C601E0006E03C0003E0380001E078 0000E0700000E070000060F0000060F0000060F0000020F0000020F0000020F8000020F8 0000007C0000007E0000003F0000003FC000001FF800000FFF800007FFF80003FFFC0000 FFFF00000FFF800000FFC000001FE0000007E0000003F0000001F0000000F0000000F800 0000F88000007880000078800000788000007880000078C0000078C0000070E00000F0E0 0000E0F00000E0F80001C0EC000380C7000700C1F01E00807FFC00800FF0001D337CB125 >83 D<00FE00000303C0000C00E00010007000100038003C003C003E001C003E001E003E 001E0008001E0000001E0000001E0000001E00000FFE0000FC1E0003E01E000F801E001F 001E003E001E003C001E007C001E00F8001E04F8001E04F8001E04F8003E04F8003E0478 003E047C005E043E008F080F0307F003FC03E01E1F7D9E21>97 D<003F8000E060038018 0700040F00041E001E1C003E3C003E7C003E7C0008780000F80000F80000F80000F80000 F80000F80000F80000F80000F800007800007C00007C00003C00011E00011E00020F0002 07000403801800E060003F80181F7D9E1D>99 D<000001E000003FE000003FE0000003E0 000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0 000001E0000001E0000001E0000001E0000001E0000001E0001F81E000F061E001C019E0 078005E00F0003E00E0003E01E0001E03C0001E03C0001E07C0001E0780001E0F80001E0 F80001E0F80001E0F80001E0F80001E0F80001E0F80001E0F80001E0F80001E0780001E0 780001E03C0001E03C0001E01C0001E01E0003E00E0005E0070009E0038011F000E061FF 003F81FF20327DB125>I<003F800000E0E0000380380007003C000E001E001E001E001C 000F003C000F007C000F0078000F8078000780F8000780F8000780FFFFFF80F8000000F8 000000F8000000F8000000F8000000F8000000780000007C0000003C0000003C0000801E 0000800E0001000F0002000780020001C00C0000F03000001FC000191F7E9E1D>I<0007 E0001C1000383800707C00E07C01E07C01C03803C00003C00003C00003C00003C00003C0 0003C00003C00003C00003C00003C00003C000FFFFC0FFFFC003C00003C00003C00003C0 0003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C0 0003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00007E0007FFF 007FFF0016327FB114>I<000000F0007F030801C1C41C0380E81C070070080F0078001E 003C001E003C003E003E003E003E003E003E003E003E003E003E003E003E001E003C001E 003C000F007800070070000780E00009C1C000087F000018000000180000001800000018 000000180000001C0000000E0000000FFFF80007FFFF0003FFFF800E000FC0180001E030 0000F070000070E0000038E0000038E0000038E0000038E0000038700000707000007038 0000E01C0001C00700070001C01C00003FE0001E2F7E9F21>I<07000F801F801F800F80 0700000000000000000000000000000000000000000000000780FF80FF800F8007800780 078007800780078007800780078007800780078007800780078007800780078007800780 07800780078007800FC0FFF8FFF80D307EAF12>105 D<0780FE001FC000FF83078060F0 00FF8C03C18078000F9001E2003C0007A001E4003C0007A000F4001E0007C000F8001E00 07C000F8001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E0007 8000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E000780 00F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000 F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0 001E000FC001F8003F00FFFC1FFF83FFF0FFFC1FFF83FFF0341F7E9E38>109 D<0780FE0000FF83078000FF8C03C0000F9001E00007A001E00007A000F00007C000F000 07C000F000078000F000078000F000078000F000078000F000078000F000078000F00007 8000F000078000F000078000F000078000F000078000F000078000F000078000F0000780 00F000078000F000078000F000078000F000078000F000078000F000078000F0000FC001 F800FFFC1FFF80FFFC1FFF80211F7E9E25>I<001FC00000F0780001C01C00070007000F 0007801E0003C01C0001C03C0001E03C0001E0780000F0780000F0780000F0F80000F8F8 0000F8F80000F8F80000F8F80000F8F80000F8F80000F8F80000F8780000F07C0001F03C 0001E03C0001E01E0003C01E0003C00F00078007800F0001C01C0000F07800001FC0001D 1F7E9E21>I<0783E0FF8C18FF907C0F907C07A07C07C03807C00007C00007C000078000 078000078000078000078000078000078000078000078000078000078000078000078000 0780000780000780000780000780000780000FC000FFFE00FFFE00161F7E9E19>114 D<01FC100E03301800F0300070600030E00030E00010E00010E00010F00010F800007E00 003FF0001FFF000FFFC003FFE0003FF00001F80000F880003C80003C80001CC0001CC000 1CE0001CE00018F00038F00030CC0060C301C080FE00161F7E9E1A>I<00400000400000 400000400000400000C00000C00000C00001C00001C00003C00007C0000FC0001FFFE0FF FFE003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003 C00003C00003C00003C00003C00003C01003C01003C01003C01003C01003C01003C01003 C01001C02001E02000E0400078C0001F00142C7FAB19>I<078000F000FF801FF000FF80 1FF0000F8001F000078000F000078000F000078000F000078000F000078000F000078000 F000078000F000078000F000078000F000078000F000078000F000078000F000078000F0 00078000F000078000F000078000F000078000F000078000F000078000F000078001F000 078001F000078001F000038002F00003C004F00001C008F800007030FF80001FC0FF8021 1F7E9E25>I E /Ft 5 85 df<00000000C00000000000E00000000001E00000000003E0 0000000003E00000000007E00000000007E0000000000FE0000000000FE0000000001FE0 000000001FE00000000037E00000000067E00000000067E000000000C7E000000000C7F0 0000000183F00000000183F00000000303F00000000703F00000000603F00000000C03F0 0000000C03F00000001803F00000001803F00000003003F00000003003F00000006003F0 000000C003F0000000C003F00000018003F00000018003F8000003FFFFF8000003FFFFF8 0000060001F800000E0001F800000C0001F80000180001F80000180001F80000300001F8 0000300001F80000600001F80000E00001F80000C00001F80001C00001F80001C00001F8 0007C00001FC001FC00003FC00FFF8007FFFE0FFF8007FFFE02B327BB135>65 D<000FFFFFFE0000000FFFFFFF800000007F000FE00000007E0003F00000007E0000F800 00007E0000FC0000007E00007C000000FC00003E000000FC00003E000000FC00003F0000 00FC00001F000001F800001F000001F800001F800001F800001F800001F800001F800003 F000001F800003F000001F800003F000001F800003F000001F800007E000003F800007E0 00003F800007E000003F800007E000003F80000FC000003F00000FC000007F00000FC000 007F00000FC000007F00001F8000007E00001F800000FE00001F800000FE00001F800000 FC00003F000001FC00003F000001F800003F000001F800003F000003F000007E000003E0 00007E000007E000007E00000FC000007E00000F800000FC00001F800000FC00003F0000 00FC00007E000000FC0000FC000001F80001F0000001F80003E0000001F8000FC0000003 F8007F000000FFFFFFFC000000FFFFFFE000000031317BB036>68 D<000FFFFFFFFC000FFFFFFFFC00007F0001FC00007E00007C00007E00003C00007E0000 3C00007E0000180000FC0000180000FC0000180000FC0000180000FC0000180001F80000 180001F80000180001F80000180001F80000180003F00080100003F00180000003F00180 000003F00180000007E00300000007E00300000007E00700000007E01F0000000FFFFE00 00000FFFFE0000000FC01E0000000FC00E0000001F800C0000001F800C0000001F800C00 00001F800C0000003F00180000003F00080000003F00000000003F00000000007E000000 00007E00000000007E00000000007E0000000000FC0000000000FC0000000000FC000000 0000FC0000000001F80000000001F80000000001F80000000003F800000000FFFFF00000 00FFFFF00000002E317BB02F>70 D<000FFFFFF000000FFFFFFE0000007F003F8000007E 000FC000007E0007E000007E0003F000007E0001F80000FC0001F80000FC0001F80000FC 0001F80000FC0001F80001F80003F80001F80003F80001F80003F80001F80003F00003F0 0007F00003F00007E00003F0000FC00003F0000FC00007E0001F000007E0007E000007E0 00FC000007E007F000000FFFFFC000000FFFFF0000000FC00F8000000FC003C000001F80 03E000001F8001F000001F8001F000001F8001F800003F0001F800003F0001F800003F00 01F800003F0001F800007E0003F800007E0003F800007E0003F000007E0003F00000FC00 07F00000FC0007F00000FC0007F00800FC0007F00C01F80007F01801F80007F01801F800 03F03003F80003F030FFFFE001F0E0FFFFE000FFC0000000003F002E327BB034>82 D<07FFFFFFFFF00FFFFFFFFFF00FC00FE003F01E000FC000F01C000FC000E018000FC000 E038000FC0006030001F8000E030001F8000E060001F8000C060001F8000C060003F0000 C0C0003F0000C0C0003F0000C0C0003F0000C080007E00008000007E00000000007E0000 0000007E0000000000FC0000000000FC0000000000FC0000000000FC0000000001F80000 000001F80000000001F80000000001F80000000003F00000000003F00000000003F00000 000003F00000000007E00000000007E00000000007E00000000007E0000000000FC00000 00000FC0000000000FC0000000000FC0000000001F80000000001F80000000001F800000 00001F80000000003F00000000003F00000000003F0000000000FF00000000FFFFFF0000 00FFFFFF0000002C3173B033>84 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300dpi TeXDict begin %%EndSetup %%Page: 0 1 0 0 bop 795 947 a Ft(D)26 b(R)g(A)f(F)h(T)225 1038 y Fs(Do)r(cumen)n(t)20 b(for)i(a)f(Standard)g(Message-P)n(assing)f(In)n (terface)621 1232 y Fr(Message)c(P)o(assing)h(In)o(terface)e(F)l(orum) 766 1359 y(Septem)o(b)q(er)g(19,)h(1996)190 1417 y(This)h(w)o(ork)f(w)o (as)h(supp)q(orted)g(in)f(part)g(b)o(y)g(NSF)g(and)h(ARP)l(A)e(under)h (NSF)g(con)o(tract)283 1475 y(CD)o(A-9115428)j(and)e(Esprit)f(under)h (pro)s(ject)e(HPC)i(Standards)g(\(21111\).)p eop %%Page: 1 2 1 1 bop 166 45 a Fq(This)20 b(is)h(the)f(result)g(of)f(a)h(LaT)l(eX)g (run)g(of)g(a)f(draft)g(of)h(a)f(single)j(c)o(hapter)d(of)h(the)g(MPIF) f(Final)75 102 y(Rep)q(ort)d(do)q(cumen)o(t.)969 2828 y(i)p eop %%Page: 1 3 1 2 bop 75 356 a Fp(Chapter)34 b(3)75 564 y Fo(Miscellan)m(y)41 b(for)g(1.2)75 805 y Fn(3.1)59 b(V)n(ersion)20 b(Numb)r(er)75 953 y Fm(Curren)o(t)12 b(Status:)k Fl(P)o(assed)d(once.)20 b Fq(In)14 b(order)f(to)f(cop)q(e)i(with)g(c)o(hanges)f(to)f(the)i(MPI) f(Standard,)g(there)g(are)75 1057 y(b)q(oth)j(compile-time)h(and)f (run-time)g(w)o(a)o(ys)e(to)h(determine)h(whic)o(h)g(v)o(ersion)g(of)f (the)h(standard)e(is)i(in)h(use)75 1114 y(in)f(the)f(en)o(vironmen)o(t) h(one)f(is)h(using.)166 1170 y(The)e(\\v)o(ersion")g(will)h(b)q(e)g (represen)o(ted)f(b)o(y)g(t)o(w)o(o)e(separate)i(in)o(tegers,)g(for)f (the)h(v)o(ersion)g(and)g(sub)o(v)o(er-)75 1227 y(sion:)166 1283 y(In)i(C,)170 1367 y Fk(#define)23 b(MPI_VERSION)94 b(1)170 1423 y(#define)23 b(MPI_SUBVERSION)f(2)75 1507 y Fq(in)16 b(F)l(ortran,)170 1591 y Fk(INTEGER)23 b(MPI_VERSION,)g (MPI_SUBVERSION)170 1648 y(PARAMETER)g(\(MPI_VERSION)94 b(=)24 b(1\))170 1704 y(PARAMETER)f(\(MPI_SUBVERSION)f(=)i(2\))75 1788 y Fq(and)15 b(in)h(C++,)170 1872 y Fk(const)24 b(int)f (MPI::VERSION)f(=)i(1)170 1928 y(const)g(int)f(MPI::SUBVERSION)f(=)i(2) 75 2012 y Fq(F)l(or)15 b(run)o(time)g(determination,)75 2163 y Fj(MPI)p 160 2163 14 2 v 16 w(GET)p 264 2163 V 17 w(VERSION\()h(version,)f(subversion)h(\))117 2240 y Fl(OUT)108 b Fj(version)456 b Fl(v)o(ersion)14 b(n)o(um)o(b)q(er)117 2313 y(OUT)108 b Fj(subversion)393 b Fl(sub)o(v)o(ersion)15 b(n)o(um)o(b)q(er)75 2438 y Fk(int)23 b(MPI)p 245 2438 15 2 v 17 w(Get)p 334 2438 V 17 w(version\()g(int)g(*version,)g(int)g (*subversion)g(\))75 2524 y(MPI)p 150 2524 V 17 w(GET)p 239 2524 V 17 w(VERSION\()f(VERSION,)h(SUBVERSION,)f(IERROR)h(\))170 2581 y(INTEGER)g(VERSION,)g(SUBVERSION,)f(IERROR)1875 2617 y Fi(>)16 b Fh(\(Oct\))75 2667 y Fk(static)23 b(int)g(MPI::Get)p 532 2667 V 17 w(version\()f(int&)i(version,)e(int&)i(subversion)e(\)) 1875 2704 y Fi(?)16 b Fh(\(Oct\))964 2828 y Fq(1)p eop %%Page: 2 4 2 3 bop 75 -100 a Fq(2)970 b Fg(CHAPTER)15 b(3.)35 b(MISCELLANY)17 b(F)o(OR)e(1.2)75 45 y Fn(3.2)59 b(T)-5 b(reatment)18 b(of)i(MPI)p 670 45 18 2 v 21 w(Status)75 147 y Fq(The)25 b(follo)o(wing)g(prop)q(osals)g(add)g(to,)g(but)g(do)g(not)f(c)o (hange,)i(the)f(functionalit)o(y)h(asso)q(ciated)f(with)75 203 y Fj(MPI)p 160 203 14 2 v 16 w(ST)l(A)l(TUS)p Fq(.)75 325 y Ff(3.2.1)49 b(P)o(assing)17 b(MPI)p 481 325 15 2 v 18 w(ST)l(A)l(TUS)p 676 325 V 18 w(IGNORE)g(fo)o(r)f(MPI)p 1044 325 V 18 w(Status)75 458 y Fm(Curren)o(t)e(Status:)j Fl(P)o(assed)e(once.)166 561 y Fq(Ev)o(ery)10 b(call)i(to)e Fj(MPI)p 507 561 14 2 v 16 w(RECV)h Fq(includes)i(a)e Fj(status)h Fq(argumen)o(t,)f(where)g(the)f(system)h(can)f(return)h (details)75 618 y(ab)q(out)18 b(the)g(message)g(receiv)o(ed.)31 b(There)18 b(are)g(also)g(a)g(n)o(um)o(b)q(er)h(of)f(other)g(MPI)g (calls,)i(particularly)f(in)-1991 b Fi(>)15 b Fh(\(Oct\))75 674 y Fq(MPI-2,)20 b(where)f Fj(status)i Fq(is)f(returned.)33 b(An)19 b(ob)s(ject)g(of)g(t)o(yp)q(e)g Fj(MPI)p 1234 674 V 16 w(ST)l(A)l(TUS)h Fq(is)g(not)f(an)g Fj(MPI)g Fq(opaque)-117 676 y Fi(?)c Fh(\(Oct\))75 731 y Fq(ob)s(ject;)h(its)h (structure)g(is)g(declared)h(in)f Fk(mpi.h)f Fq(and)h Fk(mpif.h)p Fq(,)e(and)i(it)g(exists)g(in)g(the)g(users')f(program.)75 787 y(In)i(man)o(y)e(cases)h(application)i(programs)d(are)g (constructed)i(so)e(that)h(it)g(is)g(unnecessary)h(for)f(them)g(to)75 844 y(examine)j(the)g Fk(status)f Fq(\014elds.)35 b(In)20 b(these)g(cases)f(it)h(is)g(a)g(w)o(aste)e(for)h(the)h(user)g(to)f (allo)q(cate)h(a)g(status)75 900 y(ob)s(ject,)f(and)h(it)f(is)h (particularly)g(w)o(asteful)f(for)g(the)g Fj(MPI)f Fq(implemen)o (tation)j(to)d(\014ll)j(in)f(\014elds)h(in)f(this)75 957 y(ob)s(ject.)166 1013 y(T)l(o)h(cop)q(e)h(with)g(this)g(problem,)i (there)d(is)h(a)g(pre-de\014ned)h(constan)o(t,)f Fj(MPI)p 1502 1013 V 16 w(ST)l(A)l(TUS)p 1683 1013 V 17 w(IGNORE)p Fq(,)75 1070 y(whic)o(h,)i(when)e(passed)g(to)f(a)h(receiv)o(e)h(or)e (test)g(function,)j(informs)e(the)g(implemen)o(tation)h(that)e(the)75 1126 y(status)13 b(\014elds)h(are)f(not)g(to)g(b)q(e)h(\014lled)i(in.)k (Note)13 b(that)g Fj(MPI)p 1059 1126 V 15 w(ST)l(A)l(TUS)p 1239 1126 V 18 w(IGNORE)i Fq(is)e(not)h(a)f(sp)q(ecial)i(t)o(yp)q(e)e (of)75 1183 y Fj(MPI)p 160 1183 V 16 w(ST)l(A)l(TUS)i Fq(ob)s(ject;)f(rather,)f(it)h(is)h(a)f(sp)q(ecial)i(v)m(alue)f(for)f (the)g(argumen)o(t.)k(That)c(is,)h(in)f(C)g(one)h(w)o(ould)75 1239 y(exp)q(ect)h(it)f(to)g(b)q(e)h(NULL,)g(not)e(the)i(address)f(of)g (a)g(sp)q(ecial)i Fj(MPI)p 1179 1239 V 15 w(ST)l(A)l(TUS)p Fq(.)166 1295 y(In)12 b(general,)h(this)f(optimization)g(can)g(apply)g (to)f(all)i(functions)f(for)f(whic)o(h)h Fj(MPI)p 1514 1295 V 16 w(Status)i Fq(or)d(an)g(arra)o(y)75 1352 y(of)k Fj(MPI)p 212 1352 V 16 w(Status)p Fq('s)i(is)f(an)g(argumen)o(t.)k (These)c(are)f(all)i(the)e(v)m(arious)h(forms)f(of)g Fj(MPI)p 1493 1352 V 16 w(RECV)p Fq(,)h Fj(MPI)p 1735 1352 V 16 w(TEST)p Fq(,)75 1408 y(and)i Fj(MPI)p 251 1408 V 15 w(W)l(AIT)p Fq(.)f(When)h(an)f(arra)o(y)g(is)g(passed,)h(as)f (in)h(the)p 1132 1408 V 34 w Fj(ANY)g Fq(and)p 1349 1408 V 34 w Fj(ALL)f Fq(functions,)h(a)f(separate)75 1465 y(constan)o(t,)d Fj(MPI)p 356 1465 V 16 w(ST)l(A)l(TUSES)p 589 1465 V 18 w(IGNORE)p Fq(,)i(is)f(passed)h(for)e(the)i(arra)o(y)e (argumen)o(t.)166 1521 y Fj(MPI)p 251 1521 V 16 w(ST)l(A)l(TUS)p 432 1521 V 17 w(IGNORE)g Fq(and)e Fj(MPI)p 794 1521 V 16 w(ST)l(A)l(TUSES)p 1027 1521 V 18 w(IGNORE)i Fq(are)e(not)g (required)h(to)f(ha)o(v)o(e)g(the)g(same)-1992 b Fi(>)15 b Fh(\(Oct\))75 1578 y Fq(v)m(alues)h(in)g(C)f(and)h(F)l(ortran.)166 1634 y(It)d(is)g(not)g(allo)o(w)o(ed)g(to)f(ha)o(v)o(e)g(some)h(of)f (the)h(statuses)f(in)i(an)f(arra)o(y)e(of)i(statuses)f(for)p 1574 1634 V 28 w Fj(ANY)i Fq(and)p 1782 1634 V 29 w Fj(ALL)75 1691 y Fq(functions)j(set)g(to)f Fj(MPI)p 487 1691 V 16 w(ST)l(A)l(TUS)p 668 1691 V 17 w(IGNORE)p Fq(;)i(one)e(either)i(sp)q (eci\014es)g(ignoring)f Fe(al)r(l)34 b Fq(of)17 b(the)f(statuses)g(in) 75 1747 y(suc)o(h)e(a)g(call)h(with)f Fj(MPI)p 482 1747 V 16 w(ST)l(A)l(TUSES)p 715 1747 V 18 w(IGNORE)p Fq(,)h(or)f Fe(none)i Fq(of)e(themm)g(b)o(y)f(passing)i(normal)f(statuses)f(in)75 1804 y(alll)k(p)q(ositions)f(in)g(the)f(arra)o(y)f(of)h(statuses.)-932 b Fi(?)15 b Fh(\(Oct\))75 1925 y Ff(3.2.2)49 b(Non-destructive)16 b(T)l(est)f(of)i(MPI)p 808 1925 15 2 v 18 w(Status)75 2058 y Fm(Curren)o(t)d(Status:)j Fl(P)o(assed)e(once.)166 2162 y Fq(This)22 b(call)g(is)g(useful)g(for)f(accessing)h(the)f (information)h(asso)q(ciated)f(with)h(a)f(request,)h(without)75 2218 y(deleting)c(the)f(request)f(\(in)h(case)g(the)g(user)f(is)i(exp)q (ected)f(to)f(access)h(it)g(after)f(the)g(handler\).)25 b(It)17 b(allo)o(ws)75 2275 y(one)11 b(to)e(la)o(y)o(er)i(libraries)h (more)e(con)o(v)o(enien)o(tly)l(,)i(since)g(m)o(ultiple)g(la)o(y)o(ers) e(of)h(soft)o(w)o(are)d(ma)o(y)i(access)h(the)f(same)75 2331 y(completed)17 b(request)f(and)g(extract)f(from)g(it)h(the)g (status)f(information.)21 b(This)c(will)g(also)f(b)q(e)g(imp)q(ortan)o (t)75 2388 y(for)g(language)i(in)o(terop)q(erabilit)o(y)h(if)e(w)o(e)g (decide)i(that)d(status)g(ob)s(jects)h(cannot)f(b)q(e)i(transferred)f (across)75 2444 y(language)d(b)q(oundaries,)g(since)h(with)f(this)g (function)g(the)g(request)f(ob)s(ject)g(can)h(b)q(e)g(transferred)f (instead.)1967 46 y Fh(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 3 5 3 4 bop 75 -100 a Fg(3.3.)34 b(ERR)o(OR)17 b(CLASS)f(F)o(OR)f(INV)-5 b(ALID)16 b(KEYV)-5 b(AL)818 b Fq(3)75 45 y Fj(MPI)p 160 45 14 2 v 16 w(GET)p 264 45 V 17 w(ST)l(A)l(TUS\()16 b(request,)g(\015ag,)f(status)i(\))117 122 y Fl(IN)155 b Fj(request)452 b Fl(an)14 b Fd(MPI)p 1040 122 13 2 v 14 w(REQUEST)f Fl(ob)r(ject)117 197 y(OUT)108 b Fj(\015ag)518 b Fl(\015ag,)13 b(same)g(as)h(from)e Fd(MPI)p 1325 197 V 15 w(TEST)117 273 y Fl(OUT)108 b Fj(status)476 b Fd(MPI)p 982 273 V 15 w(ST)m(A)m(TUS)12 b Fl(ob)r(ject)j(if)e(\015ag)g(is)h (true)75 397 y Fk(int)23 b(MPI)p 245 397 15 2 v 17 w(Get)p 334 397 V 17 w(status\(MPI)p 591 397 V 16 w(Request)g(request,)f(int)i (*flag,)f(MPI)p 1347 397 V 17 w(Status)g(*status\))75 483 y(MPI)p 150 483 V 17 w(GET)p 239 483 V 17 w(STATUS\()f(REQUEST,)h (FLAG,)g(STATUS,)g(IERROR\))170 540 y(INTEGER)g(REQUEST,)g(FLAG,)g (STATUS\(MPI)p 962 540 V 16 w(STATUS)p 1122 540 V 16 w(SIZE\),)h(IERROR)75 626 y(int)f(MPI::Request::Get)p 581 626 V 15 w(status\(bool&)g(flag,)g(MPI::Status&)f(status\))h(const) 166 713 y Fq(Sets)18 b(\015ag=true)g(if)h(the)f(request)h(has)f (completed,)h(and,)g(if)g(so,)f(returns)g(in)h(status)f(the)g(request) 75 769 y(status.)34 b(Ho)o(w)o(ev)o(er,)20 b(unlik)o(e)i(test)d(or)h(w) o(ait,)g(it)h(do)q(es)f(not)g(deallo)q(cate)h(or)f(inactiv)m(ate)h(the) f(request;)i(a)75 826 y(subsequen)o(t)16 b(call)g(to)f(test,)f(w)o(ait) h(or)f(free)i(should)g(b)q(e)g(executed)g(with)f(that)g(request)75 969 y Fn(3.3)59 b(Erro)n(r)21 b(Class)f(fo)n(r)h(Invalid)e(Keyval)1875 1026 y Fi(>)d Fh(\(Oct\))75 1118 y Fm(Curren)o(t)e(Status:)j Fl(P)o(assed)e(once.)166 1221 y Fq(There)d(is)f(a)h(new)f(MPI)h(error)e (class:)19 b Fj(MPI)p 875 1221 14 2 v 15 w(ERR)p 975 1221 V 18 w(KEYV)l(AL)p Fq(.)11 b(It)h(can)f(b)q(e)h(returned)g(b)o(y)f Fj(MPI)p 1692 1221 V 16 w(A)o(ttr)p 1785 1221 V 17 w(put)p Fq(,)75 1278 y Fj(MPI)p 160 1278 V 16 w(A)o(ttr)p 253 1278 V 17 w(get)p Fq(,)16 b Fj(MPI)p 440 1278 V 16 w(A)o(ttr)p 533 1278 V 16 w(delete)p Fq(,)i Fj(MPI)p 772 1278 V 16 w(Keyval)p 915 1278 V 15 w(free)p Fq(,)e Fj(MPI)p 1111 1278 V 16 w(Comm)p 1253 1278 V 14 w(dup)p Fq(,)i(and)e Fj(MPI)p 1538 1278 V 16 w(Comm)p 1680 1278 V 14 w(free)p Fq(.)23 b(The)75 1334 y(last)15 b(t)o(w)o(o)f(are)h(b)q(ecause)h Fj(k)o(eyval)f Fq(is)h(an)f(argumen)o(t)f(to)h(the)g(cop)o(y)g(and)h (delete)g(functions)g(for)e(attributes.)8 b Fi(?)16 b Fh(\(Oct\))1875 1393 y Fi(>)g Fh(\(Oct\))1875 1451 y Fi(>)g Fh(\(Oct\))75 1478 y Fn(A)k(new)f(reduce)g(op)r(eration)75 1626 y Fm(Curren)o(t)14 b(Status:)j Fl(New)d(for)g(Octob)q(er.)166 1730 y Fq(A)e(new)h(prede\014ned)h(op)q(eration,)f Fd(MPI)p 811 1730 13 2 v 14 w(REPLA)o(CE)p Fq(,)d(is)j(de\014ned.)20 b(It)13 b(corresp)q(onds)f(to)g(the)h(asso)q(ciativ)o(e)75 1786 y(function)18 b Fc(f)5 b Fq(\()p Fc(a;)j(b)p Fq(\))15 b(=)h Fc(b)p Fq(;)i(i.e.,)f(the)g(curren)o(t)h(v)m(alue)g(in)g(the)g (target)e(memory)h(is)h(replaced)g(b)o(y)f(the)h(v)m(alue)75 1843 y(supplied)g(b)o(y)d(the)g(origin.)21 b(This)15 b(op)q(eration)h(can)f(b)q(e)h(applied)h(on)e(an)o(y)g(basic)h(datat)o (yp)q(e.)166 1947 y Fm(Discussion:)35 b Fl(This)15 b(needs)g(a)f (rationale.)19 b(It)14 b(can)h(b)q(e)g(used)g(to)f(do)g(one-sided)h (broadcasts)g(\(should)f(it)g(b)q(e)75 2003 y(f\(a,b\))i(=)h(a\)?)27 b(It)17 b(w)o(ould)f(b)q(e)i(wierd)f(in)f(a)h(reduce)i(op)q(eration,)e (and)f(could)h(b)q(e)h(easily)e(done)h(b)o(y)g(the)g(user)h(as)f(a)75 2059 y(user-de\014ned)f(function.)1875 2107 y Fi(?)g Fh(\(Oct\))75 2250 y Fn(3.4)59 b(A)20 b(F)n(o)n(rtran)h(Problem)e(with) h(Register)e(Optimization)75 2399 y Fm(Curren)o(t)c(Status:)j Fl(A)d(new)g(discussion.)189 2552 y Fe(A)n(dvic)n(e)h(to)i(users.)189 2627 y Fq(MPI)d(pro)o(vides)g(op)q(erations)g(whic)o(h)h(ma)o(y)e(b)q (e)i(hidden)h(from)d(the)h(user)g(co)q(de)h(and)f(run)h(in)f(parallel) 189 2684 y(with)g(it,)h(accessing)g(the)f(same)g(memory)g(as)g(user)g (co)q(de.)20 b(Examples)15 b(include)i(the)d(data)g(transfer)-32 46 y Fh(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 4 6 4 5 bop 75 -100 a Fq(4)970 b Fg(CHAPTER)15 b(3.)35 b(MISCELLANY)17 b(F)o(OR)e(1.2)189 45 y Fq(for)10 b(an)h Fj(MPI)p 398 45 14 2 v 16 w(RECV)h Fq(The)f(optimizer)i(of)d(a)h(compiler)i(will)f (assume)f(that)g(it)g(can)h(recognize)g(p)q(erio)q(ds)189 102 y(when)j(a)g(cop)o(y)g(of)g(a)f(v)m(ariable)j(can)e(b)q(e)h(k)o (ept)f(in)h(a)e(register)h(without)g(reloading)h(from)f(or)f(storing) 189 158 y(to)e(memory)l(.)18 b(When)13 b(the)g(user)g(co)q(de)g(is)g(w) o(orking)f(with)h(a)f(register)h(cop)o(y)f(of)g(some)h(v)m(ariable)g (while)189 214 y(the)i(hidden)i(op)q(eration)f(reads)f(or)g(writes)g (the)h(memory)f(cop)o(y)l(,)g(problems)h(o)q(ccur.)k(This)c(section)189 271 y(discusses)g(register)f(optimization)h(pitfalls.)189 346 y(When)f(a)f(v)m(ariable)i(is)f(lo)q(cal)h(to)e(a)g(F)l(ortran)f (subroutine)j(\(i.e.)k(not)14 b(in)h(a)g(COMMON)f(blo)q(c)o(k\),)h(the) 189 403 y(compiler)g(will)i(assume)d(that)g(it)h(cannot)f(b)q(e)h(mo)q (di\014ed)h(b)o(y)f(a)f(called)i(subroutine)f(unless)h(it)f(is)g(an)189 459 y(actual)g(argumen)o(t)g(of)h(the)f(call.)23 b(In)16 b(the)g(most)e(common)i(link)m(age)h(con)o(v)o(en)o(tion,)e(the)h (subroutine)189 515 y(is)d(exp)q(ected)i(to)d(sa)o(v)o(e)h(and)g (restore)g(certain)h(registers.)19 b(Th)o(us,)13 b(the)g(optimizer)i (will)g(assume)e(that)189 572 y(a)k(register)h(whic)o(h)h(held)g(a)e(v) m(alid)j(cop)o(y)d(of)h(suc)o(h)g(a)g(v)m(ariable)h(b)q(efore)f(the)g (call)h(will)g(still)h(hold)e(a)189 628 y(v)m(alid)e(cop)o(y)g(on)f (return.)189 703 y(Normally)22 b(users)g(are)g(not)g(a\017icted)g(with) h(this.)41 b(But)22 b(the)g(user)g(should)h(pa)o(y)f(atten)o(tion)g(to) 189 760 y(this)f(section)g(if)g(in)h(his/her)f(program)e(a)i(bu\013er)f (argumen)o(t)g(to)g(an)h Fj(MPI)p 1498 760 V 16 w(SEND)p Fq(,)f Fj(MPI)p 1746 760 V 16 w(RECV)189 816 y Fq(etc.)25 b(uses)18 b(a)f(name)g(whic)o(h)h(hides)g(the)f(actual)h(v)m(ariables)g (in)o(v)o(olv)o(ed.)27 b Fj(MPI)p 1493 816 V 16 w(BOTTOM)18 b Fq(with)f(an)189 873 y(MPI)p 281 873 V 16 w(Datat)o(yp)q(e)i(con)o (taining)i(absolute)g(addresses)f(is)h(one)f(example.)36 b(Creating)20 b(a)f(datat)o(yp)q(e)189 929 y(whic)o(h)14 b(uses)f(one)g(v)m(ariable)i(as)e(an)g(anc)o(hor)f(and)i(brings)f (along)h(others)e(b)o(y)h(using)h Fj(MPI)p 1659 929 V 16 w(ADDRESS)189 986 y Fq(to)h(determine)j(their)f(o\013sets)e(from)g (the)i(anc)o(hor)f(is)h(another.)22 b(The)17 b(anc)o(hor)f(v)m(ariable) i(w)o(ould)e(b)q(e)189 1042 y(the)g(only)h(one)f(men)o(tioned)h(in)g (the)g(call.)24 b(Also)16 b(atten)o(tion)g(m)o(ust)g(b)q(e)h(pa)o(y)o (ed)f(if)h(MPI)f(op)q(erations)189 1099 y(are)f(used)g(that)g(run)g(in) h(parallel)h(with)f(the)f(user's)g(application.)189 1174 y(The)g(follo)o(wing)h(example)g(sho)o(ws,)e(what)h(F)l(ortran)f (compilers)i(are)f(allo)o(w)o(ed)h(to)e(do:)224 1299 y(This)i(source)f(...)663 b(can)15 b(b)q(e)h(compiled)h(as:)224 1363 y(call)f(MPI)p 399 1363 V 17 w(ADDRESS\(buf,bufaddr,ierr\))189 b(call)16 b(MPI)p 1344 1363 V 17 w(ADDRESS\(buf,...\))224 1420 y(call)g(MPI)p 399 1420 V 17 w(TYPE)p 545 1420 V 16 w(STR)o(UCT\(1,1,bufaddr,)163 b(call)16 b(MPI)p 1344 1420 V 17 w(TYPE)p 1490 1420 V 16 w(STR)o(UCT\(...\))590 1476 y(MPI)p 682 1476 V 17 w(REAL,t)o(yp,ierr\))224 1533 y(call)g(MPI)p 399 1533 V 17 w(TYPE)p 545 1533 V 16 w(COMMIT\(t)o(yp\)) 308 b(call)16 b(MPI)p 1344 1533 V 17 w(TYPE)p 1490 1533 V 16 w(COMMIT\(...\))224 1589 y(v)m(al)p 283 1589 V 17 w(old)g(=)g(buf)681 b(register)15 b(=)h(buf)1169 1646 y(v)m(al)p 1228 1646 V 17 w(old)g(=)f(register)224 1702 y(call)h(MPI)p 399 1702 V 17 w(RECV\(MPI)p 654 1702 V 16 w(BOTTOM,1,t)o(yp,...\))107 b(call)16 b(MPI)p 1344 1702 V 17 w(RECV\(MPI)p 1599 1702 V 16 w(BOTTOM,...\))224 1759 y(v)m(al)p 283 1759 V 17 w(new)g(=)f(buf)664 b(v)m(al)p 1228 1759 V 17 w(new)16 b(=)f(register)189 1884 y(The)j(compiler)g(do)q (es)g(not)g(in)o(v)m(alidate)h(the)f(register)g(b)q(ecause)g(it)g (cannot)f(see)h(that)f Fj(MPI)p 1746 1884 V 16 w(RECV)189 1940 y Fq(c)o(hanges)h(the)g(v)m(alue)i(of)e Fj(buf)p Fq(.)30 b(The)19 b(access)f(of)g Fj(buf)i Fq(is)f(hidden)h(b)o(y)e(the) h(use)f(of)g Fj(MPI)p 1659 1940 V 16 w(ADDRESS)189 1997 y Fq(and)d Fj(MPI)p 362 1997 V 16 w(BOTTOM)p Fq(.)189 2072 y(The)g(next)g(example)h(sho)o(ws)f(extreme,)g(but)g(allo)o(w)o (ed)h(p)q(ossibilities!)224 2197 y(Source)424 b(compiled)17 b(as)322 b(or)15 b(compiled)i(as)224 2261 y(call)f(MPI)p 399 2261 V 17 w(IRECV\(buf,..req\))i(call)f(MPI)p 955 2261 V 16 w(IRECV\(buf,..req\))h(call)f(MPI)p 1510 2261 V 16 w(IRECV\(buf,..req\))779 2318 y(register)e(=)h(buf)278 b(b1)15 b(=)h(buf)224 2374 y(call)g(MPI)p 399 2374 V 17 w(W)-5 b(AIT\(req,..\))104 b(call)17 b(MPI)p 955 2374 V 16 w(W)-5 b(AIT\(req,..\))104 b(call)17 b(MPI)p 1510 2374 V 16 w(W)-5 b(AIT\(req,..\))224 2431 y(b1)15 b(=)h(buf)377 b(b1)15 b(:=)g(register)189 2556 y Fj(MPI)p 274 2556 V 15 w(W)l(AIT)e Fq(or)f(a)g(parallel)i(thread)e(mo)q(di\014es)i Fj(buf)f Fq(b)q(et)o(w)o(een)g(the)f(in)o(v)o(o)q(cation)h(of)f Fj(MPI)p 1648 2556 V 16 w(IRECV)h Fq(and)189 2612 y(the)j(\014nish)h (of)f Fj(MPI)p 528 2612 V 16 w(W)l(AIT)p Fq(.)g(But)g(the)g(compiler)i (cannot)e(see)g(an)o(y)g(p)q(ossibilit)o(y)i(that)e Fj(buf)h Fq(can)f(b)q(e)189 2669 y(c)o(hanged)g(after)e Fj(MPI)p 557 2669 V 16 w(IRECV)i Fq(has)g(\014nished.)23 b(And)16 b(therefore)f(it)h(can)g(sc)o(hedule)h(the)f(load)g(of)f Fj(buf)1967 46 y Fh(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 5 7 5 6 bop 75 -100 a Fg(3.5.)34 b(TR)o(UE)15 b(EXTENT)g(OF)g(D)o(A)l(T)l (A)l(TYPES)958 b Fq(5)189 45 y(earlier)14 b(than)g(t)o(yp)q(ed)g(in)h (the)f(source.)20 b(It)13 b(has)h(no)g(reason)f(to)h(a)o(v)o(oid)f (using)i(a)e(register)h(to)f(hold)i Fj(buf)189 102 y Fq(across)f(the)h(call)i(to)d Fj(MPI)p 625 102 14 2 v 16 w(W)l(AIT)p Fq(.)h(It)g(also)g(can)h(reorder)f(the)g(instructions)h (as)f(in)h(the)f(left)h(case.)189 177 y(T)l(o)c(prev)o(en)o(t)h (instruction)h(reordering)g(or)e(the)h(allo)q(cation)h(of)f(a)g(v)m (ariable)h(or)f(bu\013er)g(in)h(a)e(register,)189 233 y(there)j(are)g(three)g(p)q(ossibilities)k(in)d(p)q(ortable)f(F)l (ortran)f(co)q(de:)243 327 y Fi(\017)23 b Fq(One)d(can)f(put)h(the)g(v) m(ariables)g(and)g(bu\013ers)f(in)o(to)h(common)f(blo)q(c)o(ks;)j(then) e(they)f(will)i(b)q(e)289 383 y(allo)q(cated)d(in)g(the)g(memory)e (while)j(an)o(y)e(library)h(subroutine)h(\(e.g.)25 b(an)18 b(MPI)f(routine\))g(is)289 440 y(called)c(pro)o(vided)f(that)f(the)h (compiler)h(cannot)e(analyze)h(that)f(the)h(library)g(subroutine)h(do)q (es)289 496 y(not)h(use)i(these)f(common)g(blo)q(c)o(ks,)h(or)243 569 y Fi(\017)23 b Fq(one)11 b(can)h(declare)g(them)f(as)g(V)o(OLA)l (TILE,)h(but)g(this)g(is)f(not)g(part)g(of)g(the)g(standards)g(F)l (ortran)289 626 y(77)f(and)h(90,)g(and)g(y)o(our)f(co)q(de)i(will)h (not)d(b)q(e)i(p)q(ortable)f(b)q(ecause)h(there)f(are)g(compilers)h (without)289 682 y(V)o(OLA)l(TILE,)k(and)f(normally)h(it)g(prev)o(en)o (ts)e(an)o(y)h(optimization,)h(or)243 755 y Fi(\017)23 b Fq(one)14 b(can)g(call)h(a)f(dumm)o(y)g(routine)g(with)g(the)h(v)m (ariable)g(or)e(bu\013er)h(as)g(an)g(actual)g(argumen)o(t;)289 812 y(if)22 b(the)h(dumm)o(y)f(routine)h(is)f(not)g(part)g(of)g(the)g (application)i(then)f(the)f(compiler)i(m)o(ust)289 868 y(allo)q(cate)16 b(the)f(v)m(ariable)i(or)e(bu\013er)h(in)g(the)f (memory)g(while)i(that)e(subroutine)h(is)g(executed;)289 925 y Fj(MPI)p 374 925 V 15 w(ADDRESS)h Fq(can)e(b)q(e)h(used)g(as)e (dumm)o(y)h(routine.)189 1019 y(High)e(qualit)o(y)g(F)l(ortran)f (implemen)o(tations)i(ma)o(y)e(pro)o(vide)i(mec)o(hanisms)f(to)f(a)o(v) o(oid)h(the)g(problem,)189 1075 y(without)18 b(disabling)j (optimizations.)31 b(Ho)o(w)o(ev)o(er,)18 b(co)q(de)h(that)f(do)q(es)h (not)f(use)h(one)g(of)f(the)h(three)189 1131 y(solutions)d(describ)q (ed)h(ab)q(o)o(v)o(e)d(ma)o(y)h(not)g(b)q(e)h(p)q(ortable.)189 1207 y(In)h(C,)f(subroutines)i(whic)o(h)g(mo)q(dify)f(v)m(ariables)h (that)e(are)h(not)f(in)i(the)f(parameter)f(list)i(will)g(not)189 1263 y(cause)e(register)h(optimization)g(problems.)25 b(This)17 b(is)g(b)q(ecause)g(taking)g(p)q(oin)o(ters)g(to)f(storage)f (ob-)189 1319 y(jects)d(b)o(y)g(using)h(the)g(&)f(op)q(erator)g(and)g (later)h(referencing)g(the)g(ob)s(jects)f(b)o(y)g(w)o(a)o(y)f(of)h(the) h(p)q(oin)o(ter)f(is)189 1376 y(an)j(in)o(tegral)g(part)g(of)f(the)i (language.)k(A)15 b(C)g(compiler)h(understands)g(the)f(implications,)i (so)e(that)189 1432 y(the)g(problem)h(should)h(not)e(o)q(ccur,)h(in)g (general.)21 b(Ho)o(w)o(ev)o(er,)14 b(some)i(compilers)g(do)g(o\013er)e (optional)189 1489 y(aggressiv)o(e)g(optimization)j(lev)o(els)f(whic)o (h)g(ma)o(y)f(not)f(b)q(e)i(safe.)189 1564 y(\()p Fe(End)f(of)i(advic)n (e)f(to)g(users.)p Fq(\))p 75 1670 827 2 v 75 1727 a(The)f(follo)o (wing)h(advice)h(m)o(ust)d(b)q(e)i(added)g(in)75 1795 y(-)f(MPI)g(2,)g(c)o(hapter)g(?.?)21 b(Handlers,)75 1851 y(-)15 b Fj(MPI)p 190 1851 14 2 v 16 w(ADDRESS)h Fq(\(MPI)f(1.1,)f (page)h(70,)f(line)j(16\),)d(and)h(in)75 1908 y(-)g Fj(MPI)p 190 1908 V 16 w(IRECV)h Fq(\(MPI)f(1.1,)e(\\Non)o(blo)q(c)o(king)k (comm)o(unication",)e(end)h(of)e(page)h(40\))189 2014 y Fe(A)n(dvic)n(e)i(to)i(users.)53 b Fq(T)l(o)17 b(prev)o(en)o(t)g (troubles)h(with)g(the)f(register)g(optimization)i(of)e(the)g(F)l (ortran)189 2071 y(compilers)i(please)g(pa)o(y)e(atten)o(tion)h(to)f (the)h(hin)o(ts)g(in)h(section)g(3.4)e(\\A)g(F)l(ortran)g(Problem)i (with)189 2127 y(register)c(optimization")h(on)f(page)g(3.)k(\()p Fe(End)d(of)g(advic)n(e)g(to)h(users.)p Fq(\))1875 2188 y Fi(?)f Fh(\(Oct\))75 2320 y Fn(3.5)59 b(T)-5 b(rue)20 b(Extent)d(of)k(Datat)n(yp)r(es)75 2469 y Fm(Curren)o(t)14 b(Status:)j Fl(P)o(assed)e(once.)166 2573 y Fq(Supp)q(ose)23 b(w)o(e)f(implemen)o(t)h(gather)f(as)g(a)f(spanning)i(tree)f(implemen)o (ted)i(on)e(top)g(of)g(p)q(oin)o(t-to-)75 2629 y(p)q(oin)o(t)e (routines.)34 b(Since)22 b(the)d(recv)h(bu\013er)g(is)g(only)h(v)m (alid)g(on)f(the)g(ro)q(ot)f(pro)q(cess,)i(one)f(will)h(need)g(to)75 2685 y(allo)q(cate)f(some)f(temp)q(orary)g(space)h(for)f(receiving)i (data)e(on)g(in)o(termediate)i(no)q(des.)33 b(The)20 b(di\016cultly)-32 46 y Fh(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 6 8 6 7 bop 75 -100 a Fq(6)970 b Fg(CHAPTER)15 b(3.)35 b(MISCELLANY)17 b(F)o(OR)e(1.2)75 45 y Fq(is)20 b(in)g(determining)h(the)f(size)g(one)f (needs)i(to)d(allo)q(ciate.)34 b(This)20 b(o)q(ccurs)g(since)h(the)e (user)h(can)f(mo)q(dify)75 102 y(the)g(exten)o(t)h(using)g(the)f Fd(MPI)p 583 102 13 2 v 14 w(UB)g Fq(and)h Fd(MPI)p 841 102 V 14 w(LB)f Fq(v)m(alues.)34 b(The)20 b(writer)f(of)g(the)g(gather) g(routine)h(could)75 158 y(determine)14 b(this)f(information)g(b)o(y)g (deco)q(ding)h(the)f(datat)o(yp)q(e.)18 b(Ho)o(w)o(ev)o(er,)12 b(this)h(is)g(more)g(w)o(ork)e(and)i(more)75 214 y(painful)j(than)e (desired.)20 b(Th)o(us,)14 b(a)g(new)g(function)h(is)g(pro)o(vided)g (whic)o(h)f(returns)g(the)h(true)e(exten)o(t)h(of)g(the)75 271 y(datat)o(yp)q(e.)75 422 y Fj(MPI)p 160 422 14 2 v 16 w(TRUE)p 294 422 V 17 w(EXTENT\(datat)o(yp)q(e,)j(true)p 775 422 V 17 w(lb,)e(true)p 929 422 V 17 w(extent\))117 499 y Fl(IN)155 b Fj(datat)o(yp)q(e)424 b Fl(datat)o(yp)q(e)14 b(to)g(get)g(information)d(on)117 574 y(OUT)108 b Fj(true)p 396 574 V 17 w(lb)461 b Fl(true)15 b(lo)o(w)o(er)e(b)q(ound)h(of)g (datat)o(yp)q(e)117 649 y(OUT)108 b Fj(true)p 396 649 V 17 w(extent)379 b Fl(true)15 b(size)g(of)e(datat)o(yp)q(e)75 774 y Fk(int)23 b(MPI)p 245 774 15 2 v 17 w(True)p 358 774 V 17 w(extent\(MPI)p 615 774 V 16 w(Datatype)f(datatype,)h(MPI)p 1156 774 V 17 w(Aint)g(*true)p 1412 774 V 17 w(lb,)g(MPI)p 1596 774 V 17 w(Aint)393 830 y(*true)p 516 830 V 17 w(extent\))75 917 y(MPI)p 150 917 V 17 w(TRUE)p 263 917 V 16 w(EXTENT\(DATATYPE,)f (TRUE)p 781 917 V 17 w(LB,)h(TRUE)p 989 917 V 17 w(EXTENT,)g(IERROR\)) 170 973 y(INTEGER)g(DATATYPE,)g(TRUE)p 699 973 V 17 w(LB,)g(TRUE)p 907 973 V 17 w(EXTENT,)g(IERROR)166 1060 y Fj(true)p 244 1060 14 2 v 17 w(lb)14 b Fq(returns)f(the)g(o\013set)g(of)g(the)g (lo)o(w)o(est)g(unit)h(of)f(store)g(whic)o(h)h(is)g(addressed)g(b)o(y)f (the)h(datat)o(yp)q(e.)75 1116 y Fj(true)p 153 1116 V 17 w(extent)k Fq(returns)f(the)g(true)f(size)i(of)e(the)h(datat)o(yp)q (e.)24 b(The)17 b Fj(true)p 1244 1116 V 17 w(extent)i Fq(is)e(the)g(minim)o(um)h(n)o(um)o(b)q(er)75 1173 y(of)13 b(b)o(ytes)g(of)h(memory)f(necessary)g(to)g(hold)i(a)e(datat)o(yp)q(e)g (and)h(ignores)g(the)f Fj(LB)h Fq(and)g Fj(UB)g Fq(that)e(ma)o(y)h(ha)o (v)o(e)75 1229 y(b)q(een)j(used)g(in)g(creating)g(the)f(datat)o(yp)q (e.)166 1326 y Fm(Alternativ)o(es)o(:)166 1376 y Fl(If)e(w)o(e)i(w)o (an)o(t)e(to)h(b)q(e)g(consisten)o(t)h(with)f(the)g(MPI1)g(exten)o(t)h (functions,)f(w)o(e)g(w)o(ould)f(need)i(three)g(functions:)166 1426 y Fd(MPI)p 243 1426 13 2 v 14 w(TRUE)p 366 1426 V 15 w(LB\(datat)o(yp)q(e,)g(lb\))166 1475 y(MPI)p 243 1475 V 14 w(TRUE)p 366 1475 V 15 w(UB\(datat)o(yp)q(e,)f(ub\))166 1525 y(MPI)p 243 1525 V 14 w(TRUE)p 366 1525 V 15 w(EXTENT\(datat)o(yp) q(e,)h(extent\))166 1575 y Fl(On)f(the)h(other)f(hand,)f(w)o(e)i(ma)o (y)c(not)j(w)o(an)o(t)g(to)f(mak)o(e)g(the)h(same)f(mistak)o(e)g(t)o (wice...)75 1766 y Fn(3.6)59 b(Datat)n(yp)r(e)18 b(extent)75 1914 y Fm(Curren)o(t)c(Status:)j Fl(P)o(assed)e(once.)166 2018 y Fq(MPI)21 b(allo)o(ws)h(one)f(to)g(c)o(hange)h(the)f(exten)o(t)g (of)g(a)h(datat)o(yp)q(e,)g(using)g(lo)o(w)o(er)f(b)q(ound)h(and)g(upp) q(er)75 2074 y(b)q(ound)e(mark)o(ers)e(\()p Fd(MPI)p 490 2074 V 14 w(LB)h Fq(and)g Fd(MPI)p 740 2074 V 14 w(UB)p Fq(\).)f(This)h(is)h(useful,)g(as)f(it)g(allo)o(ws)g(to)f(con)o (trol)h(the)g(stride)g(of)75 2131 y(successiv)o(e)e(datat)o(yp)q(es)e (that)g(are)g(replicated)i(b)o(y)e(datat)o(yp)q(e)g(constructors,)g(or) g(are)g(replicated)i(b)o(y)f(the)75 2187 y Fj(count)k Fq(argumen)o(t)e(in)i(a)e(send)h(or)g(reciev)o(e)g(call.)32 b(Ho)o(w)o(ev)o(er,)19 b(the)f(curren)o(t)h(mec)o(hanism)h(for)e(ac)o (hieving)75 2244 y(it)h(is)h(painful;)i(also)d(it)h(is)f(restrictiv)o (e,)i(as)d Fd(MPI)p 906 2244 V 15 w(LB)h Fq(and)g Fd(MPI)p 1157 2244 V 14 w(UB)g Fq(are)f(\\stic)o(ky":)28 b(once)19 b(presen)o(t)g(in)h(a)75 2300 y(datat)o(yp)q(e,)d(they)h(cannot)g(b)q (e)g(o)o(v)o(eriden)g(\(e.g.,)f(the)h(upp)q(er)h(b)q(ound)f(can)g(b)q (e)g(mo)o(v)o(ed)g(up,)g(b)o(y)g(adding)g(a)75 2357 y(new)13 b Fd(MPI)p 243 2357 V 14 w(UB)g Fq(mark)o(er,)f(but)h(cannot)g(b)q(e)g (mo)o(v)o(ed)g(do)o(wn)f(b)q(elo)o(w)i(an)f(existing)h Fj(MPI)p 1474 2357 14 2 v 15 w(UB)g Fq(mark)o(er\).)k(A)13 b(new)75 2413 y(function)j(is)g(pro)o(vided)g(to)e(facilitate)i(these)g (c)o(hanges.)1967 46 y Fh(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 7 9 7 8 bop 75 -100 a Fg(3.7.)34 b(MPI-1.0)14 b(AND)i(MPI-1.1)e(CLARIFICA)l (TIONS)802 b Fq(7)75 45 y Fj(MPI)p 160 45 14 2 v 16 w(SET)p 259 45 V 17 w(EXTENT\(datat)o(yp)q(e,)16 b(lb,)f(extent\))117 122 y Fl(INOUT)62 b Fj(datat)o(yp)q(e)424 b Fl(datat)o(yp)q(e)14 b(to)g(up)q(date)h(exten)o(t)117 197 y(IN)155 b Fj(true)p 396 197 V 17 w(lb)461 b Fl(new)15 b(lo)o(w)o(er)e(b)q(ound)h(of)f (datat)o(yp)q(e)117 273 y(IN)155 b Fj(true)p 396 273 V 17 w(extent)379 b Fl(new)15 b(exten)o(t)f(of)g(datat)o(yp)q(e)75 397 y Fk(int)23 b(MPI)p 245 397 15 2 v 17 w(Set)p 334 397 V 17 w(extent\(MPI)p 591 397 V 16 w(Datatype)g(datatype,)f(MPI)p 1132 397 V 17 w(Aint)h(lb,)h(MPI)p 1436 397 V 17 w(Aint)f(extent\))75 483 y(MPI)p 150 483 V 17 w(TRUE)p 263 483 V 16 w(EXTENT\(DATATYPE,)f (LB,)i(EXTENT,)e(IERROR\))170 540 y(INTEGER)h(DATATYPE,)g(LB,)g (EXTENT,)g(IERROR)166 626 y Fq(Resets)13 b(the)g(lo)o(w)o(er)g(b)q (ound)h(and)f(the)h(exten)o(t)e(\(and,)h(hence,)h(the)g(upp)q(er)g(b)q (ound\))f(of)g Fj(datat)o(yp)q(e)p Fq(.)21 b(The)75 683 y(new)e(settings)f(will)i(a\013ect)e(the)h(outcome)f(of)g(send)h(or)f (receiv)o(e)h(op)q(erations)g(that)e(use)i Fj(datat)o(yp)q(e)p Fq(,)i(and)75 739 y(a\013ects)14 b(the)i(v)m(alue)g(of)f(datat)o(yp)q (es)g(constructed)g(using)h Fj(datat)o(yp)q(e)h Fq(as)e(an)g(agumen)o (t,)f(after)h(the)g(call)i(w)o(as)75 796 y(made.)i(Ho)o(w)o(ev)o(er,)11 b(datat)o(yp)q(es)h(that)f(w)o(ere)h(previously)i(constructed)e(using)h Fj(datat)o(yp)q(e)h Fq(are)e(not)g(a\013ected.)189 902 y Fe(R)n(ationale.)189 977 y Fq(This)h(is)g(consisten)o(t)f(to)g(the)h (MPI)f(statemen)o(t)g(that)g(\\the)g(system)g(b)q(eha)o(v)o(es)h(as)f (if)h(input)g(datat)o(yp)q(e)189 1034 y(argumen)o(ts)h(to)h(deriv)o(ed) h(datat)o(yp)q(e)e(constructors)h(are)g(passed)g(b)o(y)g(v)m(alue".)189 1109 y(\()p Fe(End)g(of)i(r)n(ationale.)p Fq(\))189 1215 y Fe(A)n(dvic)n(e)g(to)h(users.)51 b Fq(It)17 b(is)g(recommended)h (that)f(uses)g(use)g(the)g(function)h Fj(MPI)p 1571 1215 14 2 v 16 w(SET)p 1670 1215 V 17 w(EXTENT)p Fq(,)189 1271 y(rather)c(than)h Fd(MPI)p 508 1271 13 2 v 15 w(LB)g Fq(and)g Fd(MPI)p 751 1271 V 15 w(UB)f Fq(mark)o(ers.)189 1347 y(\()p Fe(End)h(of)i(advic)n(e)f(to)g(users.)p Fq(\))1875 1408 y Fi(?)g Fh(\(Oct\))75 1540 y Fn(3.7)59 b(MPI-1.0)19 b(and)h(MPI-1.1)f(Cla)n(ri\014cations)75 1643 y Ff(3.7.1)49 b(Cla)o(ri\014cation)18 b(of)e(MPI)p 627 1643 15 2 v 18 w(FINALIZE)1875 1672 y Fi(>)g Fh(\(Oct\))75 1776 y Fm(Curren)o(t)e(Status:)j Fl(Not)d(v)o(oted)g(on.)166 1880 y Fq(This)j(routine)f(cleans)h(up)f(all)h(MPI)f(state.)22 b(Eac)o(h)16 b(pro)q(cess)g(m)o(ust)f(call)i Fj(MPI)p 1485 1880 14 2 v 16 w(FINALIZE)e Fq(b)q(efore)i(it)75 1936 y(exits.)j(Once)c(this)f(routine)g(returns,)f(no)h(MPI)g(routine)g (\(ev)o(en)g(not)f Fj(MPI)p 1338 1936 V 16 w(INIT)p Fq(\))f(ma)o(y)h(b) q(e)i(called.)21 b(Eac)o(h)75 1992 y(pro)q(cess)11 b(m)o(ust)e (complete)j(an)o(y)e(p)q(ending)i(comm)o(unication)f(it)g(initiated)h (b)q(efore)e(it)h(calls)g Fj(MPI)p 1656 1992 V 16 w(FINALIZE)p Fq(.)75 2049 y(If)16 b(the)g(call)h(returns,)e(eac)o(h)h(pro)q(cess)g (ma)o(y)f(con)o(tin)o(ue)i(lo)q(cal)f(computations,)g(or)f(exit,)h (without)g(partici-)75 2105 y(pating)g(in)g(further)g Fj(MPI)f Fq(comm)o(unication)h(with)g(other)f(pro)q(cesses.)22 b Fj(MPI)p 1362 2105 V 16 w(FINALIZE)14 b Fq(is)j(collectiv)o(e)g(on)75 2162 y Fj(MPI)p 160 2162 V 16 w(COMM)p 318 2162 V 16 w(W)o(ORLD)p Fq(.)1369 b Fi(>)16 b Fh(\(Oct\))166 2218 y Fq(It)f(is)h(not)f(required)h(that)e Fj(MPI)p 703 2218 V 16 w(Finalize)h Fq(return,)g(ev)o(en)g(on)h(one)f(pro)q(cess.)1875 2220 y Fi(?)h Fh(\(Oct\))189 2325 y Fe(A)n(dvic)n(e)11 b(to)i(implementors.)38 b Fq(Ev)o(en)11 b(though)g(a)g(pro)q(cess)h (has)f(completed)i(all)f(the)f(comm)o(unication)189 2381 y(it)j(initiated,)h(suc)o(h)g(comm)o(unication)f(ma)o(y)f(not)h(y)o(et) g(b)q(e)g(completed)h(from)e(the)h(viewp)q(oin)o(t)h(of)f(the)189 2437 y(underlying)j(MPI)f(system.)k(E.g.,)14 b(a)h(blo)q(c)o(king)i (send)f(ma)o(y)f(ha)o(v)o(e)g(completed,)i(ev)o(en)e(though)h(the)189 2494 y(data)k(is)h(still)h(bu\013ered)f(at)f(the)h(sender.)37 b(The)20 b(MPI)h(implemen)o(tation)h(m)o(ust)e(ensure)h(that)f(a)189 2550 y(pro)q(cess)14 b(has)g(completed)h(an)o(y)f(in)o(v)o(olv)o(emen)o (t)g(in)h(MPI)f(comm)o(unication)h(b)q(efore)f Fj(MPI)p 1669 2550 V 16 w(FINALIZE)189 2607 y Fq(returns.)26 b(Th)o(us,)17 b(if)h(a)e(pro)q(cess)i(exits)g(after)e(the)h(call)i(to)d Fj(MPI)p 1271 2607 V 16 w(FINALIZE)p Fq(,)g(this)i(will)h(not)e(cause) 189 2663 y(an)e(ongoing)g(comm)o(unication)h(to)e(fail.)21 b(\()p Fe(End)16 b(of)g(advic)n(e)g(to)h(implementors.)p Fq(\))-32 46 y Fh(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 8 10 8 9 bop 75 -100 a Fq(8)970 b Fg(CHAPTER)15 b(3.)35 b(MISCELLANY)17 b(F)o(OR)e(1.2)189 45 y Fe(A)n(dvic)n(e)i(to)i(implementors.)54 b Fq(Although)18 b(it)g(is)g(not)f(required,)i(it)f(has)f(b)q(een)i (found)f(that)f(users)189 102 y(exp)q(ect)f(that)f(at)f(least)i(one)f (pro)q(cess)h(return,)f(so)g(that)g(they)h(can)f(kno)o(w)g(that)g(the)g (MPI)h(p)q(ortion)189 158 y(of)e(the)g(computation)g(is)h(o)o(v)o(er.)k (In)c(addition,)g(in)g(a)f(POSIX)i(en)o(vironmen)o(t,)e(they)h(ma)o(y)e (desire)j(to)189 214 y(supply)f(an)f(exit)g(co)q(de)h(for)e(eac)o(h)h (pro)q(cess)g(that)f(returns)h(from)f Fj(MPI)p 1357 214 14 2 v 16 w(FINALIZE)p Fq(.)g(\()p Fe(End)h(of)h(advic)n(e)189 271 y(to)h(implementors.)p Fq(\))-117 332 y Fi(>)f Fh(\(Oct\))75 464 y Fn(3.8)59 b(Determining)19 b(Whether)g(MPI)h(Has)g(Finished)75 613 y Fm(Curren)o(t)14 b(Status:)j Fl(No)d(v)o(otes.)166 716 y Fq(One)i(of)f(the)h(goals)f(of)h Fj(MPI)f Fq(w)o(as)f(to)h(allo)o (w)h(for)f(la)o(y)o(ered)h(libraries.)23 b(In)16 b(order)f(for)g(a)h (library)g(to)f(do)75 773 y(this)k(cleanly)l(,)i(it)e(needs)g(to)f(kno) o(w)g(if)h Fj(MPI)f Fq(is)i(activ)o(e.)30 b(In)19 b Fj(MPI-1)f Fq(the)h(function)g Fj(MPI)p 1590 773 V 16 w(Initialized)g Fq(w)o(as)75 829 y(pro)o(vided)c(to)g(tell)g(if)g Fj(MPI)f Fq(had)h(b)q(een)h(initialzed.)23 b(The)15 b(problem)g(arises)g(in)g (kno)o(wing)g(if)g Fj(MPI)f Fq(has)h(b)q(een)75 886 y(\014nalized.)31 b(Once)19 b Fj(MPI)f Fq(has)g(b)q(een)h(\014nalized)h(it)f(is)g(no)f (longer)g(activ)o(e)h(and)f(cannot)g(b)q(e)h(restarted.)28 b(A)75 942 y(library)16 b(needs)g(to)e(b)q(e)i(able)g(to)e(determine)i (this)f(to)g(act)f(accordingly)l(.)22 b(Do)14 b(ac)o(hiev)o(e)i(this)f (the)g(follo)o(wing)75 999 y(function)h(is)g(needed:)75 1150 y Fj(MPI)p 160 1150 V 16 w(FINALIZED\(\015ag\))117 1227 y Fl(OUT)108 b Fj(\015ag)518 b Fl(true)15 b(if)e Fd(MPI)g Fl(w)o(as)h(\014nalized)g(\(logical\).)75 1351 y Fk(int)23 b(MPI)p 245 1351 15 2 v 17 w(Finalized\(int)f(*flag\))75 1438 y(MPI)p 150 1438 V 17 w(FINALIZED\(flag,)g(IERROR\)\))170 1494 y(FLAG,)i(IERROR)75 1581 y(int)f(MPI::Finalized\(bool)f (&flag\)\)?????)166 1667 y Fq(This)17 b(function)g(returns)f(true)g(if) h Fj(MPI)p 833 1667 14 2 v 16 w(FINALIZE)e Fq(has)h(returned.)23 b(Th)o(us,)16 b(it)h(will)h(return)e(false)g(if)75 1724 y(called)k(in)f(a)f(callbac)o(k)h(routine)f(in)o(v)o(ok)o(ed)h(b)o(y)f Fj(MPI)p 960 1724 V 16 w(FINALIZE)p Fq(.)f(It)h(is)g(legal)h(to)f(call) h Fj(MPI)p 1636 1724 V 16 w(FINALIZED)75 1780 y Fq(b)q(efore)c Fj(MPI)p 296 1780 V 16 w(INIT)g Fq(and)g(after)g Fj(MPI)p 694 1780 V 15 w(FINALIZE)p Fq(.)-1030 b Fi(?)15 b Fh(\(Oct\))-117 1839 y Fi(>)g Fh(\(Oct\))75 1902 y Ff(3.8.1)49 b(Cla)o(ri\014cation)18 b(of)e(status)f(after)h(MPI)p 875 1902 15 2 v 18 w(Isend)75 2035 y Fm(Curren)o(t)e(Status:)j Fl(No)d(v)o(otes.)166 2138 y Fq(A)22 b Fj(status)i Fq(returned)e(b)o(y)g(a)g(call)h(to)e Fj(MPI)p 900 2138 14 2 v 16 w(W)l(AIT)p Fq(,)g Fj(MPI)p 1145 2138 V 16 w(TEST)p Fq(,)h(or)f(an)o(y)h(of)f(the)h(other)g(deriv)o (ed)75 2195 y(functions,)j(where)f(the)f Fj(request)h Fq(corresp)q(onds)g(to)e(a)h(send)g(call,)j(is)e(unde\014ned,)i(with)d (t)o(w)o(o)f(excep-)75 2251 y(tions:)36 b(The)24 b(error)f(status)f (\014eld)j(will)g(con)o(tain)e(v)m(alid)i(information)f(if)f(the)h(w)o (ait)f(or)g(test)f(call)j(re-)75 2308 y(turned)20 b(with)g Fd(MPI)p 411 2308 13 2 v 15 w(ERROR)p 563 2308 V 13 w(IN)p 617 2308 V 15 w(ST)m(A)m(TUS)p Fq(;)f(and)h(the)g(returned)g(status)f (can)h(b)q(e)g(querried)h(b)o(y)f(the)g(call)75 2364 y Fj(MPI)p 160 2364 14 2 v 16 w(TEST)p 290 2364 V 16 w(CANCELLED)p Fq(.)-690 b Fi(?)15 b Fh(\(Oct\))75 2486 y Ff(3.8.2)49 b(Cla)o(ri\014cation)18 b(of)e(MPI)p 627 2486 15 2 v 18 w(INTERCOMM)p 936 2486 V 20 w(CREA)l(TE)75 2619 y Fm(Curren)o(t)e(Status:)j Fl(P)o(assed)e(once.)1967 46 y Fh(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 9 11 9 10 bop 75 -100 a Fg(3.8.)34 b(DETERMINING)15 b(WHETHER)h Fj(MPI)e Fg(HAS)i(FINISHED)630 b Fq(9)75 45 y Fj(The)11 b(Problem:)44 b Fq(The)12 b(MPI)e(1.1)g(standard)h(sa)o(ys,)f(in)i(the) f(discussion)h(of)f Fj(MPI)p 1389 45 14 2 v 16 w(INTERCOMM)p 1679 45 V 16 w(CREA)l(TE)p Fq(,)75 102 y(b)q(oth)k(that)189 195 y(The)g(groups)g(m)o(ust)g(b)q(e)g(disjoin)o(t)75 289 y(and)g(that)189 383 y(The)g(leaders)h(ma)o(y)e(b)q(e)i(the)g(same) e(pro)q(cess.)75 477 y(T)l(o)d(further)h(m)o(uddy)g(the)g(w)o(aters,)e (the)i(reason)f(giv)o(en)i(for)e(\\The)g(groups)g(m)o(ust)h(b)q(e)g (disjoin)o(t")g(is)g(based)g(on)75 533 y(concerns)18 b(ab)q(out)g(the)g(implmen)o(tation)g(of)g Fj(MPI)p 922 533 V 15 w(INTERCOMM)p 1211 533 V 17 w(CREA)l(TE)h Fq(that)e(are)g(not) h(applicable)75 590 y(for)d(the)g(case)g(where)h(the)f(leaders)h(are)f (the)g(same)g(pro)q(cess.)75 710 y Fj(The)h(Fix:)45 b Fq(Delete)15 b(the)h(text:)189 804 y(\(the)f(t)o(w)o(o)e(leaders)j (could)h(b)q(e)e(the)h(same)f(pro)q(cess\))75 897 y(from)f(the)i (discussion)h(of)d Fj(MPI)p 610 897 V 16 w(INTERCOMM)p 900 897 V 17 w(CREA)l(TE)p Fq(.)166 954 y(Replace)j(the)e(text:)189 1048 y(All)f(in)o(ter-comm)o(unicator)g(constructors)e(are)h(blo)q(c)o (king)h(and)g(require)g(that)e(the)i(lo)q(cal)g(and)189 1104 y(remote)g(groups)h(b)q(e)h(disjoin)o(t)g(in)g(order)f(to)f(a)o(v) o(oid)h(deadlo)q(c)o(k.)75 1198 y(with)189 1292 y(All)f(in)o(ter-comm)o (unicator)g(constructors)e(are)h(blo)q(c)o(king)h(and)g(require)g(that) e(the)i(lo)q(cal)g(and)189 1348 y(remote)g(groups)h(b)q(e)h(disjoin)o (t.)289 1454 y Fe(A)n(dvic)n(e)e(to)h(users.)40 b Fq(The)14 b(groups)g(m)o(ust)f(b)q(e)i(disjoin)o(t)g(for)e(sev)o(eral)h(reasons.) 19 b(Primar-)289 1511 y(ily)l(,)12 b(this)f(is)g(the)f(in)o(ten)o(t)h (of)f(the)g(in)o(tercomm)o(unicators)h({)f(to)g(pro)o(vide)h(a)f(comm)o (unicator)289 1567 y(for)17 b(comm)o(unication)i(b)q(et)o(w)o(een)g (disjoin)o(t)f(groups.)29 b(This)19 b(is)f(re\015ected)h(in)g(the)g (de\014-)289 1624 y(nition)e(of)f Fj(MPI)p 559 1624 V 15 w(INTERCOMM)p 848 1624 V 17 w(MERGE)p Fq(,)g(whic)o(h)h(allo)o(ws)g (the)f(user)g(to)f(con)o(trol)h(the)289 1680 y(ranking)k(of)g(the)g (pro)q(cesses)g(in)h(the)g(created)f(in)o(tracomm)o(unicator;)h(this)g (ranking)289 1737 y(mak)o(es)c(little)i(sense)f(if)g(the)g(groups)g (are)f(not)h(disjoin)o(t.)28 b(In)18 b(addition,)h(the)f(natural)289 1793 y(extension)c(of)g(collectiv)o(e)i(op)q(erations)e(to)f(in)o (tercomm)o(unicators)h(\(b)q(eing)h(considered)289 1850 y(for)i(MPI-2\))h(mak)o(es)f(the)h(most)g(sense)g(when)h(the)f(groups)g (are)f(disjoin)o(t.)30 b(It)18 b(is)g(the)289 1906 y(in)o(ten)o(t)i(of) g(the)g(MPI)g(de\014nition)i(not)e(to)f(preclude)j(future)f (extensions.)35 b(\()p Fe(End)20 b(of)289 1963 y(advic)n(e)c(to)g (users.)p Fq(\))75 2084 y Ff(3.8.3)49 b(Cla)o(ri\014cation)18 b(of)e(Binding)i(of)f(MPI)p 854 2084 15 2 v 18 w(T)l(yp)q(e)p 971 2084 V 18 w(size)75 2217 y Fm(Curren)o(t)d(Status:)j Fl(P)o(assed)e(once.)166 2321 y Fq(This)i(clari\014cation)h(is)f (needed)h(in)f(the)g(MPI-1)f(description)i(of)e Fj(MPI)p 1369 2321 14 2 v 16 w(T)l(yp)q(e)p 1477 2321 V 17 w(size)p Fq(,)h(since)g(the)g(issue)75 2378 y(rep)q(eatedly)f(arises.)k(It)c(is) f(a)g(clari\014cation)i(of)e(the)g(binding.)189 2484 y Fe(A)n(dvic)n(e)20 b(to)j(users.)76 b Fq(The)21 b(binding)i(of)e Fj(MPI)p 1004 2484 V 16 w(T)l(yp)q(e)p 1112 2484 V 17 w(size)p Fq(,)i(sp)q(eci\014cally)l(,)i(whether)d(the)f(output)189 2540 y(argumen)o(t)e(in)i(C)g(is)f(of)g(t)o(yp)q(e)h Fk(int)f Fq(as)g(in)h(the)f(curren)o(t)h(standard)f(or)f(should)j(b)q (e)f(c)o(hanged)g(to)189 2597 y(something)14 b(else,)i(has)e(b)q(een)i (extensiv)o(ely)f(discussed)h(b)o(y)f(the)g(MPI)f(F)l(orum.)19 b(The)c(\014nal)g(decision)189 2653 y(is)g(to)g(lea)o(v)o(e)g(it)h(an)f Fk(int)p Fq(.)k(\()p Fe(End)d(of)g(advic)n(e)g(to)h(users.)p Fq(\))-32 46 y Fh(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 10 12 10 11 bop 75 -100 a Fq(10)952 b Fg(CHAPTER)16 b(3.)29 b(MISCELLANY)17 b(F)o(OR)e(1.2)75 45 y Ff(3.8.4)49 b(Cla)o (ri\014cation)18 b(of)e(MPI)p 627 45 15 2 v 18 w(REDUCE)75 131 y Fq(The)f(curren)o(t)h(text)e(on)h(p.)20 b(115)15 b(lines)i(25-28)d(from)g(MPI-1.1)h(\(June)h(12,)e(1995\))f(sa)o(ys:)166 187 y(The)j Fj(datat)o(yp)q(e)i Fq(argumen)o(t)e(of)f Fj(MPI)p 783 187 14 2 v 16 w(REDUCE)i Fq(m)o(ust)f(b)q(e)g(compatible)i (with)e Fj(op)p Fq(.)23 b(Prede\014ned)18 b(op-)75 244 y(erators)h(w)o(ork)g(only)i(with)f(the)g Fj(MPI)g Fq(t)o(yp)q(es)g (listed)h(in)g(Sec.)35 b(4.9.2)18 b(and)j(Sec.)35 b(4.9.3.)d (User-de\014ned)75 300 y(op)q(erators)14 b(ma)o(y)h(op)q(erate)g(on)g (general,)g(deriv)o(ed)h(datat)o(yp)q(es.)166 357 y(This)g(text)e(is)i (c)o(hanged)g(to:)166 413 y(The)g Fj(datat)o(yp)q(e)i Fq(argumen)o(t)e(of)f Fj(MPI)p 783 413 V 16 w(REDUCE)i Fq(m)o(ust)f(b)q(e)g(compatible)i(with)e Fj(op)p Fq(.)23 b(Prede\014ned)18 b(op-)75 470 y(erators)c(w)o(ork)h(only)h(with)f(the) h Fj(MPI)f Fq(t)o(yp)q(es)g(listed)i(in)f(Sec.)21 b(4.9.2)14 b(and)h(Sec.)21 b(4.9.3.)e(F)l(urthermore,)c(the)75 526 y Fj(datat)o(yp)q(e)20 b Fq(and)e Fj(op)h Fq(giv)o(en)f(for)g (prede\014ned)i(op)q(erators)d(m)o(ust)h(b)q(e)h(the)f(same)g(on)g(all) h(pro)q(cesses.)30 b(User-)75 583 y(de\014ned)15 b(op)q(erators)e(ma)o (y)g(op)q(erate)g(on)h(general,)g(deriv)o(ed)h(datat)o(yp)q(es)e (without)g(the)h(restrictions)g(ab)q(o)o(v)o(e)75 639 y(for)h(prede\014ned)h(op)q(erators.)-681 b Fi(>)15 b Fh(\(Oct\))166 695 y Fq(Note)10 b(that)f(it)i(is)f(p)q(ossible)i(for)e (users)g(to)f(supply)j(di\013eren)o(t)e(user-de\014ned)i(op)q(erations) e(to)g Fj(MPI)p 1750 695 V 16 w(REDUCE)75 752 y Fq(in)23 b(eac)o(h)e(pro)q(cess.)40 b Fj(MPI)21 b Fq(do)q(es)h(not)f(de\014ne)i (whic)o(h)g(op)q(erations)e(are)h(used)g(in)h(this)f(case)f(on)h(whic)o (h)75 808 y(op)q(erands.)189 915 y Fe(A)n(dvic)n(e)e(to)h(users.)71 b Fq(Users)20 b(should)i(mak)o(e)d(no)i(assumptions)f(ab)q(out)g(ho)o (w)g Fj(MPI)p 1638 915 V 16 w(RECUCE)h Fq(is)189 971 y(implemen)o(ted.)29 b(Safest)17 b(is)h(to)f(ensure)i(that)e(the)g (same)h(function)g(is)g(passed)g(to)f Fj(MPI)p 1685 971 V 16 w(REDUCE)189 1028 y Fq(b)o(y)e(eac)o(h)g(pro)q(cess.)20 b(\()p Fe(End)c(of)g(advic)n(e)g(to)h(users.)p Fq(\))-117 1089 y Fi(?)e Fh(\(Oct\))75 1221 y Fn(3.9)59 b(New)20 b(Datat)n(yp)r(es)75 1324 y Ff(3.9.1)49 b(Simple)17 b(Struct)d(T)l(yp)q (es)75 1457 y Fm(Curren)o(t)g(Status:)j Fl(Not)d(v)o(oted)g(on)f(y)o (et.)75 1655 y Fj(MPI)p 160 1655 V 16 w(TYPE)p 293 1655 V 17 w(SIMPLE)p 469 1655 V 15 w(STRUCT\(count,)k(a)o(rra)o(y)p 908 1655 V 14 w(of)p 959 1655 V 16 w(blo)q(cklengths,)h(a)o(rra)o(y)p 1334 1655 V 14 w(of)p 1385 1655 V 16 w(t)o(yp)q(es,)e(newt)o(yp)q(e\)) 117 1732 y Fl(IN)155 b Fj(count)482 b Fl(n)o(um)o(b)q(er)13 b(of)h(blo)q(c)o(ks)g(\(in)o(teger\))117 1807 y(IN)155 b Fj(a)o(rra)o(y)p 416 1807 V 15 w(of)p 468 1807 V 16 w(blo)q(cklengths)191 b Fl(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h (eac)o(h)g(blo)q(c)o(k)g(\(arra)o(y)g(of)f(in)o(teger\))117 1882 y(IN)155 b Fj(a)o(rra)o(y)p 416 1882 V 15 w(of)p 468 1882 V 16 w(t)o(yp)q(es)327 b Fl(t)o(yp)q(e)15 b(of)e(elemen)o(ts)h (in)f(eac)o(h)h(blo)q(c)o(k)g(\(arra)o(y)g(of)f(handle\))117 1958 y(OUT)108 b Fj(newt)o(yp)q(e)433 b Fl(new)15 b(datat)o(yp)q(e)f (\(handle\))166 2082 y Fq(Eac)o(h)c(t)o(yp)q(e)h(in)g(the)g Fj(a)o(rra)o(y)p 592 2082 V 14 w(of)p 643 2082 V 16 w(t)o(yp)q(es)h Fq(list)g(m)o(ust)e(b)q(e)h(a)f Fe(p)n(ortable)h Fq(datat)o(yp)q(e:)17 b(I.e.,)10 b(a)h(basic)g(datat)o(yp)q(e,)f(or)75 2138 y(a)g(datat)o(yp)q(e)f(constructed)i(from)e(a)h(p)q(ortable)h(datat)o (yp)q(e)e(using)i(one)f(of)g(the)g(constructors)g Fj(MPI)p 1681 2138 V 15 w(TYPE)p 1813 2138 V 17 w(CONTIGUOUS,)75 2195 y(MPI)p 160 2195 V 16 w(TYPE)p 293 2195 V 17 w(VECTOR,)16 b(MPI)p 600 2195 V 15 w(TYPE)p 732 2195 V 17 w(INDEXED,)f Fq(or)g Fj(MPI)p 1110 2195 V 16 w(TYPE)p 1243 2195 V 16 w(SIMPLE)p 1418 2195 V 16 w(STRUCT)p Fq(.)166 2251 y(This)22 b(t)o(yp)q(e)g(constructor)f(is)i(similar)f(to)g Fj(MPI)p 988 2251 V 15 w(TYPE)p 1120 2251 V 17 w(STRUCT)p Fq(,)g(except)h(that)e(the)h(user)g(do)q(es)75 2308 y(not)e(supply)h (an)f(arra)o(y)e(of)i(displacemen)o(ts.)35 b(Instead,)22 b(the)e(successiv)o(e)h(blo)q(c)o(ks)f(are)g(assumed)g(to)f(b)q(e)75 2364 y(con)o(tiguous.)g(P)o(adding)11 b(spaces)h(are)e(added,)j (according)e(to)g(the)g(default)h(alignmen)o(t)g(rules)g(for)e (structure)75 2421 y(comp)q(onen)o(ts.)21 b(This)c(facilitates)f(the)g (construction)g(of)f(datat)o(yp)q(es)g(for)g(structures,)g(as)h(the)f (user)h(need)75 2477 y(not)f(compute)g(displacemen)o(ts.)-743 b Fi(>)15 b Fh(\(Oct\))166 2534 y Fq(This)h(can)f(probably)h(b)q(e)g (more)e(elegan)o(tly)i(expressed)g(in)g(terms)f(of)g(exten)o(ts.)-117 2536 y Fi(?)g Fh(\(Oct\))75 2640 y Fb(Example)j(3.1)k Fq(The)14 b(follo)o(wing)g(co)q(de)f(declares)h(a)f(structure)g(t)o(yp) q(e)g(and)h(creates)f(a)g(datat)o(yp)q(e)f(for)h(that)75 2696 y(structure)i(t)o(yp)q(e.)1967 46 y Fh(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 11 13 11 12 bop 75 -100 a Fg(3.9.)34 b(NEW)15 b(D)o(A)l(T)l(A)l(TYPES)1245 b Fq(11)75 45 y Fk(struct)23 b(record)g({)147 102 y(char)g(name;)147 158 y(double)g(position[3];)147 214 y(float)47 b(mass;)147 271 y(})75 384 y(MPI_Datatype)22 b(record_type;)75 440 y(MPI_datatype)g(types[3])h(=)h({MPI_CHAR,)e(MPI_DOUBLE,)h(MPI_FLOAT};) 75 497 y(int)238 b(lengths[3])23 b(=)g({1,)h(3,)f(1};)75 610 y(MPI_Type_simple_struct\()e(3,)i(lengths,)g(types,)g (record_type\);)189 704 y Fe(R)n(ationale.)76 b Fq(The)21 b(co)q(de)h(ab)q(o)o(v)o(e)f(is)h(m)o(uc)o(h)f(simpler)h(and)g(more)e (natural)i(than)f(the)g(curren)o(t)189 760 y(approac)o(h,)14 b(that)h(uses)g Fj(MPI)p 677 760 14 2 v 16 w(TYPE)p 810 760 V 17 w(STRUCT)p Fq(,)g(and)h(requires)g(to)e(compute)h(displacemen) o(ts.)189 834 y(It)i(also)g(pro)o(vides)h(a)f(de\014nition)i(of)e(the)h (record)f(t)o(yp)q(e)h(that)e(is)i(arc)o(hitecture)g(indep)q(enden)o(t) i(\(as-)189 890 y(suming)c(that)e(compilers)j(don't)e(\014ddle)i(with)e (the)h(order)f(of)g(\014elds,)h(whic)o(h)g(is)g(true)f(in)i(C\))d({)i (this)189 947 y(facilitates)f(transfer)e(of)h(records)g(using)h(RMA,)f (an)g(\014ts)g(w)o(ell)h(with)g(v)m(arious)f(prop)q(osals)h(for)e (third)189 1003 y(part)o(y)h(transfers.)19 b(\()p Fe(End)d(of)g(r)n (ationale.)p Fq(\))189 1097 y Fe(A)n(dvic)n(e)h(to)h(users.)50 b Fq(Datat)o(yp)q(es)16 b(constructed)h(using)h Fj(MPI)p 1231 1097 V 16 w(TYPE)p 1364 1097 V 16 w(SIMPLE)p 1539 1097 V 16 w(STRUCT)g Fq(should)189 1154 y(b)q(e)i(used)g(only)g(for)f (structures)g(that)g(use)h(the)g(default)g(alignmen)o(t)g(rules.)34 b(They)20 b(should)g(not)189 1210 y(b)q(e)c(used)h(if)f(alignmen)o(t)g (rules)h(ha)o(v)o(e)f(b)q(een)h(c)o(hanged,)f(using)g(compiler)h (options)f(or)g(compilation)189 1267 y(pragmas.)j(\()p Fe(End)c(of)i(advic)n(e)f(to)g(users.)p Fq(\))189 1361 y Fe(A)n(dvic)n(e)g(to)h(implementors.)47 b Fq(The)16 b(datat)o(yp)q(e)g(constructor)f Fj(MPI)p 1330 1361 V 16 w(TYPE)p 1463 1361 V 17 w(SIMPLE)p 1639 1361 V 15 w(STRUCT)i Fq(is)189 1417 y(most)e(easily)i(implemen)o(ted)h(on)e (systems)g(that)f(use)h Fe(natur)n(al)g Fq(data)g(alignmen)o(t)h (rules.)23 b(On)17 b(suc)o(h)189 1474 y(systems,)12 b(eac)o(h)h(basic)g (datat)o(yp)q(e)f(has)g(an)h(exten)o(t)f(and)h(an)g(alignmen)o(t;)h (the)e(alignmen)o(t)i(of)e(a)g(com-)189 1530 y(p)q(ound)j(t)o(yp)q(e)g (is)h(the)f(strictest)f(alignmen)o(t)i(of)e(a)g(comp)q(onen)o(t.)20 b(P)o(adding)c(is)f(added)g(so)g(that)f(eac)o(h)189 1587 y(comp)q(onen)o(t)f(is)h(aligned)g(at)f(its)g(natural)g(alignmen)o(t.) 20 b(W)l(e)13 b(can)g(then)h(deriv)o(e)g(ob)o(vious)f(de\014nitions)189 1643 y(for)h(the)h(exten)o(t)g(and)h(alignmen)o(t)g(of)e(p)q(ortable)i (MPI)f(datat)o(yp)q(es:)189 1700 y(The)g(exten)o(t)f(and)h(alignmen)o (t)g(of)f(a)h(basic)g(datat)o(yp)q(e)f(is)i(the)e(exten)o(t)h(and)g (alignmen)o(t)g(of)f(the)h(cor-)189 1756 y(resp)q(onding)h(language)f (t)o(yp)q(e.)189 1812 y(The)10 b(alignmen)o(t)h(of)f(a)f(datat)o(yp)q (e)h(built)i(with)e(one)g(of)g(the)g(constructors)g Fj(MPI)p 1487 1812 V 15 w(TYPE)p 1619 1812 V 17 w(CONTIGUOUS,)189 1869 y(MPI)p 274 1869 V 15 w(TYPE)p 406 1869 V 17 w(VECTOR,)26 b Fq(and)e Fj(PI)p 780 1869 V 16 w(TYPE)p 913 1869 V 17 w(INDEXED)h Fq(is)g(the)f(alignmen)o(t)h(of)f(the)h(comp)q(onen)o(t) 189 1925 y(datat)o(yp)q(e;)14 b(the)h(exten)o(t)g(of)g(the)g(comp)q (ound)h(datat)o(yp)q(e)f(is)g(as)g(de\014ned)i(in)f(MPI1.)189 1982 y(The)h(alignmen)o(t)g(of)f(a)g(datat)o(yp)q(e)h(built)h(b)o(y)e (a)h(call)g(to)f Fj(MPI)p 1221 1982 V 16 w(TYPE)p 1354 1982 V 17 w(SIMPLE)p 1530 1982 V 16 w(STRUCT\(n,)h(lens,)189 2038 y(t)o(yp)q(es,)i(newt)o(yp)q(e\))h Fq(is)e(max)643 2045 y Fa(i)672 2038 y Fk(alignment)p Fq(\()p Fk(types)p Fq([)p Fk(i)p Fq(]\);)e(the)i(exten)o(t)g(is)g(computed)h(b)o(y)f(la)o (ying)g(out)189 2095 y(the)h(comp)q(onen)o(ts)g(with)h(minimal)h (padding)f(added)g(so)f(that)g(eac)o(h)h(comp)q(onen)o(t)f(starts)f(at) h(its)189 2151 y(alignmen)o(t.)189 2225 y(Some)g(compilers)h(treat)f (the)g(\014rst)g(comp)q(onen)o(t)g(of)g(a)g(structure)g(di\013eren)o (tly)l(.)33 b(E.g.,)18 b(the)i(IBM)189 2281 y(compiler)d(aligns)h (doubles)f(at)f(w)o(ord)g(b)q(oundaries,)i(but)f(align)g(a)f(structure) g(that)g(starts)f(with)i(a)189 2338 y(double)j(comp)q(onen)o(t)f(at)f (doublew)o(ord)i(b)q(oundaries.)32 b(The)19 b(de\014nition)i(of)d(the)h (alignmen)o(t)h(of)e(a)189 2394 y(datat)o(yp)q(e)h(built)i(with)f Fj(MPI)p 690 2394 V 16 w(TYPE)p 823 2394 V 16 w(SIMPLE)p 998 2394 V 16 w(STRUCT)h Fq(is)f(c)o(hanged,)h(accordingly)l(.)34 b(\()p Fe(End)20 b(of)189 2450 y(advic)n(e)c(to)g(implementors.)p Fq(\))75 2571 y Ff(3.9.2)49 b(Contiguous)17 b(Struct)e(T)l(yp)q(es)75 2704 y Fm(Curren)o(t)f(Status:)j Fl(F)m(ailed)c(6-7-10)f(at)i(Septem)o (b)q(er)g(meeting.)-32 46 y Fh(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 12 14 12 13 bop 75 -100 a Fq(12)947 b Fg(CHAPTER)15 b(3.)35 b(MISCELLANY)17 b(F)o(OR)e(1.2)166 45 y Fq(Data)j(structures)i(ma)o(y)f (con)o(tain)h(padding)g(holes,)i(added)e(for)f(alignmen)o(t.)34 b(The)20 b(de\014nition)h(of)75 102 y(an)f(MPI)g(datat)o(yp)q(e,)g (done)g(using)g(the)g Fj(MPI)p 865 102 14 2 v 16 w(TYPE)p 998 102 V 17 w(STRUCT)g Fq(constructor,)g(do)q(es)g(not)g(distinguish) 75 158 y(b)q(et)o(w)o(een)15 b(holes)g(that)f(con)o(tains)g(no)h (signi\014can)o(t)g(data,)f(and)h(are)f(there)h(only)g(for)f(alignmen)o (t)h(purp)q(oses,)75 214 y(and)k(b)q(et)o(w)o(een)g(holes)g(that)f(ma)o (y)f(con)o(tain)i(signi\014can)o(t)g(data)f(and)h(w)o(ere)f(purp)q (osefully)j(left)e(out)f(b)o(y)g(a)75 271 y(user)k(that)e(wishes)i(not) f(to)g(o)o(v)o(erwrite)g(some)g(of)g(the)g(information)h(in)g(a)f (structure.)38 b(As)22 b(a)f(result,)75 327 y(MPI)g(cannot)g(o)o(v)o (erwrite)f(these)h(holes,)i(this)f(prev)o(en)o(ts)e(the)i(ob)o(vious)f (optimization)h(of)e(shipping)j(a)75 384 y(data)13 b(structure)h(as)g (a)f(con)o(tiguous)h(blo)q(c)o(k)h(of)f(data,)f(padding)i(holes)f (included.)22 b(The)15 b(t)o(yp)q(e)f(constructor)75 440 y Fj(MPI)p 160 440 V 16 w(TYPE)p 293 440 V 17 w(CONTIGUOUS)p 598 440 V 18 w(STRUCT)d Fq(marks)f(an)o(y)g(suc)o(h)h(holes)g(as)g (\\don)o(t-care")e(lo)q(cations,)j(allo)o(wing)75 497 y(the)j(implemen)o(tation)i(to)d(o)o(v)o(erwrite)h(them.)75 648 y Fj(MPI)p 160 648 V 16 w(TYPE)p 293 648 V 17 w(CONTIGUOUS)p 598 648 V 18 w(STRUCT\(count,)j(a)o(rra)o(y)p 1041 648 V 14 w(of)p 1092 648 V 16 w(blo)q(cklengths,)h(a)o(rra)o(y)p 1468 648 V 14 w(of)p 1519 648 V 16 w(displacements,)g(a)o(r-)75 704 y(ra)o(y)p 136 704 V 15 w(of)p 188 704 V 16 w(t)o(yp)q(es,)e(newt)o (yp)q(e\))117 781 y Fl(IN)155 b Fj(count)482 b Fl(n)o(um)o(b)q(er)13 b(of)h(blo)q(c)o(ks)g(\(in)o(teger\))117 856 y(IN)155 b Fj(a)o(rra)o(y)p 416 856 V 15 w(of)p 468 856 V 16 w(blo)q(cklengths) 191 b Fl(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(eac)o(h)g(blo)q (c)o(k)g(\(arra)o(y)g(of)f(in)o(teger\))117 932 y(IN)155 b Fj(a)o(rra)o(y)p 416 932 V 15 w(of)p 468 932 V 16 w(displacements)164 b Fl(b)o(yte)14 b(displacemen)o(t)g(of)f(eac)o(h)h(blo)q(c)o(k)g (\(arra)o(y)g(of)f(in)o(teger\))117 1007 y(IN)155 b Fj(a)o(rra)o(y)p 416 1007 V 15 w(of)p 468 1007 V 16 w(t)o(yp)q(es)327 b Fl(t)o(yp)q(e)15 b(of)e(elemen)o(ts)h(in)f(eac)o(h)h(blo)q(c)o(k)g (\(arra)o(y)g(of)f(handle\))117 1082 y(OUT)108 b Fj(newt)o(yp)q(e)433 b Fl(new)15 b(datat)o(yp)q(e)f(\(handle\))166 1206 y Fq(The)c(seman)o(tics)h(of)e Fj(MPI)p 587 1206 V 16 w(TYPE)p 720 1206 V 17 w(CONTIGUOUS)p 1025 1206 V 18 w(STRUCT)i Fq(are)f(iden)o(tical)i(to)d Fj(MPI)p 1611 1206 V 16 w(TYPE)p 1744 1206 V 17 w(STRUCT)p Fq(,)75 1263 y(except)15 b(that,)e(if)h(a)g(receiv)o(e)h(call)g(uses)g(a)e(datat)o(yp)q(e)h (constructed)g(with)h(this)f(constructor,)f(then)i(the)f(in-)75 1319 y(ternal)f(holes)h(in)f(the)g(structure)g(can)g(b)q(e)g(o)o(v)o (erwritten.)18 b(The)13 b(same)g(rule)h(\(namely)l(,)f(that)f(holes)i (b)q(et)o(w)o(een)75 1376 y(comp)q(onen)o(ts)c(can)g(b)q(e)h(o)o(v)o (erwritten\))e(applies)j(also)e(to)g(datat)o(yp)q(es)f(constructed)h (using)h Fj(MPI)p 1635 1376 V 16 w(TYPE)p 1768 1376 V 17 w(SIMPLE)p 1944 1376 V 16 w(STRUCT)p Fq(.)166 1479 y Fm(Discussion:)29 b Fl(It)10 b(is)f(reasonable)g(to)g(assume)g(that)h (holes)f(in)g(la)o(y)o(outs)f(de\014ned)j(using)e Fd(MPI)p 1576 1479 13 2 v 14 w(TYPE)p 1698 1479 V 14 w(SIMPLE)p 1857 1479 V 15 w(STRUCT)75 1536 y Fl(con)o(tain)14 b(no)h(signi\014can) o(t)f(information,)e(since)k(this)f(datat)o(yp)q(e)g(constructor)h(is)f (exp)q(ected)i(to)e(b)q(e)g(used)h(for)f(sp)q(ec-)75 1592 y(ifying)f(the)h(la)o(y)o(out)f(of)h(structures.)24 b(Users)16 b(will)e(b)q(e)i(required)g(to)f(use)h Fd(MPI)p 1271 1592 V 14 w(TYPE)p 1393 1592 V 14 w(STRUCT)f Fl(when)g(holes)h (hold)75 1649 y(signi\014can)o(t)d(information.)189 1802 y Fe(A)n(dvic)n(e)e(to)i(users.)38 b Fq(Users)11 b(should)i(arrange)d (the)i(\014elds)g(in)g(a)f(C)h(struct)e(t)o(yp)q(e)i(in)g(descending)h (order)189 1859 y(of)i(their)i(size.)24 b(This)17 b(arrangemen)o(t)e (has)h(t)o(w)o(o)f(adv)m(an)o(tages:)21 b(\014rstly)c(it)f(ma)o(y)g (reduce)h(the)f(size)h(of)189 1915 y(the)i(struct)f(b)o(y)h (eliminating)i(padding)f(b)q(et)o(w)o(een)f(elemen)o(ts)h(with)f (di\013eren)o(t)g(alignmen)o(ts)h(\(this)189 1972 y(is)c(generally)i (true)e(for)g(C)g(programs\).)22 b(Secondly)l(,)c(it)e(ma)o(y)g (increase)h(the)g(n)o(um)o(b)q(er)f(of)g(cases)h(for)189 2028 y(whic)o(h)d(the)f(implemen)o(tation)h(can)g(p)q(erform)f(blo)q(c) o(k)g(copies)i(b)o(y)e(ensuring)h(that)e(the)h(\014rst)g(elemen)o(t)189 2085 y(of)19 b(the)g(structure)g(has)h(the)f(most)g(stringen)o(t)g (alignmen)o(t)h(requiremen)o(ts.)33 b(\()p Fe(End)19 b(of)i(advic)n(e)f(to)189 2141 y(users.)p Fq(\))189 2247 y Fe(A)n(dvic)n(e)15 b(to)i(implementors.)39 b Fq(An)16 b(implemen)o(tation)g(ma)o(y)f(ignore)g(the)h(constructor)189 2304 y Fj(MPI)p 274 2304 14 2 v 15 w(TYPE)p 406 2304 V 17 w(CONTIGUOUS)p 711 2304 V 18 w(STRUCT)p Fq(,)21 b(and)f(handle)i(it)e(exactly)h(as)f Fj(MPI)p 1539 2304 V 16 w(TYPE)p 1672 2304 V 17 w(STRUCT)p Fq(.)189 2360 y(Th)o(us,)f(there)g(is)h(\(almost\))d(no)i(o)o(v)o(erhead)g(in)g(supp) q(orting)h(this)f(constructor,)g(without)g(taking)189 2417 y(adv)m(an)o(tage)f(of)h(it.)32 b(One)20 b(p)q(ossible)h(w)o(a)o (y)d(of)h(taking)g(adv)m(an)o(tage)g(of)g(it,)h(is)g(to)e(adopt)h(a)g (canoni-)189 2473 y(cal)d(\\wire")g(represen)o(tation)g(for)g (structures.)22 b(Supp)q(ose)17 b(that)f(the)g(compiler)h(uses,)g(b)o (y)f(default,)189 2530 y(natural)g(alignmen)o(t)i(rules)f(for)f (structures:)23 b(eac)o(h)16 b(basic)i(comp)q(onen)o(t)e(is)i(aligned)g (at)e(a)g(natural)189 2586 y(b)q(oundary)l(.)k(Supp)q(ose)14 b(that)f(one)h(adopts)f(natural)h(alignmen)o(t)g(rules)h(for)e (messages)g(on)g(the)h(wire,)189 2642 y(in)19 b(a)f(homogeneous)g(en)o (vironmen)o(t:)27 b(messages)18 b(are)g(padded)h(so)f(that)g(eac)o(h)g (basic)h(elemen)o(t)g(is)189 2699 y(aligned)d(at)f(a)g(natural)g(b)q (oundary)l(.)20 b(Then)1967 46 y Fh(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 13 15 13 14 bop 75 -100 a Fg(3.9.)34 b(NEW)15 b(D)o(A)l(T)l(A)l(TYPES)1245 b Fq(13)189 45 y(\(1\))14 b(A)i(receiv)o(er)g(can)f(alw)o(a)o(ys)g (deco)q(de)h(an)g(incoming)h(message)d(and)i(retriev)o(e)g(correctly)f (the)h(ba-)189 102 y(sic)k(comp)q(onen)o(ts.)31 b(This)20 b(b)q(ecause)g(the)g(basic)g(datat)o(yp)q(e)e(of)h(the)g(elemen)o(t)h (indicates)h(at)e(what)189 158 y(b)q(oundary)c(it)h(is)f(aligned.)189 214 y(\(2\))f(A)h(sender)g(can)h(blo)q(c)o(k)f(cop)o(y)g(a)g(structure) g(on)o(to)f(the)h(\\wire")g(\(most)f(lik)o(ely)l(,)j(on)o(to)d(a)g (message)189 271 y(bu\013er\),)19 b(when)g(the)g(structure)g(is)g (naturally)h(aligned,)h Fe(r)n(elative)e(to)h(the)g(start)g(of)f(the)h (sending)189 327 y(bu\013er)p Fq(.)26 b(This)18 b(optimization)g(do)q (es)g(not)e(require)j(that)d(the)h(sending)i(datat)o(yp)q(e)e(b)q(e)h (built)g(using)189 384 y Fj(MPI)p 274 384 14 2 v 15 w(TYPE)p 406 384 V 17 w(CONTIGUOUS)p 711 384 V 18 w(STRUCT)p Fq(:)c(The)g(MPI)f (library)i(can)e(scan)h(datat)o(yp)q(es,)f(when)h(they)189 440 y(are)h(committed,)f(and)i(recognize)g(when)g(a)e(datat)o(yp)q(e)h (\(or)f(a)h(datat)o(yp)q(e)g(fragmen)o(t\))f(is)h(naturally)189 497 y(aligned.)24 b(Blo)q(c)o(k)17 b(cop)o(ying)g(can)g(b)q(e)g(used)g (ev)o(en)f(if)h(the)g(holes)g(in)g(the)f(datat)o(yp)q(e)g(con)o(tain)g (signi\014-)189 553 y(can)o(t)e(information.)189 610 y(\(3\))h(A)h(receiv)o(er)h(can)f(blo)q(c)o(k)h(cop)o(y)f(an)g (incoming)i(message)d(on)o(to)h(the)g(receiv)o(e)h(bu\013er)f(when)h (the)189 666 y(receiving)e(structure)f(is)h(naturally)g(aligned)h Fe(r)n(elative)f(to)g(the)h(start)g(of)f(the)h(r)n(e)n(c)n(eiving)d (bu\013er)i Fq(and)189 723 y(holes)10 b(can)h(b)q(e)g(ignored.)18 b(I.e.,)11 b(when)g(the)f(receiving)i(datat)o(yp)q(e)d(is)i(built)g (using)g Fj(MPI)p 1597 723 V 16 w(TYPE)p 1730 723 V 17 w(CONTIGUOUS)p 2035 723 V 18 w(STRUCT)189 779 y Fq(or)i Fj(MPI)p 328 779 V 16 w(TYPE)p 461 779 V 17 w(SIMPLE)p 637 779 V 16 w(STRUCT)h Fq(and)g(is)h(naturally)f(aligned,)i(relativ)o (e)e(to)g(its)g(\014rst)f(p)q(osition.)189 853 y(The)k(same)h(wire)g (represen)o(tation)f(ma)o(y)g(b)q(e)h(used)g(in)h(a)e(heterogenous)h (en)o(vironmen)o(t,)g(in)g(cases)189 909 y(where)h(data)e(con)o(v)o (ersion)i(is)g(done)g(at)f(the)h(destination;)i(the)e(destination)g (has)g(to)f(kno)o(w,)g(not)189 966 y(only)c(the)h(basic)g(datat)o(yp)q (es)e(used)i(b)o(y)f(the)h(source,)f(but)g(also)h(the)f(alignmen)o(t)h (rules)g(used)g(b)o(y)f(the)189 1022 y(source.)189 1096 y(Example:)34 b(Consider)23 b(a)f(C)g(structure)g Fk(struct)h({char)g (a[2];)g(float)g(b[2];)h(double)f(c})p Fq(;)189 1153 y(assume)16 b(that)g(\015oats)g(are)g(4)h(b)o(ytes)f(and)h(doubles)h (are)e(8)g(b)o(ytes,)h(and)g(structures)f(are)g(naturally)189 1209 y(aligned.)36 b(The)20 b(memory)f(la)o(y)o(out)h(is)44 b Fk([c][c]xx[ffff][ffff]xxxx[dddd)o(dddd])p Fq(,)18 b(and)i(the)189 1266 y(structure)15 b(is)g(aligned)i(to)d(an)i(8)e(b)o (yte)h(b)q(oundary)l(.)189 1340 y(Case)c(1:)18 b(The)12 b(en)o(tire)g(structure)f(is)h(comm)o(unicated.)20 b(Sender)12 b(constructs)f(a)h(datat)o(yp)q(e)f(for)g(struc-)189 1396 y(ture)f(using)h Fj(MPI)p 479 1396 V 15 w(TYPE)p 611 1396 V 17 w(STRUCT,)g(MPI)p 910 1396 V 15 w(TYPE)p 1042 1396 V 17 w(SIMPLE)p 1218 1396 V 16 w(STRUCT)g Fq(or)f Fj(MPI)p 1554 1396 V 15 w(TYPE)p 1686 1396 V 17 w(CONTIGUOUS)p 1991 1396 V 18 w(STRUCT)p Fq(;)189 1452 y(receiv)o(er)16 b(constructs)g(a)g(datat)o(yp)q(e)g(for)f(this)i(structure,)e(using)i Fj(MPI)p 1379 1452 V 16 w(CONTIGUOUS)p 1683 1452 V 18 w(STRUCT)189 1509 y Fq(or)12 b Fj(MPI)p 327 1509 V 16 w(TYPE)p 460 1509 V 16 w(SIMPLE)p 635 1509 V 16 w(STRUCT)p Fq(.)h(Then)g(the)g(la)o(y)o(outs)f(in)h(the)g(sender)g(memory)l(,)g (on)f(the)h(wire,)189 1565 y(and)i(in)h(the)f(receiv)o(er)h(memory)f (are)g(iden)o(tical,)h(and)g(data)e(can)i(b)q(e)g(simply)g(copied.)189 1639 y(Case)10 b(2:)17 b(comm)o(unication)12 b(in)o(v)o(olv)o(es)f (only)h(the)f(second)g(of)f(the)h(t)o(w)o(o)f(c)o(haracters,)g(and)h (the)g(remain-)189 1696 y(der)f(of)g(the)g(structure.)18 b(Receiv)o(er)11 b(constructs)f(the)g(datat)o(yp)q(e)g(using)h Fj(MPI)p 1441 1696 V 16 w(CONTIGUOUS)p 1745 1696 V 18 w(STRUCT)p Fq(.)189 1752 y(The)19 b(memory)f(la)o(y)o(out)g(is)h Fk([c]xx[ffff][ffff]xxxx[dddddd)o(dd])p Fq(,)d(and)j(the)g(wire)g(la)o (y)o(out)f(is)189 1809 y Fk([c]xxx[ffff][ffff]xxxx[)o(dddddddd)o(])p Fq(.)e(The)c(datat)o(yp)q(e)f(is)h(not)g(naturally)g(aligned,)h(so)f (that)189 1865 y(blo)q(c)o(k)k(cop)o(ying)g(cannot)f(b)q(e)h(used)g(at) f(the)h(sender)g(or)f(the)g(receiv)o(er.)22 b(Ho)o(w)o(ev)o(er,)14 b(a)h(smart)f(imple-)189 1922 y(men)o(tation)h(ma)o(y)g(realize)i(that) e(the)h(fragmen)o(t)f(starting)g(from)g(the)h(\015oat)f(is)i(naturally) f(aligned,)189 1978 y(relativ)o(e)e(to)e(the)i(start)e(of)h(the)g (bu\013er,)h(so)f(that)f(blo)q(c)o(k)j(cop)o(ying)e(can)h(b)q(e)g(used) g(for)f(this)g(fragmen)o(t.)189 2052 y(Case)f(3:)19 b(comm)o(unication) 14 b(in)o(v)o(olv)o(es)g(only)f(the)g(t)o(w)o(o)f(\015oats)h(and)g(the) g(double.)21 b(The)13 b(memory)g(la)o(y-)189 2108 y(out)c(is)i Fk([ffff][ffff]xxxx[dddddddd])o Fq(.)16 b(The)10 b(la)o(y)o(out)g(on)g (the)g(wire)g(is)h Fk([ffff][ffff][dddddddd])p Fq(.)189 2165 y(The)j(datat)o(yp)q(e)f(is)i(not)f(naturally)g(aligned,)i (relativ)o(e)e(to)g(its)g(start,)e(so)i(that)f(blo)q(c)o(k)i(cop)o (ying)g(can-)189 2221 y(not)f(b)q(e)i(used.)21 b(\()p Fe(End)15 b(of)i(advic)n(e)f(to)g(implementors.)p Fq(\))166 2360 y Fm(Alternativ)o(es)o(:)134 2432 y Fl(1.)22 b(No)14 b(new)h(function)f Fd(MPI)p 581 2432 13 2 v 14 w(TYPE)p 703 2432 V 15 w(CONTIGUOUS)p 985 2432 V 12 w(STRUCT)p Fl(:)g(instead,)g(holes)h(can)g(b)q(e)g(o)o(v)o(erwritten)g(only)189 2482 y(for)f(datat)o(yp)q(es)i(built)e(with)g Fd(MPI)p 717 2482 V 15 w(TYPE)p 840 2482 V 14 w(SIMPLE)p 999 2482 V 15 w(STRUCT)p Fl(.)f(The)j(argumen)o(t)e(is)g(that,)h(at)g(least)g (in)f(the)189 2532 y(scenario)f(I)g(presen)o(ted,)i(hole)d(o)o(v)o (erwriting)h(is)g(tak)o(en)g(adv)n(an)o(tage)f(of)g(only)g(for)h (structures)i(with)e(a)f(natural)189 2581 y(alignmen)o(t.)j(But,)f (then,)g(their)h(la)o(y)o(out)d(can)i(b)q(e)h(sp)q(eci\014ed)g(using)f Fd(MPI)p 1308 2581 V 14 w(TYPE)p 1430 2581 V 15 w(SIMPLE)p 1590 2581 V 14 w(STRUCT)p Fl(.)134 2654 y(2.)22 b(Both)d(sender)h(and)f (receiv)o(er)h(are)g(required)f(to)g(use)h Fd(MPI)p 1126 2654 V 14 w(TYPE)p 1248 2654 V 14 w(CONTIGUOUS)p 1529 2654 V 13 w(STRUCT)p Fl(,)e(and)g(the)189 2704 y(datat)o(yp)q(es)c(on)f (b)q(oth)h(ends)g(ha)o(v)o(e)g(to)f(b)q(e)h(structurally)g(equiv)n (alen)o(t,)f(i.e.,)f(de\014ned)j(b)o(y)e(the)h(same)f(sequence)-32 46 y Fh(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 14 16 14 15 bop 75 -100 a Fq(14)947 b Fg(CHAPTER)15 b(3.)35 b(MISCELLANY)17 b(F)o(OR)e(1.2)189 45 y Fl(of)e(de\014nitions.)18 b(There)d(are)f(sev)o(eral)h(p)q(ossible)f(v)n(arian)o(ts)f(for)g(suc)o (h)i(prop)q(osal)189 107 y(\(a\))20 b(This)g(construct)i(can)e(b)q(e)h (only)e(used)i(in)f(a)g(homogeneous)f(en)o(vironmen)o(t.)36 b(But,)22 b(then,)g(w)o(e)e(lose)189 157 y(p)q(ortabilit)o(y)11 b(to)h(heterogeneous)j(en)o(vironmen)o(t,)c(and)i(w)o(e)g(ac)o(hiev)o (e)f(little)g(that)h(could)f(not)h(b)q(e)g(ac)o(hiev)o(ed)g(b)o(y)189 207 y(using)g Fd(MPI)p 374 207 13 2 v 15 w(BYTE)p Fl(.)189 269 y(\(b\))18 b(This)g(construct)i(can)e(b)q(e)g(used)h(p)q(ortably)m (,)f(with)g(enhanced)h(p)q(erformance)f(obtained)g(in)f(a)h(homo-)189 319 y(geneous)i(en)o(vironmen)o(t.)36 b(E.g.,)20 b(a)g(homogeneous)f (implem)o(en)o(tation)e(will)i(use)h(blo)q(c)o(k)g(cop)o(ying,)g(while) 189 369 y(a)d(heterogeneous)j(implemen)o(tatio)o(n)15 b(will)i(mo)o(v)o(e)f(elemen)o(t)h(b)o(y)h(elemen)o(t)f(\(at)h(least,)g (for)g(heterogeneous)189 419 y(comm)o(unicators\).)j(This)16 b(assumes)g(that,)g(in)f(a)h(homogeneous)e(en)o(vironmen)o(t,)h (di\013eren)o(t)i(pro)q(cesses)i(are)189 469 y(compiled)8 b(with)i(the)g(same)f(data)h(alignmen)o(t)d(options.)17 b(Also,)10 b(this)g(runs)g(coun)o(ter)h(the)g(datat)o(yp)q(e)f(matc)o (hing)189 518 y(rules)k(in)g(MPI,)f(where)i(only)e(signature)i(matc)o (hing)d(is)h(required.)189 581 y(\(c\))d Fd(MPI)p 326 581 V 14 w(TYPE)p 448 581 V 14 w(CONTIGUOUS)p 729 581 V 13 w(STRUCT)f Fl(can)g(b)q(e)i(matc)o(hed)d(with)h(other)i(datat)o (yp)q(es,)f(as)g(long)e(as)i(signa-)189 630 y(tures)g(matc)o(h;)f (impro)o(v)o(ed)f(p)q(erformance)h(is)g(ac)o(hiev)o(ed)h(when)f(b)q (oth)h(ends)g(use)g Fd(MPI)p 1463 630 V 14 w(TYPE)p 1585 630 V 15 w(CONTIGUOUS)p 1867 630 V 12 w(STRUCT)p Fl(.)189 680 y(But,)i(then,)h(the)g(wire)g(proto)q(col)f(m)o(ust)f(b)q(e)i(so)f (that)h(individual)d(elemen)o(ts)i(can)h(b)q(e)f(extracted,)i(in)e (case)h(the)189 730 y(receiv)o(er)k(do)q(es)f(not)f(use)i Fd(MPI)p 664 730 V 14 w(TYPE)p 786 730 V 14 w(CONTIGUOUS)p 1067 730 V 13 w(STRUCT)p Fl(.)d(In)i(this)f(case,)i(asking)e(the)h (sender)h(to)189 780 y(use)g Fd(MPI)p 340 780 V 14 w(TYPE)p 462 780 V 15 w(CONTIGUOUS)p 744 780 V 12 w(STRUCT)f Fl(do)q(es)h(not)g (pro)o(vide)f(additional)f(function)h(o)o(v)o(er)h(the)g(main)189 830 y(prop)q(osal.)g(The)d(di\013erence)h(is)f(only)e(whether)j(the)f (use)g(asserts)i(that)d(the)h(sending)g(datat)o(yp)q(e)f(has)h(a)f (nat-)189 879 y(ural)i(la)o(y)o(out)g(\(using)h Fd(MPI)p 608 879 V 14 w(TYPE)p 730 879 V 14 w(CONTIGUOUS)p 1011 879 V 13 w(STRUCT)p Fl(\),)f(or)h(whether)h(MPI)f(disco)o(v)o(ers)h (that)f(fact)189 929 y(b)o(y)d(itself.)75 1123 y Ff(3.9.3)49 b(Convenient)17 b(F)o(o)o(rm)d(of)j(MPI)p 731 1123 15 2 v 18 w(T)l(yp)q(e)p 848 1123 V 18 w(indexed)p 1018 1123 V 18 w(blo)q(ck)75 1256 y Fm(Curren)o(t)d(Status:)j Fl(P)o(assed)e(once.)166 1360 y Fq(This)d(prop)q(osal)g(is)g(to)f(allo) o(w)h(constan)o(t)f(blo)q(c)o(ksize)i(and)f(arbitrary)f(displacemen)o (ts)i(in)f(the)g(in)o(terface)75 1416 y(to)e Fj(MPI)p 211 1416 14 2 v 15 w(TYPE)p 343 1416 V 17 w(INDEXED)p Fq(.)g(The)g(routine)h(no)o(w)f(tak)o(es)f Fj(a)o(rra)o(y)p 1114 1416 V 14 w(of)p 1165 1416 V 16 w(blo)q(cklengths)k Fq(and)d Fj(a)o(rra)o(y)p 1605 1416 V 15 w(of)p 1657 1416 V 16 w(displacements)75 1473 y Fq(as)17 b(argumen)o(ts.)27 b(There)18 b(are)f(man)o(y)h(co)q(des)g(using)g(indirect)i(addressing)e (arising)g(from)f(unstructured)75 1529 y(grids)f(where)h(the)f(blo)q(c) o(ksize)i(is)e(alw)o(a)o(ys)f(1)h(\(gather/scatter\).)k(W)l(e)c(prop)q (ose)g(this)h(in)o(terface,)f(allo)o(wing)75 1586 y(for)f(constan)o(t)f (blo)q(c)o(ksize)j(and)e(arbitrary)g(displacemen)o(ts.)75 1737 y Fj(MPI)p 160 1737 V 16 w(TYPE)p 293 1737 V 17 w(INDEXED)p 505 1737 V 16 w(BLOCK\()g(count,)i(blo)q(cklength,)h(a)o (rra)o(y)p 1180 1737 V 14 w(of)p 1231 1737 V 16 w(displacements,)g (oldt)o(yp)q(e,)f(newt)o(yp)q(e\))117 1870 y Fl(IN)155 b Fj(count)482 b Fl(length)14 b(of)f(arra)o(y)h(of)f(displacemen)o(ts) 117 1945 y(IN)155 b Fj(blo)q(cklength)371 b Fl(size)15 b(of)e(blo)q(c)o(k)117 2021 y(IN)155 b Fj(a)o(rra)o(y)p 416 2021 V 15 w(of)p 468 2021 V 16 w(displacements)164 b Fl(arra)o(y)14 b(of)f(displacemen)o(ts)117 2096 y(IN)155 b Fj(oldt)o(yp)q(e)450 b Fl(old)13 b(datat)o(yp)q(e)117 2171 y(OUT)108 b Fj(newt)o(yp)q(e)433 b Fl(new)15 b(datat)o(yp)q(e)75 2295 y Fk(int)23 b(MPI)p 245 2295 15 2 v 17 w(Type)p 358 2295 V 17 w(Indexed)p 543 2295 V 16 w(Block\()g(int)h(count,)f(int) g(blocklength,)393 2352 y(int*)g(array)p 635 2352 V 17 w(of)p 700 2352 V 17 w(displacements,)f(MPI)p 1147 2352 V 17 w(Datatype)g(oldtype,)393 2408 y(MPI)p 468 2408 V 17 w(Datatype*)h(newtype)f(\))75 2495 y(MPI)p 150 2495 V 17 w(TYPE)p 263 2495 V 16 w(INDEXED)p 447 2495 V 17 w(BLOCK\(COUNT,)g(BLOCKLENGTH,)g(ARRAY)p 1204 2495 V 17 w(OF)p 1269 2495 V 16 w(DISPLACEMENTS,)g(OLDTYPE,)393 2551 y(NEWTYPE,)h(IERROR\))170 2608 y(INTEGER)g(COUNT,)g(INTEGER)g (BLOCKLENGTH,)g(ARRAY)p 1153 2608 V 16 w(OF)p 1217 2608 V 17 w(DISPLACEMENT\(*\),)f(OLDTYPE,)170 2664 y(NEWTYPE,)h(IERROR)-117 2700 y Fi(>)15 b Fh(\(Oct\))1967 46 y(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 15 17 15 16 bop 75 -100 a Fg(3.10.)34 b(MINOR)16 b(CORRECTIONS)1109 b Fq(15)75 45 y Fk(int)23 b(MPI::Datatype::Indexed)p 701 45 15 2 v 15 w(Block\()g(int)g(count,)g(int)h(blocklength,)393 102 y(const)f(int)h(array)p 755 102 V 16 w(of)p 819 102 V 17 w(displacements[],)393 158 y(const)f(MPI::Datatype&)f(oldtype)h (\))1875 198 y Fi(?)16 b Fh(\(Oct\))75 331 y Fn(3.10)59 b(Mino)n(r)21 b(Co)n(rrections)75 480 y Fm(Curren)o(t)14 b(Status:)j Fl(P)o(assed)e(once.)166 584 y Fq(The)i(follo)o(wing)i (corrections)e(to)g Fj(MPI-1.1)f Fq(are)h(\(all)h(page)f(and)g(line)i (n)o(um)o(b)q(ers)f(are)f(for)g(the)g(June)75 640 y(12,)d(1995)g(v)o (ersion)i(without)f(c)o(hangebars\):)143 746 y Fi(\017)23 b Fq(P)o(age)14 b(19,)g(lines)j(1-2)e(reads)189 803 y(for)f(\(64)g (bit\))i(C)f(in)o(tegers)g(declared)i(to)d(b)q(e)i(of)f(t)o(yp)q(e)g Fj(longlong)g(int)189 859 y Fq(but)g(should)h(read)189 916 y(for)e(\(64)g(bit\))i(C)f(in)o(tegers)g(declared)i(to)d(b)q(e)i (of)f(t)o(yp)q(e)g Fj(long)g(long)g(int)189 1088 y Fm(Discussion)o(:) 189 1156 y Fl(Just)f(to)g(mak)o(e)f(it)g(correct)j(C)d(so)h(it)g(is)g (long)f(long.)1875 1212 y Fi(>)j Fh(\(Oct\))143 1297 y Fi(\017)23 b Fq(P)o(age)14 b(41,)g(lines)j(16-18)d(reads)189 1354 y(A)19 b Fb(empt)o(y)g Fq(status)g(is)h(a)f(status)g(whic)o(h)i (is)f(set)f(to)g(return)h Fj(tag)g(=)g(MPI)p 1460 1354 14 2 v 16 w(ANY)p 1568 1354 V 17 w(T)l(A)o(G)p Fq(,)f Fj(source)i(=)189 1410 y(MPI)p 274 1410 V 15 w(ANY)p 381 1410 V 18 w(SOURCE)p Fq(,)11 b(and)g(is)h(also)e(in)o(ternally)j (con\014gured)e(so)g(that)f(calls)h(to)g Fj(MPI)p 1601 1410 V 15 w(GET)p 1704 1410 V 17 w(COUNT)189 1467 y Fq(and)k Fj(MPI)p 362 1467 V 16 w(GET)p 466 1467 V 17 w(ELEMENTS)g Fq(return)g Fj(count)i(=)e(0)p Fq(.)189 1523 y(but)g(should)h(read)189 1580 y(A)j Fb(empt)o(y)g Fq(status)g(is)h(a)f(status)g(whic)o(h)i(is)f (set)f(to)g(return)h Fj(tag)g(=)g(MPI)p 1460 1580 V 16 w(ANY)p 1568 1580 V 17 w(T)l(A)o(G)p Fq(,)f Fj(source)i(=)189 1636 y(MPI)p 274 1636 V 15 w(ANY)p 381 1636 V 18 w(SOURCE)p Fq(,)c Fj(erro)o(r)e(=)i(MPI)p 842 1636 V 16 w(SUCCESS)p Fq(,)h(and)f(is)g(also)g(in)o(ternally)h(con\014gured)f(so)g(that)189 1693 y(calls)f(to)e Fj(MPI)p 430 1693 V 16 w(GET)p 534 1693 V 17 w(COUNT)i Fq(and)f Fj(MPI)p 893 1693 V 16 w(GET)p 997 1693 V 17 w(ELEMENTS)g Fq(return)g Fj(count)i(=)e(0)p Fq(.)189 1865 y Fm(Discussion)o(:)189 1933 y Fl(Filling)d(in)h(the)h (error)h(\014eld)f(got)g(forgotten.)1875 1989 y Fi(?)i Fh(\(Oct\))143 2074 y Fi(\017)23 b Fq(P)o(age)14 b(71,)g(line)j(10)e (reads)189 2131 y(and)g(do)g(not)g(a\013ect)f(the)i(the)f(con)o(ten)o (t)g(of)f(a)h(message)189 2187 y(but)g(should)h(read)189 2244 y(and)f(do)g(not)g(a\013ect)f(the)i(con)o(ten)o(t)e(of)h(a)g (message)189 2416 y Fm(Discussion)o(:)189 2484 y Fl(This)e(is)h(a)g (trivial)e(t)o(yp)q(o)i(of)f(ha)o(ving)g(the)h(2)g(times.)143 2626 y Fi(\017)23 b Fq(P)o(age)14 b(74,)g(lines)j(39-45)d(read)189 2682 y(A)i(datat)o(yp)q(e)f(ma)o(y)g(sp)q(ecify)i(o)o(v)o(erlapping)g (en)o(tries.)22 b(The)16 b(use)g(of)g(suc)o(h)g(a)f(datat)o(yp)q(e)h (in)g(a)g(receiv)o(e)-32 46 y Fh(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 16 18 16 17 bop 75 -100 a Fq(16)947 b Fg(CHAPTER)15 b(3.)35 b(MISCELLANY)17 b(F)o(OR)e(1.2)189 45 y Fq(op)q(eration)f(is)h (erroneous.)k(\(This)14 b(is)h(erroneous)f(ev)o(en)g(if)h(the)f(actual) g(message)g(receiv)o(ed)h(is)g(short)189 102 y(enough)g(not)g(to)g (write)g(an)o(y)g(en)o(try)g(more)f(than)h(once.\))189 174 y(A)i(datat)o(yp)q(e)g(ma)o(y)g(sp)q(ecify)i(o)o(v)o(erlapping)g (en)o(tries.)27 b(If)18 b(suc)o(h)g(a)g(datat)o(yp)q(e)f(is)h(used)g (in)h(a)e(receiv)o(e)189 231 y(op)q(eration,)d(that)h(is,)g(if)g(some)g (part)f(of)h(the)f(receiv)o(e)i(bu\013er)f(is)h(written)f(more)f(than)h (once)g(b)o(y)g(the)189 287 y(receiv)o(e)h(op)q(eration,)f(then)g(the)h (call)g(is)g(erroneous.)189 359 y(The)i(\014rst)f(part)g(w)o(as)h(an)f Fj(MPI-1.1)g Fq(addition.)29 b(The)18 b(second)g(part)f(o)o(v)o(erlaps) h(with)g(it.)28 b(The)18 b(old)189 416 y(text)c(will)j(b)q(e)f(remo)o (v)o(ed)f(so)g(it)g(no)o(w)g(reads)189 472 y(A)h(datat)o(yp)q(e)f(ma)o (y)g(sp)q(ecify)i(o)o(v)o(erlapping)g(en)o(tries.)22 b(The)16 b(use)g(of)g(suc)o(h)g(a)f(datat)o(yp)q(e)h(in)g(a)g(receiv)o (e)189 529 y(op)q(eration)e(is)h(erroneous.)k(\(This)14 b(is)h(erroneous)f(ev)o(en)g(if)h(the)f(actual)g(message)g(receiv)o(ed) h(is)g(short)189 585 y(enough)g(not)g(to)g(write)g(an)o(y)g(en)o(try)g (more)f(than)h(once.\))189 698 y Fm(Discussion)o(:)189 764 y Fl(The)f(in)o(ten)o(t)h(is)f(to)g(remo)o(v)o(e)f(duplicate)h (text)h(whic)o(h)f(seems)h(to)f(ha)o(v)o(e)g(b)q(een)h(left.)k(No)14 b(c)o(hange)h(in)f(meaning)189 814 y(is)f(in)o(tended.)143 950 y Fi(\017)23 b Fq(P)o(age)14 b(90,)g(line)j(3)e(reads)189 1006 y(MPI)p 281 1006 14 2 v 16 w(P)o(ac)o(k)p 393 1006 V 16 w(size\(coun)o(t,)g(MPI)p 724 1006 V 16 w(CHAR,)h(&k2\);)189 1063 y(but)f(should)h(read)189 1119 y(MPI)p 281 1119 V 16 w(P)o(ac)o(k)p 393 1119 V 16 w(size\(coun)o(t,)f(MPI)p 724 1119 V 16 w(CHAR,)h(comm,)e(&k2\);)189 1289 y Fm(Discussion)o(:)189 1355 y Fl(This)f(is)h(a)g(minor)e(\014x)i(to)f(an)h(example)f(for)g(a)h (missing)e(argumen)o(t.)143 1490 y Fi(\017)23 b Fq(P)o(age)14 b(90,)g(line)j(10)e(reads)189 1547 y(MPI)p 281 1547 V 16 w(P)o(ac)o(k\(c)o(hr,)f(coun)o(t,)g(MPI)p 726 1547 V 17 w(CHAR,)h(&lbuf,)h(k,)f(&p)q(osition,)h(comm\);)189 1603 y(but)f(should)h(read)189 1660 y(MPI)p 281 1660 V 16 w(P)o(ac)o(k\(c)o(hr,)e(coun)o(t,)g(MPI)p 726 1660 V 17 w(CHAR,)h(lbuf,)h(k,)f(&p)q(osition,)h(comm\);)189 1829 y Fm(Discussion)o(:)189 1895 y Fl(This)e(is)h(a)f(minor)f(\014x)i (to)g(an)f(example)g(for)g(an)g(argumen)o(t)g(that)h(is)g(declared)g (to)g(b)q(e)g(a)g(p)q(oin)o(ter)g(and)f(then)189 1945 y(passed)h(with)e(&.)143 2081 y Fi(\017)23 b Fq(P)o(age)14 b(109,)g(lines)j(26/27)d(and)h(page)g(110,)f(lines)j(28/29)d(reads)189 2137 y(The)h Fk(j)p Fq(th)g(blo)q(c)o(k)g(of)g(data)f(sen)o(t)h(from)f (eac)o(h)h(pro)q(cess)g(is)h(receiv)o(ed)g(b)o(y)f(ev)o(ery)f(pro)q (cess)i(and)f(placed)189 2194 y(in)h(the)f Fk(j)p Fq(th)g(blo)q(c)o(k)h (of)f(the)g(bu\013er)g Fj(recvbuf)p Fq(.)189 2250 y(but)g(should)h (read)189 2307 y(The)g(blo)q(c)o(k)i(of)e(data)g(sen)o(t)g(from)g(the)g Fk(j)p Fq(th)h(pro)q(cess)g(is)g(receiv)o(ed)g(b)o(y)g(ev)o(ery)f(pro)q (cess)h(and)g(placed)189 2363 y(in)f(the)f Fk(j)p Fq(th)g(blo)q(c)o(k)h (of)f(the)g(bu\013er)g Fj(recvbuf)p Fq(.)189 2533 y Fm(Discussion)o(:) 189 2598 y Fl(This)e(is)g(related)h(to)f(a)f(subtle)i(p)q(oin)o(t)f(in) g(usage.)18 b(The)13 b(sendbuf)h(is)f(a)g(single)g(item)f(whereas)i (the)g(recvbuf)g(is)189 2648 y(an)f(\\arra)o(y")g(of)h(items.)1967 46 y Fh(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 17 19 17 18 bop 75 -100 a Fg(3.10.)34 b(MINOR)16 b(CORRECTIONS)1109 b Fq(17)143 45 y Fi(\017)23 b Fq(P)o(age)14 b(117,)g(lines)j(22/23)d (reads)189 102 y(MPI)h(pro)o(vides)g(sev)o(en)h(suc)o(h)f(prede\014ned) i(datat)o(yp)q(es.)189 158 y(but)e(should)h(read)189 214 y(MPI)f(pro)o(vides)g(nine)i(suc)o(h)e(prede\014ned)i(datat)o(yp)q (es.)189 442 y Fm(Discussion)o(:)189 509 y Fl(This)c(is)h(an)g(ob)o (vious)f(mistak)o(e.)k(There)e(are)f(nine)g(listed)g(on)f(line)h (28-40.)143 647 y Fi(\017)23 b Fq(P)o(age)14 b(122,)g(lines)j(35-36)d (read)204 704 y Fj(MPI)p 289 704 14 2 v 16 w(OP)p 367 704 V 16 w(FREE\()i(op\))230 893 y Fl(IN)156 b Fj(op)541 b Fl(op)q(eration)14 b(\(handle\))189 1035 y Fq(but)h(should)h(read)204 1091 y Fj(MPI)p 289 1091 V 16 w(OP)p 367 1091 V 16 w(FREE\()g(op\))230 1280 y Fl(INOUT)63 b Fj(op)541 b Fl(op)q(eration)14 b(\(handle\))189 1463 y Fm(Discussion)o(:)189 1530 y Fl(This)j(app)q(ears)g(to)g(ha)o(v) o(e)g(b)q(een)h(a)f(mistak)o(e.)26 b(This)17 b(function)g(frees)h(the)g (function)e(p)q(oin)o(ter)i(and)f(sets)h(its)189 1580 y(v)n(alue)10 b(to)h(MPI)p 424 1580 13 2 v 15 w(OP)p 499 1580 V 15 w(NULL.)g(Th)o(us,)g(it)g(is)g(b)q(oth)g(input)g(and)f (output)i(as)f(are)g(the)g(free)h(routines)g(for)e(Groups,)189 1629 y(Comms,)g(etc.)143 1768 y Fi(\017)23 b Fq(P)o(age)14 b(125,)g(line)j(1)e(reads)189 1824 y(CALL)h(MPI)p 420 1824 14 2 v 16 w(ALLREDUCE\(sum,)f(c,)g(n,)g(MPI)p 1040 1824 V 17 w(REAL,)g(MPI)p 1300 1824 V 17 w(SUM,)g(0,)f(comm,)g(ierr\)) 189 1881 y(but)h(should)h(read)189 1937 y(CALL)g(MPI)p 420 1937 V 16 w(ALLREDUCE\(sum,)f(c,)g(n,)g(MPI)p 1040 1937 V 17 w(REAL,)g(MPI)p 1300 1937 V 17 w(SUM,)g(comm,)f(ierr\))189 2108 y Fm(Discussion)o(:)189 2175 y Fl(The)g(extra)h(argumen)o(t)e(app) q(ears)j(to)e(b)q(e)h(left)f(o)o(v)o(er)g(from)f(the)i(previous)f (example)f(with)h(MPI)p 1670 2175 13 2 v 16 w(REDUCE.)189 2225 y(This)f(is)h(a)g(mistak)o(e.)143 2363 y Fi(\017)23 b Fq(P)o(age)14 b(194,)g(line)j(48)e(reads)189 2420 y Fj(MPI)p 274 2420 14 2 v 15 w(ERRHANDLER)p 582 2420 V 18 w(CREA)l(TE\(FUNCTION,)h(HANDLER,)g(IERROR\))189 2476 y Fq(but)f(should)h(read)189 2533 y Fj(MPI)p 274 2533 V 15 w(ERRHANDLER)p 582 2533 V 18 w(CREA)l(TE\(FUNCTION,)g(ERRHANDLER,) h(IERROR\))189 2704 y Fm(Discussion)o(:)-32 46 y Fh(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 18 20 18 19 bop 75 -100 a Fq(18)947 b Fg(CHAPTER)15 b(3.)35 b(MISCELLANY)17 b(F)o(OR)e(1.2)189 45 y Fl(This)e(is)h(a)g(small)d(t)o (yp)q(o)j(to)g(matc)o(h)f(all)f(the)j(other)f(names)f(for)h(this)g (argumen)o(t.)143 186 y Fi(\017)23 b Fq(P)o(age)14 b(196,)g(lines)j (1-2)e(reads)204 243 y Fj(MPI)p 289 243 14 2 v 16 w(ERRHANDLER)p 598 243 V 17 w(FREE\()h(errhandler)f(\))230 433 y Fl(IN)156 b Fj(errhandler)397 b Fl(MPI)14 b(error)h(handler)f(\(handle\))189 576 y Fq(but)h(should)h(read)204 633 y Fj(MPI)p 289 633 V 16 w(ERRHANDLER)p 598 633 V 17 w(FREE\()g(errhandler)f(\))230 823 y Fl(INOUT)63 b Fj(errhandler)397 b Fl(MPI)14 b(error)h(handler)f (\(handle\))189 1007 y Fm(Discussion)o(:)189 1075 y Fl(This)c(app)q (ears)h(to)f(ha)o(v)o(e)h(b)q(een)g(a)f(mistak)o(e.)16 b(This)10 b(function)g(frees)i(the)f(function)f(p)q(oin)o(ter)g(and)h (sets)g(its)g(v)n(alue)189 1125 y(to)h(MPI)p 321 1125 13 2 v 15 w(ERRHANDLER)p 636 1125 V 14 w(NULL.)g(Th)o(us,)h(it)f(is)g (b)q(oth)h(input)f(and)g(output)h(as)f(are)h(the)g(free)g(routines)g (for)189 1175 y(Groups,)g(Comms,)e(etc.)143 1316 y Fi(\017)23 b Fq(P)o(age)14 b(203,)g(line)j(1)e(reads)189 1372 y Fj(MPI)p 274 1372 14 2 v 15 w(PCONTROL\(level\))189 1429 y Fq(but)g(should)h(read)189 1485 y Fj(MPI)p 274 1485 V 15 w(PCONTROL\(LEVEL\))189 1657 y Fm(Discussion)o(:)189 1726 y Fl(This)d(is)h(a)g(minor)e(t)o(yp)q(o.)143 1867 y Fi(\017)23 b Fq(P)o(age)14 b(210,)g(line)j(44)e(reads)189 1923 y Fd(MPI)p 266 1923 13 2 v 14 w(PENDING)189 1980 y Fq(but)g(should)h(read)189 2036 y Fd(MPI)p 266 2036 V 14 w(ERR)p 359 2036 V 14 w(PENDING)189 2209 y Fm(Discussion)o(:)189 2277 y Fl(This)d(is)h(a)g(t)o(yp)q(o.)k(See)c(page)g(197,)f(line)g(20)h (for)f(the)i(original)d(de\014nition.)143 2418 y Fi(\017)23 b Fq(P)o(age)14 b(211,)g(line)j(44)e(reads)189 2475 y Fd(MPI)p 266 2475 V 14 w(DOUBLE)p 445 2475 V 14 w(COMPLEX)189 2531 y Fq(but)g(should)h(b)q(e)g(mo)o(v)o(ed)f(to)f(P)o(age)h(212,)f (line)j(22)d(since)j(it)e(is)h(an)f(optional)h(F)l(ortran)e(datat)o(yp) q(e.)189 2647 y Fm(Discussion)o(:)1967 46 y Fh(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 19 21 19 20 bop 75 -100 a Fg(3.10.)34 b(MINOR)16 b(CORRECTIONS)1109 b Fq(19)189 45 y Fl(On)20 b(page)h(19,)f(lines)h(2-3,)f(it)g(clearly)h (states)g(this)f(is)h(an)f(optional)f(F)m(ortran)h(datat)o(yp)q(e.)37 b(It)21 b(is)f(listed)189 95 y(incorrectly)14 b(in)g(the)g(app)q (endix.)1875 150 y Fi(>)i Fh(\(Oct\))143 236 y Fi(\017)23 b Fq(P)o(age)14 b(212,)g(add)h(new)h(lines)h(of)d(text)h(at)g(line)i (22)d(and)i(line)g(25)f(to)g(read:)189 292 y(etc.)189 349 y(Th)o(us,)f(the)i(text)e(will)j(no)o(w)e(read:)189 518 y Fk(/*)23 b(optional)g(datatypes)g(\(Fortran\))f(*/)189 574 y(MPI_INTEGER1)189 631 y(MPI_INTEGER2)189 687 y(MPI_INTEGER4)189 744 y(MPI_REAL2)189 800 y(MPI_REAL4)189 857 y(MPI_REAL8)189 913 y(etc.)189 1026 y(/*)h(optional)g(datatypes)g(\(C\))g(*/)189 1082 y(MPI_LONG_LONG_INT)189 1139 y(etc.)189 1292 y Fm(Discussion)o(:) 189 1360 y Fl(On)15 b(page)g(19,)f(line)g(6,)h(concludes)h(the)f(list)f (of)h(optional)e(datat)o(yp)q(es)j(with)e(etc.)22 b(It)15 b(w)o(as)g(not)g(in)o(tended)g(to)189 1410 y(b)q(e)f(a)g(complete)g (list)g(of)f(all)g(p)q(ossible)i(ones.)20 b(Some)13 b(ha)o(v)o(e)h(in)o (terpreted)i(the)e(app)q(endix)h(to)f(b)q(e)h(a)f(complete)189 1460 y(list)f(since)h(only)f(the)h(ones)h(explictly)e(on)g(page)h(19)f (are)h(listed.)k(Adding)13 b(etc.,)h(to)g(the)g(F)m(ortran)f(and)h(C)f (list)189 1510 y(in)g(the)i(app)q(endix)e(will)g(mak)o(e)f(it)i(clear)g (these)h(are)g(only)e(some)g(of)g(the)i(p)q(ossible)f(optional)e(datat) o(yp)q(es.)1875 1565 y Fi(?)k Fh(\(Oct\))143 1651 y Fi(\017)23 b Fq(P)o(age)14 b(213,)g(line)j(28.)i(The)d(follo)o(wing)g(text)e (should)i(b)q(e)g(added:)189 1707 y(/*)9 b(Prede\014ned)j(functions)f (in)g(C)e(and)i(F)l(ortran)e(*/)g(MPI)p 1137 1707 14 2 v 17 w(NULL)p 1278 1707 V 17 w(COPY)p 1428 1707 V 17 w(FN)g(MPI)p 1607 1707 V 17 w(NULL)p 1748 1707 V 17 w(DELETE)p 1954 1707 V 16 w(FN)189 1764 y(MPI)p 281 1764 V 16 w(DUP)p 397 1764 V 16 w(FN)189 1880 y Fm(Discussion)o(:)189 1948 y Fl(These)15 b(are)f(missing)e(and)i(de\014ned)h(on)e(pages)i(169,)d (line)i(17)f(and)h(p.)k(170,)13 b(line)g(4.)143 2089 y Fi(\017)23 b Fq(P)o(age)14 b(220,)g(lines)j(19-20)d(reads)189 2146 y Fj(int)i(double)g(MPI)p 479 2146 V 16 w(Wtime\(void\))189 2202 y(int)g(double)g(MPI)p 479 2202 V 16 w(Wtick\(void\))189 2258 y Fq(but)f(should)h(read)189 2315 y Fj(double)g(MPI)p 413 2315 V 16 w(Wtime\(void\))189 2371 y(double)g(MPI)p 413 2371 V 16 w(Wtick\(void\))189 2544 y Fm(Discussion)o(:)189 2612 y Fl(This)d(w)o(as)h(a)g(mistak)o(e)e(caused)j(b)o(y)f(the)g (macros.)-32 46 y Fh(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 20 22 20 21 bop 75 -100 a Fq(20)947 b Fg(CHAPTER)15 b(3.)35 b(MISCELLANY)17 b(F)o(OR)e(1.2)143 45 y Fi(\017)23 b Fq(P)o(age)14 b(222,)g(line)j(34)e(reads)189 102 y Fj(INTEGER)f (REQUEST,)g(COUNT,)g(D)o(A)l(T)l(A)l(TYPE,)g(DEST,)g(T)l(A)o(G,)f (COMM,)g(REQUEST,)i(IERROR)189 158 y Fq(but)g(should)h(read)189 214 y Fj(INTEGER)f(COUNT,)h(D)o(A)l(T)l(A)l(TYPE,)g(DEST,)f(T)l(A)o(G,) g(COMM,)g(REQUEST,)h(IERROR)189 387 y Fm(Discussion)o(:)189 455 y Fl(An)e(extra)g(argumen)o(t)f(got)g(added.)19 b(It)14 b(needs)h(to)f(b)q(e)g(remo)o(v)o(ed.)143 596 y Fi(\017)23 b Fq(P)o(age)14 b(222,)g(line)j(38)e(reads)189 653 y Fj(INTEGER)f(REQUEST,)g(COUNT,)g(D)o(A)l(T)l(A)l(TYPE,)g(DEST,)g(T)l(A) o(G,)f(COMM,)g(REQUEST,)i(IERROR)189 709 y Fq(but)g(should)h(read)189 766 y Fj(INTEGER)f(COUNT,)h(D)o(A)l(T)l(A)l(TYPE,)g(DEST,)f(T)l(A)o(G,) g(COMM,)g(REQUEST,)h(IERROR)189 938 y Fm(Discussion)o(:)189 1006 y Fl(An)e(extra)g(argumen)o(t)f(got)g(added.)19 b(It)14 b(needs)h(to)f(b)q(e)g(remo)o(v)o(ed.)143 1147 y Fi(\017)23 b Fq(P)o(age)14 b(227,)g(lines)j(19-20)d(reads)189 1204 y Fj(MPI)p 274 1204 14 2 v 15 w(INTERCOMM)p 563 1204 V 17 w(MERGE\(INTERCOMM,)i(HIGH,)f(INTRA)o(COMM,)g(IERROR\))189 1260 y(INTEGER)g(INTERCOMM,)h(INTRA)o(COMM,)f(IERROR)189 1317 y Fq(but)g(should)h(read)189 1373 y Fj(MPI)p 274 1373 V 15 w(INTERCOMM)p 563 1373 V 17 w(MERGE\(INTERCOMM,)g(HIGH,)f (NEWINTRA)o(COMM,)g(IERROR\))189 1430 y(INTEGER)g(INTERCOMM,)h (NEWINTRA)o(COMM,)f(IERROR)189 1602 y Fm(Discussion)o(:)189 1670 y Fl(This)g(mak)o(es)e(it)i(matc)o(h)f(the)i(de\014nition)e(in)h (the)g(text)h(on)f(P)o(age)g(159,)f(lines)h(34-36.)20 b(It)15 b(only)g(c)o(hanges)g(the)189 1720 y(name)d(of)i(the)g(argumen) o(t.)143 1861 y Fi(\017)23 b Fq(P)o(age)14 b(228,)g(line)j(46)e(reads) 189 1918 y Fj(MPI)p 274 1918 V 15 w(ERRHANDLER)p 582 1918 V 18 w(CREA)l(TE\(FUNCTION,)h(HANDLER,)g(IERROR\))189 1974 y Fq(but)f(should)h(read)189 2031 y Fj(MPI)p 274 2031 V 15 w(ERRHANDLER)p 582 2031 V 18 w(CREA)l(TE\(FUNCTION,)g (ERRHANDLER,)h(IERROR\))189 2203 y Fm(Discussion)o(:)189 2271 y Fl(This)c(c)o(hange)i(mak)o(es)d(the)j(app)q(endix)f(consistan)o (t)g(with)g(c)o(hange)g(on)g(p.)k(194,)12 b(line)i(48.)143 2412 y Fi(\017)23 b Fq(P)o(age)14 b(229,)g(line)j(33)e(reads)189 2469 y Fj(MPI)p 274 2469 V 15 w(PCONTROL\(level\))189 2525 y Fq(but)g(should)h(read)189 2582 y Fj(MPI)p 274 2582 V 15 w(PCONTROL\(LEVEL\))1967 46 y Fh(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 21 23 21 22 bop 75 -100 a Fg(3.10.)34 b(MINOR)16 b(CORRECTIONS)1109 b Fq(21)189 45 y Fm(Discussion)o(:)189 114 y Fl(This)13 b(is)h(to)g(mak)o(e)e(the)j(language)e(binding)g(the)h(same)f(as)h(the) h(c)o(hange)f(to)g(P)o(age)f(203,)g(line)g(1.)-32 46 y Fh(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From mpi-comm-human@mcs.anl.gov Tue Oct 1 22:20:39 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id WAA19375; Tue, 1 Oct 1996 22:20:38 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id UAA23838 for mpi-comm-out; Tue, 1 Oct 1996 20:58:56 -0500 Received: from Aurora.CS.MsState.Edu (aurora.cs.msstate.edu [130.18.208.91]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id NAA17219; Tue, 1 Oct 1996 13:49:04 -0500 Received: (tony@localhost); by Aurora.CS.MsState.Edu (SMI-8.6/7.0m-FWP-MsState); id NAA14832; Tue, 1 Oct 1996 13:48:59 -0500 Date: Tue, 1 Oct 1996 13:48:59 -0500 From: Tony Skjellum Message-Id: <199610011848.NAA14832@Aurora.CS.MsState.Edu> To: mpi-comm@mcs.anl.gov, mpi-core@mcs.anl.gov Subject: New collective chapter Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk The new collective chapter, as mentioned on mpi-coll group, is available. -Tony See ftp://aurora.cs.msstate.edu/pub/mpi/coll-2-30sep96.ps (or .ps.gz) Please review as it is on time and ready for firm consideration, following updates since last meeting. From mpi-comm-human@mcs.anl.gov Tue Oct 8 00:05:01 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id AAA19824; Tue, 8 Oct 1996 00:04:58 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id WAA02177 for mpi-comm-out; Mon, 7 Oct 1996 22:49:46 -0500 Received: from mcs.anl.gov (donner.mcs.anl.gov [140.221.5.134]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id WAA02163; Mon, 7 Oct 1996 22:49:26 -0500 Message-Id: <199610080349.WAA02163@antares.mcs.anl.gov> To: mpi@whale.cdac.ernet.in (MPI account) cc: mpi-bugs@mcs.anl.gov, gropp@mcs.anl.gov, mpi-comm@mcs.anl.gov, mpi-impl@mcs.anl.gov, mpi-maint@antares.mcs.anl.gov Subject: Re: [MPI #2000] MPI related Problems on SMPs In-reply-to: Your message of "Mon, 07 Oct 1996 21:05:09 +0700." <9610071535.AA05076@whale.parcom.ernet.in> X-Request-Do: resolve Date: Mon, 07 Oct 1996 22:49:26 -0500 From: Rusty Lusk Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk | MPI : MPICH version 1.12 over p4 (tcp/ip) | You should get 1.0.13 and apply the patches available at http://www.mcs.anl.gov/mpi/mpich (get the big patch) | Configuration option: -arch=solaris -device=ch_p4 -comm=shared | We have not extensively testing this configuration with solaris but it should work. | | Questions: | 1. What is the suitable programming model for our architecture with MPIlike SIMD,MIMD.. MPI provides access to the message-passing model. The shared memory will be used to provide fast message passing within a machine. | 2. How do we place the tasks for SIMD or MIMD | eg; in the pg file You should read the sections on this in the User's Manual in the doc directory. | 3. Our MPI v1.12 with -comm=shared has been configured on a 8 node Ultra2s and with SIMD code ( all same executables ) it goes through , but with MIMD code ( different executables sharing memory ) it does'nt work , Why? To get shared memory, you must fork, thus getting multiple copies of the same executable. | 4. Is there any other MPI version specific for SMPs or any other configuration option to be done for MPICH | You can use your configuration without the -comm=shared option, getting sockets for all communication. From owner-mpi-comm@CS.UTK.EDU Mon Nov 18 10:33:01 1996 Return-Path: Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id KAA12632; Mon, 18 Nov 1996 10:33:01 -0500 Received: from localhost (root@localhost) by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id KAA12051; Mon, 18 Nov 1996 10:25:46 -0500 Received: from wotan.inpro.de ([193.96.109.254]) by CS.UTK.EDU with ESMTP (cf v2.9s-UTK) id KAA12029; Mon, 18 Nov 1996 10:25:38 -0500 Received: from bonnie.inpro.de by wotan.inpro.de with ESMTP (1.39.111.2/GEN-1.2.10) via EUnet id AA149719117; Mon, 18 Nov 1996 15:58:37 +0100 From: "R.Kuepfer" Message-Id: <199611181516.AA253180177@bonnie.inpro.de> Received: by bonnie.inpro.de (1.37.109.18/GEN-1.1.5) id AA253180177; Mon, 18 Nov 1996 16:16:17 +0100 Subject: add to list To: mpi-comm@CS.UTK.EDU Date: Mon, 18 Nov 96 16:16:17 MEZ Mailer: Elm [revision: 70.85] please add kuepfer@inpro.de to the mailing list. thanks rodrigo kuepfer From mpi-comm-human@mcs.anl.gov Mon Dec 16 23:14:02 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id XAA26927; Mon, 16 Dec 1996 23:14:01 -0500 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id VAA14780 for mpi-comm-out; Mon, 16 Dec 1996 21:55:07 -0600 Received: from palrel3.hp.com (palrel3.hp.com [15.253.88.10]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id QAA10522 for ; Mon, 16 Dec 1996 16:39:10 -0600 Received: from hplrss.hpl.hp.com (hplrss.hpl.hp.com [15.9.128.220]) by palrel3.hp.com with ESMTP (8.7.5/8.7.3) id OAA24321 for ; Mon, 16 Dec 1996 14:39:24 -0800 (PST) Received: (from schreibr@localhost) by hplrss.hpl.hp.com (8.7.1/8.7.1) id OAA27477 for mpi-comm@mcs.anl.gov; Mon, 16 Dec 1996 14:40:38 -0800 (PST) Date: Mon, 16 Dec 1996 14:40:38 -0800 (PST) From: Rob Schreiber Message-Id: <199612162240.OAA27477@hplrss.hpl.hp.com> To: mpi-comm@mcs.anl.gov Subject: PPoPP Conference Mime-Version: 1.0 Content-Type: text/plain; charset=X-roman8 Content-Transfer-Encoding: 7bit Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk On behalf of the program committee, I would like to invite submissions to the ACM PPoPP (Principles and Practices of Parallel Programming) Conference. The developments in MPI implementation and use are very apropriate for this years conference, which empahsizes practical aspects of parallel programming. The conference will take place July 18-21, 1997 in Las Vegas, Nevada. For information about the conference, see the web page: http://www.tc.cornell.edu/PPoPP/ The deadline for submission, December 13, has passed; it can be extended by contacting the program committee chair, Keshav Pingali (pingali@cs.cornell.edu). Please pass along this invitation to other interested HPF users, implementors, and researchers. -- Rob Schreiber From mpi-comm-human@mcs.anl.gov Fri Dec 20 09:56:40 1996 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id JAA23782; Fri, 20 Dec 1996 09:56:39 -0500 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id IAA16301 for mpi-comm-out; Fri, 20 Dec 1996 08:23:36 -0600 Received: from acacia.sucs.soton.ac.uk (acacia.sucs.soton.ac.uk [152.78.128.232]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id IAA16275; Fri, 20 Dec 1996 08:22:35 -0600 Received: from bright.ecs.soton.ac.uk (bright.ecs.soton.ac.uk [152.78.64.201]) by acacia.sucs.soton.ac.uk (8.8.2/server) with SMTP id OAA00851; Fri, 20 Dec 1996 14:22:22 GMT Received: from bacchus by bright.ecs.soton.ac.uk; Fri, 20 Dec 96 14:21:39 GMT Message-Id: <32BAA133.59E2B600@ecs.soton.ac.uk> Date: Fri, 20 Dec 1996 14:22:43 +0000 From: Robin Allen Organization: ECS X-Mailer: Mozilla 2.0 (X11; I; SunOS 4.1.3_U1 sun4c) Mime-Version: 1.0 To: mpi-core@mcs.anl.gov Cc: mpi-coll@mcs.anl.gov, mpi-comm@mcs.anl.gov, hpff@cs.rice.edu Subject: Announcement: Upcoming Software Tools Workshop Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk Hi all. This is to announce an upcoming European software tools workshop, Brussels, 5 February 1997. If you are interested in attending, please reply to the address at the end of the message, not the mail address of this message! Thanks. European Commission workshop for High-Performance Computing and Networking (HPCN) Software Tools Brussels, 5 February 1997 The European Commission recently engaged Smith System Engineering and the PAC to undertake a survey of HPCN software tools. The Commission's objectives in performing this survey were to assess the current status of HPCN tools and to develop a strategy for further investment and exploitation. The results of the survey will be presented by Smith and the PAC at the forthcoming workshop in Brussels. Details of the venue and agenda will be distributed in January. In addition to providing feedback on the findings of the survey, the workshop will be used to stimulate discussion on the 'way ahead' for European HPCN software tools. In particular, it will provide a forum for debate on software tool development, the EC's framework programme and the newly forming Technology Transfer Node (TTN) network. The workshop will last one day with the morning dedicated to presentation of the study's results and the EC's perspectives. In the afternoon, two parallel sessions will debate the current issues in HPCN software tool development. The findings of the two sessions will then be consolidated and discussed. If you are interested in taking part in the workshop please mail your details To: Dr Daron Green Fax: +44 1483 442304 Name: Address: Company: Tel: email: Stuart A Haire System Engineer ///////////////////////////////// / Smith System Engineering Ltd / / Surrey Research Park / / Guildford / / Surrey / / GU2 5YP / / e-mail: sahaire@smithsys.co.uk/ / tel: +44 (0)1483 442103 / / fax: +44 (0)1483 442304 / / mobile: 0468 946960 / ///////////////////////////////// From mpi-comm-human@mcs.anl.gov Sat Jan 11 20:35:46 1997 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id UAA01706; Sat, 11 Jan 1997 20:35:45 -0500 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id TAA12044 for mpi-comm-out; Sat, 11 Jan 1997 19:24:41 -0600 Received: from franklin.sdsc.edu (franklin.sdsc.edu [132.249.40.106]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id TAA27558 for ; Fri, 10 Jan 1997 19:55:24 -0600 Received: from flopsy.sdsc.edu (flopsy.sdsc.edu [132.249.34.127]) by franklin.sdsc.edu (8.8.3/8.8.3/SDSCserver-14) with SMTP id RAA19519 for ; Fri, 10 Jan 1997 17:55:32 -0800 (PST) From: Susan Lindsey Received: by flopsy.sdsc.edu (4.1/1.11-client) id AA04331; Fri, 10 Jan 97 17:55:32 PST Message-Id: <9701110155.AA04331@flopsy.sdsc.edu> Subject: subscribe mpi-comm lindsey@sdsc.edu To: mpi-comm@mcs.anl.gov Date: Fri, 10 Jan 1997 17:55:31 -0800 (PST) Reply-To: lindsey@sdsc.edu X-Susan'S-Band-O-The-Month: Sublime X-Mailer: ELM [version 2.4 PL25] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk subscribe mpi-comm lindsey@sdsc.edu From mpi-comm-human@mcs.anl.gov Mon Jan 13 19:25:23 1997 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id TAA19198; Mon, 13 Jan 1997 19:25:22 -0500 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id SAA23373 for mpi-comm-out; Mon, 13 Jan 1997 18:00:06 -0600 Received: from tbag.osc.edu (tbag.osc.edu [128.146.36.50]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id RAA22802 for ; Mon, 13 Jan 1997 17:37:57 -0600 Received: for gdburns@tbag.osc.edu by tbag.osc.edu (8.7.1/950822.1) id SAA24739; Mon, 13 Jan 1997 18:38:19 -0500 (EST) Date: Mon, 13 Jan 1997 18:38:19 -0500 (EST) From: Greg Burns Message-Id: <199701132338.SAA24739@tbag.osc.edu> To: mpi-comm@mcs.anl.gov Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk unsubscribe gdburns@osc.edu From mpi-comm-human@mcs.anl.gov Fri Jan 24 02:06:40 1997 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id CAA25463; Fri, 24 Jan 1997 02:06:40 -0500 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id AAA05267 for mpi-comm-out; Fri, 24 Jan 1997 00:48:43 -0600 Received: from hitiij.hitachi.co.jp (root@hitiij.hitachi.co.jp [133.145.224.3]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id AAA05262 for ; Fri, 24 Jan 1997 00:48:39 -0600 Received: from hcrlgw92.crl.hitachi.co.jp by hitiij.hitachi.co.jp (8.8.3+2.6Wbeta9/3.4Wbeta6-hitiij) id PAA20192; Fri, 24 Jan 1997 15:50:09 +0900 (JST) Received: from hcrlgw2.crl.hitachi.co.jp by hcrlgw92.crl.hitachi.co.jp (4.1/6.4J.6) id AA11417; Fri, 24 Jan 97 15:49:54 JST Received: from [133.144.18.33] by hcrlgw2.crl.hitachi.co.jp (5.61/6.4J.6) id AA16638; Fri, 24 Jan 97 15:49:52 +0900 Message-Id: <32E85E0E.78C9@crl.hitachi.co.jp> Date: Fri, 24 Jan 1997 16:00:30 +0900 From: Nobutoshi Sagawa Reply-To: sagawa@crl.hitachi.co.jp Organization: Hitachi Central Research Lab. X-Mailer: Mozilla 3.0 (Win16; I) Mime-Version: 1.0 To: mpi-comm@mcs.anl.gov Cc: KENTA.NAKAMURA@soft12.nichijo.soft.hitachi.co.jp Subject: typo? Content-Type: text/plain; charset=iso-2022-jp Content-Transfer-Encoding: 7bit Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk We seem to have found a typo error in MPI1.1. The C argument "recvtag" for MPI_Sendrecv is defined as "MPI_Dagatype" which we believe should be just "int". Forgive us if this issue has been already raised. --------------------------------------------------------------- Kenta Nakamura Software Development Center, Hitachi Ltd. Nobutoshi Sagawa Processor System Department Central Research Laboratory, Hitachi Ltd. --------------------------------------------------------------- From owner-mpi-comm@CS.UTK.EDU Sun Jun 29 15:07:32 1997 Return-Path: Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id PAA22519; Sun, 29 Jun 1997 15:07:31 -0400 Received: from localhost (root@localhost) by CS.UTK.EDU with SMTP (cf v2.9s-UTK) id PAA06215; Sun, 29 Jun 1997 15:11:52 -0400 Received: from home.iis.com.br (root@home.iis.com.br [200.255.216.6]) by CS.UTK.EDU with ESMTP (cf v2.9s-UTK) id PAA06208; Sun, 29 Jun 1997 15:11:47 -0400 Received: from LOGIN.iis.com.br (rio-as7-tty10.iis.com.br [200.255.216.169]) by home.iis.com.br (8.7.5/8.6.11) with SMTP id QAA28956 for ; Sun, 29 Jun 1997 16:08:49 -0300 Message-Id: <1.5.4.32.19970629190850.0066ead8@hotmail.com> X-Sender: rmalmeida@hotmail.com (Unverified) X-Mailer: Windows Eudora Light Version 1.5.4 (32) Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Sun, 29 Jun 1997 16:08:50 -0300 To: mpi-comm@CS.UTK.EDU From: Reginaldo Almeida From mpi-comm-human@mcs.anl.gov Tue Jul 22 18:43:59 1997 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id SAA17910; Tue, 22 Jul 1997 18:43:59 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id RAA22006 for mpi-comm-out; Tue, 22 Jul 1997 17:20:56 -0500 Received: from beowulf.ucsd.edu (greinman@beowulf.ucsd.edu [132.239.17.2]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id RAA21997 for ; Tue, 22 Jul 1997 17:20:49 -0500 Received: (from greinman@localhost) by beowulf.ucsd.edu (8.8.3/8.8.3/External-1.2) id PAA15024 for mpi-comm@mcs.anl.gov; Tue, 22 Jul 1997 15:20:43 -0700 (PDT) Date: Tue, 22 Jul 1997 15:20:43 -0700 (PDT) From: Glenn Reinman Message-Id: <199707222220.PAA15024@beowulf.ucsd.edu> To: mpi-comm@mcs.anl.gov Subject: MPI v2.0 - shared memory and signals? Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk Hi, I'm interested in using shared memory between two processes running MPI. Is this to be implemented in the new version, and if so, will it be limited by the constraints of the operating system's shared memory primitives (i.e. UNIX only allows 6 shared memory segments per process). Also, are signals (i.e. some form of asynchronous interrupt, hopefully handled by some custom procedure) to be implemented in the new version? My intent is to handle contention at a processor by migrating "work" (rather than processes as in CONDOR) by signalling a slave to communicate results/data and terminate. I hope this is the correct forum to ask this question, and if it has already been discussed, if you could please direct me to the archives, I'd appreciate it. Thanks. Glenn From mpi-comm-human@mcs.anl.gov Wed Jul 23 10:43:13 1997 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id KAA27768; Wed, 23 Jul 1997 10:43:12 -0400 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id JAA07729 for mpi-comm-out; Wed, 23 Jul 1997 09:26:02 -0500 Received: from parrot.rz.kp.dlr.de (parrot.rz.kp.dlr.de [129.247.102.29]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id JAA07718 for ; Wed, 23 Jul 1997 09:25:54 -0500 Received: from parrot ([129.247.102.29]) by parrot.rz.kp.dlr.de (Netscape Mail Server v2.02) with SMTP id AAA10895 for ; Wed, 23 Jul 1997 16:25:02 +0200 Message-ID: <33D6143D.EE4@dlr.de> Date: Wed, 23 Jul 1997 16:25:01 +0200 From: Martin Strietzel Organization: DLR DV-HP X-Mailer: Mozilla 3.0 (X11; I; SunOS 5.5.1 sun4m) MIME-Version: 1.0 To: mpi-comm@mcs.anl.gov Subject: Process management Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk Hello, we'd like to communicate between to different programs using MPI intercommunicators. The point is how to start this programs in one MPI-world. We could do this by using MPICH and manipulating the proc-group files by hand, but I would prefer a MPI implementation which already allows to start two different executables with one 'mpirun' statement, or to spawn a new process by 'mpi_comm_spawn'? Does anybody know which implementation includes these MPI 2 functionalities already? Thanks, Martin -- Martin Strietzel Deutsche Forschungsanstalt fuer Luft- und Raumfahrt Linder Hoehe 51147 Koeln (Wahnheide) mailto:Martin.Strietzel@dlr.de Tel.: +49 (0)2203/601-2059 Fax: +49 (0)2203/601-2104 or +49 (0)2203/601-3620 From mpi-comm-human@mcs.anl.gov Mon Nov 16 13:44:39 1998 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id NAA16382; Mon, 16 Nov 1998 13:44:38 -0500 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id MAA23277 for mpi-comm-out; Mon, 16 Nov 1998 12:09:00 -0600 Received: from timbuk-e1.cray.com (timbuk-e1.cray.com [128.162.1.30]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id MAA23270; Mon, 16 Nov 1998 12:08:48 -0600 Received: from ledzep.cray.com (ledzep.cray.com [137.38.226.97]) by timbuk-e1.cray.com (8.8.8/CRI-gate-news-1.3) with ESMTP id MAA29531; Mon, 16 Nov 1998 12:08:40 -0600 (CST) Received: from ironwood-e185.cray.com (root@ironwood-e185.cray.com [128.162.185.212]) by ledzep.cray.com (8.8.8/craymail-smart) with ESMTP id MAA5482983; Mon, 16 Nov 1998 12:08:39 -0600 (CST) Received: from fsgi118.cray.com (fsgi118 [128.162.190.136]) by ironwood-e185.cray.com (8.8.4/ASC-news-e1.0) with ESMTP id MAA27571; Mon, 16 Nov 1998 12:08:28 -0600 (CST) From: Karl Feind Received: by fsgi118.cray.com (8.8.8/SGI-client.1.6) id MAA09295; Mon, 16 Nov 1998 12:05:51 -0600 (CST) Message-Id: <199811161805.MAA09295@fsgi118.cray.com> Date: Mon, 16 Nov 1998 12:05:51 -0600 (CST) To: mpi-comm@mcs.anl.gov, kaf@sgi.com Subject: MPI-2 thread safety and collectives Cc: judith@sgi.com, gropp@mcs.anl.gov Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk I'd like to get some opinions about interpreting an aspect of the MPI-2 thread safety specification which deals with collectives. ----------------------------------------------------------------------------- MPI-2 Standard, Section 8.7.1, paragraphs 1-3: Threads In a thread-compliant implementation, an MPI process is a process that may be multi-threaded. Each thread can issue MPI calls; however, threads are not separately addressable: a rank in a send or receive call identifies a process, not a thread. A message sent to a process can be received by any thread in this process. Rationale. This model corresponds to the POSIX model of interprocess communication: the fact that a process is multi-threaded, rather than single-threaded, does not affect the external interface of this process. MPI implementations where MPI `processes' are POSIX threads inside a single POSIX process are not thread-compliant by this definition (indeed, their ``processes'' are single-threaded). ( End of rationale.) Advice to users. It is the user's responsibility to prevent races when threads within the same application post conflicting communication calls. The user can make sure that two threads in the same process will not issue conflicting communication calls by using distinct communicators at each thread. ( End of advice to users.) ----------------------------------------------------------------------------- The basic question is simple. How do you define "conflicting communication calls"? There are two possible interpretations: 1) Any MPI collective call on the same communicator. or 2) The same MPI collective call on the same communicator. When I read the text of the standard, I tend to take interpretation #1, but one MPI Forum member I talked to recalls the forum specifically taking interpretation #2. Hence I want to get more feedback to ensure that I'm interpreting this correctly. I think that choosing interpretation #1 would be very desirable because it would permit some MPI collectives to be layered on top of other collectives without introducing thread-safety problems. As a common example, the ROMIO MPI I/O software is layered on MPI collectives. Interpretation #1 would permit ROMIO to be layered and still not violate thread-safety. Consider scenario S1 below, which illustrates this thread-safety issue as it affects the ROMIO MPI-2 I/O layered implementation. ROMIO function MPI_File_open() calls MPI_Allreduce() using the communicator passed by the user into MPI_File_open. Suppose the user had another thread executing an MPI_Allreduce() collective operation on the same communicator. If a race condition allows the threads to execute in different orders, we would get an incorrect result: Scenario S1 ----------- | Process 0 | Process 1 | --------- | --------- | | Time | thread A calls MPI_Allreduce | thread C calls MPI_Allreduce | | via MPI_File_open | | | | | | thread B calls MPI_Allreduce | thread D calls MPI_Allreduce | | | via MPI_File_open V | | Under interpretation #1, you could say that the user needs to ensure that all threads issuing any collective calls on the same communicator must be properly ordered by the user. With this interpretation, scenario S1 would be erroneous and the implementation wouldn't need to deal with the matching up of the collective calls on multiple threads. Notice that under both interpretations, a scenario like S2 should be erroneous because the same collective function is called by the two threads. Scenario S2 ----------- | Process 0 | Process 1 | --------- | --------- | | Time | thread A calls MPI_Allreduce | thread C calls MPI_Allreduce | | for MPI_MAX operation | for MPI_SUM operation | | | | | thread B calls MPI_Allreduce | thread D calls MPI_Allreduce | | for MPI_SUM operation | for MPI_MAX operation V | | Thanks for any opinions or recollections of forum discussions on this matter. Karl Feind E-Mail: kaf@sgi.com Silicon Graphics Phone: 612/683-5673 655F Lone Oak Drive Fax: 612/683-5276 Eagan, MN 55121 From mpi-comm-human@mcs.anl.gov Mon Nov 16 15:40:26 1998 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id PAA18659; Mon, 16 Nov 1998 15:40:26 -0500 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id OAA28787 for mpi-comm-out; Mon, 16 Nov 1998 14:03:00 -0600 Received: from Mpi.Mpi-softtech.Com (mpi.MPI-SoftTech.Com [208.152.187.97]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id OAA28594; Mon, 16 Nov 1998 14:00:21 -0600 Received: from localhost (tony@localhost); by Mpi.Mpi-softtech.Com using SMTP (8.8.5/7.0m-FWP-MsState); id OAA11782; Mon, 16 Nov 1998 14:01:33 -0600 (CST) Date: Mon, 16 Nov 1998 14:01:32 -0600 (CST) From: Tony Skjellum To: Karl Feind cc: mpi-comm@mcs.anl.gov, judith@sgi.com, gropp@mcs.anl.gov Subject: Re: MPI-2 thread safety and collectives In-Reply-To: <199811161805.MAA09295@fsgi118.cray.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk Karl, First of all, we discussed for a long time that if one serialized the sequence of collective operations in a set of threads (same communicator), and that if this sequence was the same across the communicator (independent of number of threads in each process), that this was prima facie legal. If this is not what was finally agreed on, then interpretation #1 is the right one in my opinion. Common implementations allow collectives to be insulated from each other by the semantics of the operations (with a backup that each collective has its own tag, but that is generally unneeded). That would suggest your interpretation #2, but that depends generally on the strategy used in MPICH and other implementations to utilize collective tags per operation. In fact, for most collective operations, the effective total order is enough to gain separation (avoid race condition) from multiple invocations of the same operation or different operations without the tag, so that anything that implicates this tag as a condition of thread safety is not desirable IMHO. -Tony Anthony Skjellum, PhD, President (tony@mpi-softtech.com) MPI Software Technology, Inc., Ste 201, 1 Research Blvd., Starkville, MS 39759 +1-(601)320-4300 x15; FAX: +1-(601)320-4301; http://www.mpi-softtech.com *The source for MPI, MPI-2, MPI/RT, Embedded & High Performance Middleware(tm)* On Mon, 16 Nov 1998, Karl Feind wrote: > > > I'd like to get some opinions about interpreting an aspect of the MPI-2 thread > safety specification which deals with collectives. > > ----------------------------------------------------------------------------- > MPI-2 Standard, Section 8.7.1, paragraphs 1-3: > > Threads > > In a thread-compliant implementation, an MPI process is a process that > may be multi-threaded. Each thread can issue MPI calls; however, > threads are not separately addressable: a rank in a send or receive > call identifies a process, not a thread. A message sent to a process > can be received by any thread in this process. > > Rationale. > > This model corresponds to the POSIX model of interprocess > communication: the fact that a process is multi-threaded, rather than > single-threaded, does not affect the external interface of this > process. MPI implementations where MPI `processes' are POSIX threads > inside a single POSIX process are not thread-compliant by this > definition (indeed, their ``processes'' are single-threaded). ( End of > rationale.) > > Advice to users. > > It is the user's responsibility to prevent races when threads within > the same application post conflicting communication calls. The user > can make sure that two threads in the same process will not issue > conflicting communication calls by using distinct communicators at > each thread. ( End of advice to users.) > ----------------------------------------------------------------------------- > > The basic question is simple. How do you define "conflicting communication > calls"? There are two possible interpretations: > > 1) Any MPI collective call on the same communicator. > or 2) The same MPI collective call on the same communicator. > > When I read the text of the standard, I tend to take interpretation #1, > but one MPI Forum member I talked to recalls the forum specifically taking > interpretation #2. Hence I want to get more feedback to ensure that I'm > interpreting this correctly. > > I think that choosing interpretation #1 would be very desirable because it > would permit some MPI collectives to be layered on top of other collectives > without introducing thread-safety problems. As a common example, the ROMIO > MPI I/O software is layered on MPI collectives. Interpretation #1 would > permit ROMIO to be layered and still not violate thread-safety. > > Consider scenario S1 below, which illustrates this thread-safety issue as > it affects the ROMIO MPI-2 I/O layered implementation. > > ROMIO function MPI_File_open() calls MPI_Allreduce() using the communicator > passed by the user into MPI_File_open. Suppose the user had another thread > executing an MPI_Allreduce() collective operation on the same communicator. > > If a race condition allows the threads to execute in different orders, > we would get an incorrect result: > > Scenario S1 > ----------- > > | Process 0 | Process 1 > | --------- | --------- > | | > Time | thread A calls MPI_Allreduce | thread C calls MPI_Allreduce > | | via MPI_File_open | > | | | > | | thread B calls MPI_Allreduce | thread D calls MPI_Allreduce > | | | via MPI_File_open > V | | > > > Under interpretation #1, you could say that the user needs to ensure > that all threads issuing any collective calls on the same communicator > must be properly ordered by the user. With this interpretation, scenario > S1 would be erroneous and the implementation wouldn't need to deal with > the matching up of the collective calls on multiple threads. > > Notice that under both interpretations, a scenario like S2 should > be erroneous because the same collective function is called by > the two threads. > > Scenario S2 > ----------- > > | Process 0 | Process 1 > | --------- | --------- > | | > Time | thread A calls MPI_Allreduce | thread C calls MPI_Allreduce > | | for MPI_MAX operation | for MPI_SUM operation > | | | > | | thread B calls MPI_Allreduce | thread D calls MPI_Allreduce > | | for MPI_SUM operation | for MPI_MAX operation > V | | > > > > Thanks for any opinions or recollections of forum discussions on this matter. > > > Karl Feind E-Mail: kaf@sgi.com > Silicon Graphics Phone: 612/683-5673 > 655F Lone Oak Drive Fax: 612/683-5276 > Eagan, MN 55121 > From mpi-comm-human@mcs.anl.gov Mon Nov 16 17:31:41 1998 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id RAA20226; Mon, 16 Nov 1998 17:31:41 -0500 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id QAA04886 for mpi-comm-out; Mon, 16 Nov 1998 16:11:30 -0600 Received: from smtp4.ny.us.ibm.COM (smtp4.ny.us.ibm.com [198.133.22.43]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id PAA03096 for ; Mon, 16 Nov 1998 15:33:25 -0600 From: snir@us.ibm.com Received: from northrelay01.pok.ibm.com (northrelay01.pok.ibm.com [9.117.200.21]) by smtp4.ny.us.ibm.COM (8.8.7/8.8.7) with ESMTP id QAA46590; Mon, 16 Nov 1998 16:19:52 -0500 Received: from us.ibm.com (d51mta03.pok.ibm.com [9.117.200.31]) by northrelay01.pok.ibm.com (8.8.7/NCO v1.2) with SMTP id QAA106110; Mon, 16 Nov 1998 16:32:48 -0500 Received: by us.ibm.com(Lotus SMTP MTA Internal build v4.6.2 (651.2 6-10-1998)) id 852566BE.007659B2 ; Mon, 16 Nov 1998 16:32:41 -0500 X-Lotus-FromDomain: IBMUS To: Karl Feind cc: mpi-comm@mcs.anl.gov Message-ID: <852566BE.0076591C.00@us.ibm.com> Date: Mon, 16 Nov 1998 16:22:59 -0500 Subject: Re: MPI-2 thread safety and collectives Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk The MPI2 standards specifies: "matching of collective calls on a communicator, window or file handle is done according to the order in which the call are issued on each process. If concurrent threads issue such calls on the same communicator, window or file handle, it is up to the user to ensure that the call are correctly ordered, using interthread communication. " The text speaks of calls done on the same object (communicator, file,, or window), not on calls done on distinct objects. A code of the form Process 0 Process 1 thread 0.0 thread 0.1 thread 1.0 thread 1.1 MPI_Barrier(comm1) MPI_Barrier(comm2) MPI_Barrier(comm1) MPI_Barrier(comm2) is perfectly legal, and should work. (Suppose that the threads do not execute concurrently: thread 0.0 and 1.1 start. At this point both threads are blocked and cannot continue. But "Blocking MPI calls will block the calling thread only". Thus, 0.1, and 1.0 can start executing. At this point, matching calls occurred at both processes and the collective calls must complete, according to the usual MPI progress rule.) Similarly a collective call on a window can be concurrent with a collective call with a communicator, and the code should work -- the argument is same as in my simpler example. If Romio uses the user provided communicator to synchronize file accesses, then the implementation is incorrect, since it may lead to illegal code, where concurrent threads issue collective calls on THE SAME communicator in the worng order. Besides, such an implementation violates a frequently reiterated rule of good MPI design: if one develops a collective library, then such a library should use private communicators for its internal communications, to avoid races. The recommended approach would be to cache this private communicator with the window object. Marc Snir Senior Manager, Scalable Parallel Systems IBM T J Watson Research Center, PO Box 218, Yorktown Heights 10598 Tel: 914 945 3204 (862 3204) Fax: 914 945 4425 (862 44245) URL: http://www.research.ibm.com/people/s/snir ...................... Consider scenario S1 below, which illustrates this thread-safety issue as it affects the ROMIO MPI-2 I/O layered implementation. ROMIO function MPI_File_open() calls MPI_Allreduce() using the communicator passed by the user into MPI_File_open. Suppose the user had another thread executing an MPI_Allreduce() collective operation on the same communicator. If a race condition allows the threads to execute in different orders, we would get an incorrect result: Scenario S1 ----------- | Process 0 | Process 1 | --------- | --------- | | Time | thread A calls MPI_Allreduce | thread C calls MPI_Allreduce | | via MPI_File_open | | | | | | thread B calls MPI_Allreduce | thread D calls MPI_Allreduce | | | via MPI_File_open V | | Under interpretation #1, you could say that the user needs to ensure that all threads issuing any collective calls on the same communicator must be properly ordered by the user. With this interpretation, scenario S1 would be erroneous and the implementation wouldn't need to deal with the matching up of the collective calls on multiple threads. Notice that under both interpretations, a scenario like S2 should be erroneous because the same collective function is called by the two threads. Scenario S2 ----------- | Process 0 | Process 1 | --------- | --------- | | Time | thread A calls MPI_Allreduce | thread C calls MPI_Allreduce | | for MPI_MAX operation | for MPI_SUM operation | | | | | thread B calls MPI_Allreduce | thread D calls MPI_Allreduce | | for MPI_SUM operation | for MPI_MAX operation V | | Thanks for any opinions or recollections of forum discussions on this matter. Karl Feind E-Mail: kaf@sgi.com Silicon Graphics Phone: 612/683-5673 655F Lone Oak Drive Fax: 612/683-5276 Eagan, MN 55121 From mpi-comm-human@mcs.anl.gov Mon Nov 16 17:34:14 1998 Return-Path: Received: from antares.mcs.anl.gov by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id RAA20287; Mon, 16 Nov 1998 17:34:13 -0500 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id QAA05442 for mpi-comm-out; Mon, 16 Nov 1998 16:23:08 -0600 Received: from smtp4.ny.us.ibm.COM (smtp4.ny.us.ibm.com [198.133.22.43]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id PAA03096 for ; Mon, 16 Nov 1998 15:33:25 -0600 From: snir@us.ibm.com Received: from northrelay01.pok.ibm.com (northrelay01.pok.ibm.com [9.117.200.21]) by smtp4.ny.us.ibm.COM (8.8.7/8.8.7) with ESMTP id QAA46590; Mon, 16 Nov 1998 16:19:52 -0500 Received: from us.ibm.com (d51mta03.pok.ibm.com [9.117.200.31]) by northrelay01.pok.ibm.com (8.8.7/NCO v1.2) with SMTP id QAA106110; Mon, 16 Nov 1998 16:32:48 -0500 Received: by us.ibm.com(Lotus SMTP MTA Internal build v4.6.2 (651.2 6-10-1998)) id 852566BE.007659B2 ; Mon, 16 Nov 1998 16:32:41 -0500 X-Lotus-FromDomain: IBMUS To: Karl Feind cc: mpi-comm@mcs.anl.gov Message-ID: <852566BE.0076591C.00@us.ibm.com> Date: Mon, 16 Nov 1998 16:22:59 -0500 Subject: Re: MPI-2 thread safety and collectives Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk The MPI2 standards specifies: "matching of collective calls on a communicator, window or file handle is done according to the order in which the call are issued on each process. If concurrent threads issue such calls on the same communicator, window or file handle, it is up to the user to ensure that the call are correctly ordered, using interthread communication. " The text speaks of calls done on the same object (communicator, file,, or window), not on calls done on distinct objects. A code of the form Process 0 Process 1 thread 0.0 thread 0.1 thread 1.0 thread 1.1 MPI_Barrier(comm1) MPI_Barrier(comm2) MPI_Barrier(comm1) MPI_Barrier(comm2) is perfectly legal, and should work. (Suppose that the threads do not execute concurrently: thread 0.0 and 1.1 start. At this point both threads are blocked and cannot continue. But "Blocking MPI calls will block the calling thread only". Thus, 0.1, and 1.0 can start executing. At this point, matching calls occurred at both processes and the collective calls must complete, according to the usual MPI progress rule.) Similarly a collective call on a window can be concurrent with a collective call with a communicator, and the code should work -- the argument is same as in my simpler example. If Romio uses the user provided communicator to synchronize file accesses, then the implementation is incorrect, since it may lead to illegal code, where concurrent threads issue collective calls on THE SAME communicator in the worng order. Besides, such an implementation violates a frequently reiterated rule of good MPI design: if one develops a collective library, then such a library should use private communicators for its internal communications, to avoid races. The recommended approach would be to cache this private communicator with the window object. Marc Snir Senior Manager, Scalable Parallel Systems IBM T J Watson Research Center, PO Box 218, Yorktown Heights 10598 Tel: 914 945 3204 (862 3204) Fax: 914 945 4425 (862 44245) URL: http://www.research.ibm.com/people/s/snir ...................... Consider scenario S1 below, which illustrates this thread-safety issue as it affects the ROMIO MPI-2 I/O layered implementation. ROMIO function MPI_File_open() calls MPI_Allreduce() using the communicator passed by the user into MPI_File_open. Suppose the user had another thread executing an MPI_Allreduce() collective operation on the same communicator. If a race condition allows the threads to execute in different orders, we would get an incorrect result: Scenario S1 ----------- | Process 0 | Process 1 | --------- | --------- | | Time | thread A calls MPI_Allreduce | thread C calls MPI_Allreduce | | via MPI_File_open | | | | | | thread B calls MPI_Allreduce | thread D calls MPI_Allreduce | | | via MPI_File_open V | | Under interpretation #1, you could say that the user needs to ensure that all threads issuing any collective calls on the same communicator must be properly ordered by the user. With this interpretation, scenario S1 would be erroneous and the implementation wouldn't need to deal with the matching up of the collective calls on multiple threads. Notice that under both interpretations, a scenario like S2 should be erroneous because the same collective function is called by the two threads. Scenario S2 ----------- | Process 0 | Process 1 | --------- | --------- | | Time | thread A calls MPI_Allreduce | thread C calls MPI_Allreduce | | for MPI_MAX operation | for MPI_SUM operation | | | | | thread B calls MPI_Allreduce | thread D calls MPI_Allreduce | | for MPI_SUM operation | for MPI_MAX operation V | | Thanks for any opinions or recollections of forum discussions on this matter. Karl Feind E-Mail: kaf@sgi.com Silicon Graphics Phone: 612/683-5673 655F Lone Oak Drive Fax: 612/683-5276 Eagan, MN 55121 From mpi-comm-human@mcs.anl.gov Sun Feb 28 14:21:23 1999 Return-Path: Received: from antares.mcs.anl.gov (mcs.anl.gov [140.221.9.6]) by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id OAA03687; Sun, 28 Feb 1999 14:21:22 -0500 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id NAA03183 for mpi-comm-out; Sun, 28 Feb 1999 13:14:12 -0600 Received: from miller.cs.uwm.edu (fethiye@miller.cs.uwm.edu [129.89.143.22]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id NAA03178 for ; Sun, 28 Feb 1999 13:14:06 -0600 Received: from localhost (fethiye@localhost) by miller.cs.uwm.edu (8.9.1a/8.9.1a) with SMTP id NAA24170 for ; Sun, 28 Feb 1999 13:14:04 -0600 (CST) Date: Sun, 28 Feb 1999 13:14:02 -0600 (CST) From: Fethiye Akbulut To: mpi-comm@mcs.anl.gov Subject: MPI_Abort Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk Hi all, I am so new with MPI and not so sure if this is the right forum to ask questions about the difficulties I am facing with MPI, but here comes one: I have several processors competing to finish a job. All I want to do is to kill the other processes when one is done. Yes, I know MPI_Abort would do that, and it really kills but it also generates all the error messages. So, is there another way of doing this? Or, is there a way not to show these error messages generated? Also, what about the error_code part of MPI_Abort? Is there any site which gives a listing of which integer correspond to which kind of `error' generated? Thanks, Fethiye From mpi-comm-human@mcs.anl.gov Sun Feb 28 16:10:26 1999 Return-Path: Received: from antares.mcs.anl.gov (mcs.anl.gov [140.221.9.6]) by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id QAA06018; Sun, 28 Feb 1999 16:10:25 -0500 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id PAA04707 for mpi-comm-out; Sun, 28 Feb 1999 15:07:09 -0600 Received: from george.lbl.gov (george.lbl.gov [131.243.2.12]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id PAA04702 for ; Sun, 28 Feb 1999 15:07:03 -0600 Received: from localhost (wcs@localhost) by george.lbl.gov (8.9.2/8.9.2) with SMTP id NAA14446; Sun, 28 Feb 1999 13:06:56 -0800 (PST) X-Authentication-Warning: george.lbl.gov: wcs owned process doing -bs Date: Sun, 28 Feb 1999 13:06:55 -0800 (PST) From: Bill Saphir X-Sender: wcs@george.lbl.gov To: Fethiye Akbulut cc: mpi-comm@mcs.anl.gov Subject: Re: MPI_Abort In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk On Sun, 28 Feb 1999, Fethiye Akbulut wrote: > Hi all, > > I am so new with MPI and not so sure if this is the right forum to ask > questions about the difficulties I am facing with MPI, but here comes one: > > I have several processors competing to finish a job. All I want to do is > to kill the other processes when one is done. Yes, I know MPI_Abort would > do that, and it really kills but it also generates all the error messages. > So, is there another way of doing this? Or, is there a way not to show > these error messages generated? No. The general rule in MPI is that operations are cooperative. That is, one process cannot do something to another without the explicit cooperation of the other process. This both improves performance and eliminates race conditions. The "MPI" way of shutting down is for all processes to call MPI_Finalize() and exit. In your case, the process that "wins" would need to send a message to the others to enable cooperative shutdown, and all processes would need to poll. Admittedly not pretty. As for not showing the error messages, the standard is silent. It would be possible for an implementation to decide not to show an error message, or to not show an error message for certain error codes, but this would be implementation dependent. > Also, what about the error_code part of MPI_Abort? Is there any site which > gives a listing of which integer correspond to which kind of `error' > generated? The code can be anything you want. I don't believe this code is necessarily related to error codes returned by MPI routines, as described in section 7.3 of the standard, "Error codes and classes", at http://www.mpi-forum.org/docs/mpi-11-html/node149.html#Node149. Bill From mpi-comm-human@mcs.anl.gov Mon Mar 1 07:19:41 1999 Return-Path: Received: from antares.mcs.anl.gov (mcs.anl.gov [140.221.9.6]) by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id HAA23487; Mon, 1 Mar 1999 07:19:39 -0500 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id FAA12353 for mpi-comm-out; Mon, 1 Mar 1999 05:34:17 -0600 Received: from marraco.udl.es (marraco.udl.es [193.144.8.14]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id FAA12244 for ; Mon, 1 Mar 1999 05:24:20 -0600 Received: from eup.udl.es (fermat.udl.net [10.50.54.28]) by marraco.udl.es (8.8.5/8.8.5) with SMTP id MAA17460 for ; Mon, 1 Mar 1999 12:19:14 -0100 (GMT) Received: from alumnes.eup.udl.es by eup.udl.es (SMI-8.6/SMI-SVR4) id MAA16038; Mon, 1 Mar 1999 12:22:22 +0100 Received: by alumnes.eup.udl.es (SMI-8.6/SMI-SVR4) id MAA25398; Mon, 1 Mar 1999 12:15:36 +0100 Date: Mon, 1 Mar 1999 12:15:36 +0100 From: d4373174@alumnes.eup.udl.es (Ferran Eloi Gutierrez Martos) Message-Id: <199903011115.MAA25398@alumnes.eup.udl.es> To: mpi-comm@mcs.anl.gov Subject: Having problems with LAM under LINUX Content-Type: X-sun-attachment Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk ---------- X-Sun-Data-Type: text X-Sun-Data-Description: text X-Sun-Data-Name: text X-Sun-Charset: us-ascii X-Sun-Content-Lines: 44 Hi all: My name is Ferran I am new with MPI LAM and I have not any problems with mpich, but with LAM I can't execute: lamboot. My problem is: It seems that when I install LAM 6.1 (under Linux RedHat 5.2) the PATH is not correctely set. The steps that I follow to install are: (untar the file into /home/arqui/lam61) tar xvf lam61-patch.tar First I apply the patches with : cat lam61-patch[0-9][0-9] | patch -p0 Next I create the symbolic link Config/config pointing config.linux.ix86 with: ln -s config.linux.ix86 config The I edit config and I set the variable HOME = /home/arqui/lam61 (and I save config) After I do: make >& LOG & And the when I try to run lamboot it falls, it seems that paths doesn't set. And hcc falls, and recon falls, and all falls. Well , I attach my LOG in this mail. Thank you. Ferran Eloi ---------- X-Sun-Data-Type: default X-Sun-Data-Description: default X-Sun-Data-Name: LOG X-Sun-Encoding-Info: uuencode X-Sun-Content-Lines: 2448 begin 600 LOG M2!@ M+VAO;64O87)Q=6DO;&%M-C$O;&EB+V]T8BUO8FHO87)G2!@+VAO;64O87)Q=6DO;&%M-C$O;&EB+V]T8BUO8FHO;7!I M)PHO8FEN+W)M("UF(')P:2YS:&TN;R!T8W N;&]W+F\@7-V+F\@6-R96%T92YO(&ME>69R964N;R!L M86UA='1R+F\@;&%M8G5F+F\@;&%M8VED+F\@;&%M8VQO8VMS+F\@;&%M8V]M M;2YO(&QA;61E9F5R7!E+F\@;&%M9F%U;'0N;R!L86UI;FET M+F\@;&%M:6YI=&5D+F\@;&%M;6ES8RYO(&QA;7!A8VLN;R!L86UP;W)TF4N;R!P M8V]N=')O;"YO('!OF5?9BYO(&=A=&AE&-L7V8N;R!G5]F+F\@=V%I='-O;65?9BYO('=T M:6-K7V8N;R!W=&EM95]F+F\@;&EB;7!I+F$*;6%K95LR73H@3&5A=FEN9R!D M:7)E8W1O2!@+VAO;64O87)Q=6DO M;&%M-C$O;&EB+V]T8BUO8FHO="<*"0EE=&,Z"FUA:V5;,UTZ($5N=&5R:6YG M(&1I2!@+VAO;64O87)Q=6DO;&%M-C$O;&EB+V]T8BUO8FHO="]E M=&,G"B]B:6XO"YO(&QD=&EM97(N M;R!L:6]A='1A8V@N;R!L<&%T=&%C:"YO(&QP9&5T86-H+F\@;6EC2!@+VAO;64O87)Q=6DO;&%M-C$O;&EB+V]T8BUO M8FHO="]F&ET+F\@"YO M(')F2!@ M+VAO;64O87)Q=6DO;&%M-C$O;&EB+V]T8BUO8FHO="]K2!@+VAO;64O87)Q=6DO;&%M M-C$O;&EB+V]T8BUO8FHO="]N2!@+VAO;64O M87)Q=6DO;&%M-C$O;&EB+V]T8BUO8FHO="]T;V]L"YO(%1224PN;R!L:6)T M2!@+VAO;64O87)Q=6DO;&%M-C$O=&]O;',M M;V)J)PH):&)O;W0Z"FUA:V5;,ETZ($5N=&5R:6YG(&1I2!@+VAO M;64O87)Q=6DO;&%M-C$O=&]O;',M;V)J+VAB;V]T)PHO8FEN+W)M("UF(&AB M;V]T+F\@<')O8U]S8VAE;6$N;R!H8F]O= IM86ME6S)=.B!,96%V:6YG(&1I M2!@+VAO;64O87)Q=6DO;&%M-C$O=&]O;',M;V)J+VAB;V]T)PH) M:&-C.@IM86ME6S)=.B!%;G1E&5C+F\@;&%M;F5T+F\@2!@+VAO;64O87)Q=6DO;&%M-C$O=&]O;',M;V)J+W)E8V]N M)PH)=&MI;&PZ"FUA:V5;,ETZ($5N=&5R:6YG(&1I2!@+VAO;64O M87)Q=6DO;&%M-C$O=&]O;',M;V)J+W1K:6QL)PHO8FEN+W)M("UF('1K:6QL M+F\@2!@+VAO;64O87)Q=6DO;&%M-C$O;W1B M+6]B:B]B9F-T;"<*+V)I;B]R;2 M9B!B9F-T;"YO(&)F8W1L"FUA:V5;,ETZ M($QE879I;F<@9&ER96-T;W)Y(& O:&]M92]A2!@+VAO;64O87)Q=6DO;&%M M-C$O;W1B+6]B:B]D;V]M)PHO8FEN+W)M("UF(&1O;VTN;R!D;V]M"FUA:V5; M,ETZ($QE879I;F<@9&ER96-T;W)Y(& O:&]M92]A2!@+VAO;64O M87)Q=6DO;&%M-C$O;W1B+6]B:B]F8W1L)PH)9G-T871E.@IM86ME6S)=.B!% M;G1E2!@+VAO;64O87)Q=6DO;&%M-C$O;W1B+6]B:B]F M2!@+VAO;64O87)Q=6DO;&%M-C$O;W1B+6]B:B]L86UG&5C+F\@;&%M2!@+VAO;64O87)Q=6DO;&%M-C$O;W1B+6]B:B]L;V%D9V\G"@EM<&EM2!@+VAO;64O87)Q=6DO;&%M M-C$O;W1B+6]B:B]M<&EM2!@+VAO;64O87)Q=6DO M;&%M-C$O;W1B+6]B:B]M<&ER=6XG"@EM<&ET87-K.@IM86ME6S)=.B!%;G1E M"YO(&9Q&ET+F\@ M;G)E8W8N;R!N2!@+VAO;64O M87)Q=6DO;&%M-C$O;W1B+6]B:B]T<&EN9R<*;6%K95LQ73H@3&5A=FEN9R!D M:7)E8W1O2!@+VAO;64O87)Q=6DO;&%M-C$O;6%N)PIS:&%R92]H.@IM86ME6S%= M.B!%;G1E2!@+VAO;64O87)Q=6DO;&%M-C$O2!@+VAO;64O87)Q=6DO;&%M M-C$O2!@+VAO;64O87)Q=6DO;&%M-C$O;&EB+V]T8BUO8FHG"@EA2!@+VAO;64O87)Q=6DO;&%M M-C$O;&EB+V]T8BUO8FHO="<*"0EE=&,Z"FUA:V5;,UTZ($5N=&5R:6YG(&1I M2!@+VAO;64O87)Q=6DO;&%M-C$O;&EB+V]T8BUO8FHO="]E=&,G M"F=E;F5R871I;F<@9&5P96YD96YC:65S"FUA:V5;,UTZ($QE879I;F<@9&ER M96-T;W)Y(& O:&]M92]A2!@+VAO;64O M87)Q=6DO;&%M-C$O;&EB+V]T8BUO8FHO="]T;V]L2!@+VAO;64O87)Q M=6DO;&%M-C$O=&]O;',M;V)J)PH):&)O;W0Z"FUA:V5;,ETZ($5N=&5R:6YG M(&1I2!@+VAO;64O87)Q=6DO;&%M-C$O=&]O;',M;V)J+VAB;V]T M)PIG96YE2!@+VAO;64O87)Q=6DO;&%M-C$O=&]O;',M;V)J+VAB;V]T)PH) M:&-C.@IM86ME6S)=.B!%;G1E2!@+VAO;64O87)Q=6DO;&%M-C$O=&]O;',M;V)J+W)E M8V]N)PIG96YE2!@+VAO;64O87)Q=6DO;&%M-C$O=&]O;',M;V)J+W)E8V]N M)PH)=&MI;&PZ"FUA:V5;,ETZ($5N=&5R:6YG(&1I2!@+VAO;64O M87)Q=6DO;&%M-C$O=&]O;',M;V)J+W1K:6QL)PIG96YE2!@+VAO;64O87)Q M=6DO;&%M-C$O=&]O;',M;V)J+W1K:6QL)PH)=VEP93H*;6%K95LR73H@16YT M97)I;F<@9&ER96-T;W)Y(& O:&]M92]A2!@+VAO;64O87)Q=6DO;&%M M-C$O=&]O;',M;V)J)PIO=&(M;V)J.@IM86ME6S%=.B!%;G1E2!@+VAO;64O87)Q=6DO;&%M-C$O;W1B+6]B M:B]B9F-T;"<*"6)F2!@+VAO;64O87)Q=6DO;&%M-C$O;W1B+6]B:B]B9G-T871E)PIG96YE2!@ M+VAO;64O87)Q=6DO;&%M-C$O;W1B+6]B:B]B9G-T871E)PH)9&]O;3H*;6%K M95LR73H@16YT97)I;F<@9&ER96-T;W)Y(& O:&]M92]A2!@+VAO M;64O87)Q=6DO;&%M-C$O;W1B+6]B:B]L86UC;&5A;B<*9V5N97)A=&EN9R!D M97!E;F1E;F-I97,*;6%K95LR73H@3&5A=FEN9R!D:7)E8W1O2!@+VAO;64O87)Q=6DO;&%M-C$O;W1B+6]B M:B]L;V%D9V\G"F=E;F5R871I;F<@9&5P96YD96YC:65S"FUA:V5;,ETZ($QE M879I;F<@9&ER96-T;W)Y(& O:&]M92]A2!@+VAO;64O87)Q=6DO;&%M-C$O;W1B+6]B:B]M M<&ER=6XG"@EM<&ET87-K.@IM86ME6S)=.B!%;G1E2!@+VAO;64O87)Q=6DO;&%M-C$O;W1B+6]B M:B]S>7,O;&%M9"<*9V5N97)A=&EN9R!D97!E;F1E;F-I97,*;6%K95LS73H@ M3&5A=FEN9R!D:7)E8W1O#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO+BXO#@V("U)+VAO M;64O87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO+BXO#@V("U)+VAO;64O87)Q=6DO;&%M M-C$O:" @+6,@+BXO+BXO+BXO#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO+BXO M2!@+VAO;64O87)Q=6DO;&%M-C$O;&EB+V]T8BUO8FHO87)G M#@V("U) M+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U0 M25]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO7-V+F,*9V-C("U/("U$3$E.55@@+41I>#@V("U)+VAO;64O M87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(] M." M8R N+B\N+B\N+B]S:&%R92]M<&DO#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)0 M25]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R M92]M<&DO86QL#@V("U)+VAO M;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]' M15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO86QL=&]A;&PN8PIG8V,@+4\@ M+41,24Y56" M1&EX.#8@+4DO:&]M92]A#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)0 M25]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R M92]M<&DO8V%R=&1I;6=E="YC"F=C8R M3R M1$Q)3E58("U$:7@X-B M22]H M;VUE+V%R<75I+VQA;38Q+V@@+4124$E?4TA-("U$4TA-7U-94U8@+41-4$E? M1T52/3@@+6,@+BXO+BXO+BXO#@V("U)+VAO;64O87)Q M=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M M8R N+B\N+B\N+B]S:&%R92]M<&DO8V%R=')A;FLN8PIG8V,@+4\@+41,24Y5 M6" M1&EX.#8@+4DO:&]M92]A#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE3 M5B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO8V-M<"YC"F=C M8R M3R M1$Q)3E58("U$:7@X-B M22]H;VUE+V%R<75I+VQA;38Q+V@@+412 M4$E?4TA-("U$4TA-7U-94U8@+41-4$E?1T52/3@@+6,@+BXO+BXO+BXOF4N8PIG8V,@+4\@+41,24Y56" M M1&EX.#8@+4DO:&]M92]A#@V("U)+VAO M;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]' M15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO9'5P9FXN8PIG8V,@+4\@+41, M24Y56" M1&EX.#8@+4DO:&]M92]A#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U? M4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO9W)A<&AD M:6US9V5T+F,*9V-C("U/("U$3$E.55@@+41I>#@V("U)+VAO;64O87)Q=6DO M;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N M+B\N+B\N+B]S:&%R92]M<&DO9W)A<&AG970N8PIG8V,@+4\@+41,24Y56" M M1&EX.#8@+4DO:&]M92]A#@V M("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO9W)A<&AN8G)C;W5N M="YC"F=C8R M3R M1$Q)3E58("U$:7@X-B M22]H;VUE+V%R<75I+VQA;38Q M+V@@+4124$E?4TA-("U$4TA-7U-94U8@+41-4$E?1T52/3@@+6,@+BXO+BXO M+BXO#@V("U)+VAO;64O87)Q M=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M M8R N+B\N+B\N+B]S:&%R92]M<&DO9W1R86YK#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)0 M25]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R M92]M<&DO:6-M97)G92YC"F=C8R M3R M1$Q)3E58("U$:7@X-B M22]H;VUE M+V%R<75I+VQA;38Q+V@@+4124$E?4TA-("U$4TA-7U-94U8@+41-4$E?1T52 M/3@@+6,@+BXO+BXO+BXO#@V("U)+VAO;64O87)Q M=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M M8R N+B\N+B\N+B]S:&%R92]M<&DO:6YF;V1U<"YC"F=C8R M3R M1$Q)3E58 M("U$:7@X-B M22]H;VUE+V%R<75I+VQA;38Q+V@@+4124$E?4TA-("U$4TA- M7U-94U8@+41-4$E?1T52/3@@+6,@+BXO+BXO+BXO#@V("U)+VAO;64O87)Q=6DO;&%M M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N M+B\N+B]S:&%R92]M<&DO:6YF;V=E="YC"F=C8R M3R M1$Q)3E58("U$:7@X M-B M22]H;VUE+V%R<75I+VQA;38Q+V@@+4124$E?4TA-("U$4TA-7U-94U8@ M+41-4$E?1T52/3@@+6,@+BXO+BXO+BXO#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]3 M2$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M M<&DO:6YI=&5D+F,*9V-C("U/("U$3$E.55@@+41I>#@V("U)+VAO;64O87)Q M=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M M8R N+B\N+B\N+B]S:&%R92]M<&DO:7!R;V)E+F,*9V-C("U/("U$3$E.55@@ M+41I>#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U? M4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO:7)E8W8N M8PIG8V,@+4\@+41,24Y56" M1&EX.#8@+4DO:&]M92]A6-R96%T92YC"F=C8R M3R M1$Q)3E58 M("U$:7@X-B M22]H;VUE+V%R<75I+VQA;38Q+V@@+4124$E?4TA-("U$4TA- M7U-94U8@+41-4$E?1T52/3@@+6,@+BXO+BXO+BXO69R M964N8PIG8V,@+4\@+41,24Y56" M1&EX.#8@+4DO:&]M92]A#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)0 M25]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R M92]M<&DO;&%M8VED+F,*9V-C("U/("U$3$E.55@@+41I>#@V("U)+VAO;64O M87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(] M." M8R N+B\N+B\N+B]S:&%R92]M<&DO;&%M8VQO8VMS+F,*9V-C("U/("U$ M3$E.55@@+41I>#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@ M+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO M;&%M8V]M;2YC"F=C8R M3R M1$Q)3E58("U$:7@X-B M22]H;VUE+V%R<75I M+VQA;38Q+V@@+4124$E?4TA-("U$4TA-7U-94U8@+41-4$E?1T52/3@@+6,@ M+BXO+BXO+BXO7!E+F,*9V-C("U/("U$3$E.55@@+41I>#@V("U)+VAO;64O87)Q=6DO;&%M M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N M+B\N+B]S:&%R92]M<&DO;&%M9F%U;'0N8PIG8V,@+4\@+41,24Y56" M1&EX M.#8@+4DO:&]M92]A#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S M:&%R92]M<&DO;&%M:6YI=&5D+F,*9V-C("U/("U$3$E.55@@+41I>#@V("U) M+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U0 M25]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO;&%M;6ES8RYC"F=C8R M M3R M1$Q)3E58("U$:7@X-B M22]H;VUE+V%R<75I+VQA;38Q+V@@+4124$E? M4TA-("U$4TA-7U-94U8@+41-4$E?1T52/3@@+6,@+BXO+BXO+BXO#@V("U)+VAO;64O87)Q=6DO M;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N M+B\N+B\N+B]S:&%R92]M<&DO;&%M#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U? M4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO;&%M#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)0 M25]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R M92]M<&DO;&%M=')A8V4N8PIG8V,@+4\@+41,24Y56" M1&EX.#8@+4DO:&]M M92]A#@V("U) M+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U0 M25]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO;7!I;%]S:6=N86PN8PIG M8V,@+4\@+41,24Y56" M1&EX.#8@+4DO:&]M92]A#@V("U) M+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U0 M25]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO;7!I;%]T#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@ M+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO M;F%M975N<'5B+F,*9V-C("U/("U$3$E.55@@+41I>#@V("U)+VAO;64O87)Q M=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M M8R N+B\N+B\N+B]S:&%R92]M<&DO;W!C#@V("U) M+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U0 M25]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO<&%C:W-I>F4N8PIG8V,@ M+4\@+41,24Y56" M1&EX.#8@+4DO:&]M92]A#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S M:&%R92]M<&DO#@V("U)+VAO M;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]' M15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO#@V("U)+VAO;64O87)Q M=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M M8R N+B\N+B\N+B]S:&%R92]M<&DO#@V("U) M+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U0 M25]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO#@V("U)+VAO;64O M87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(] M." M8R N+B\N+B\N+B]S:&%R92]M<&DO#@V("U)+VAO;64O87)Q=6DO;&%M M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N M+B\N+B]S:&%R92]M<&DO#@V("U) M+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U0 M25]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)0 M25]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R M92]M<&DO#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE3 M5B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO=&-O;6UI="YC M"F=C8R M3R M1$Q)3E58("U$:7@X-B M22]H;VUE+V%R<75I+VQA;38Q+V@@ M+4124$E?4TA-("U$4TA-7U-94U8@+41-4$E?1T52/3@@+6,@+BXO+BXO+BXO M#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@ M+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO M=&5S=&%L;"YC"F=C8R M3R M1$Q)3E58("U$:7@X-B M22]H;VUE+V%R<75I M+VQA;38Q+V@@+4124$E?4TA-("U$4TA-7U-94U8@+41-4$E?1T52/3@@+6,@ M+BXO+BXO+BXO#@V("U)+VAO;64O87)Q=6DO;&%M M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N M+B\N+B]S:&%R92]M<&DO=&5S='-O;64N8PIG8V,@+4\@+41,24Y56" M1&EX M.#8@+4DO:&]M92]A#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S M:&%R92]M<&DO=&9R964N8PIG8V,@+4\@+41,24Y56" M1&EX.#8@+4DO:&]M M92]A#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@ M+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO M=&AV96-T;W(N8PIG8V,@+4\@+41,24Y56" M1&EX.#8@+4DO:&]M92]A#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@ M+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO M='-T#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+413 M2$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO=V%I M=&%L;"YC"F=C8R M3R M1$Q)3E58("U$:7@X-B M22]H;VUE+V%R<75I+VQA M;38Q+V@@+4124$E?4TA-("U$4TA-7U-94U8@+41-4$E?1T52/3@@+6,@+BXO M+BXO+BXO#@V("U)+VAO M;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]' M15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO=W1I;64N8PIG8V,@+4\@+41, M24Y56" M1&EX.#8@+4DO:&]M92]A#@V("U)+VAO;64O87)Q=6DO M;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N M+B\N+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@ M+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO M9C#@V("U)+VAO M;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]' M15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO M;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N M+B\N+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@ M+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO M9C#@V M("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S M:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@ M+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO M9C#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)0 M25]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R M92]M<&DO9C#@V M("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO;&%M M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N M+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q M=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M M8R N+B\N+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]3 M2$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M M<&DO9C#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)0 M25]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R M92]M<&DO9C#@V("U) M+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U0 M25]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO;&%M M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N M+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q M=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M M8R N+B\N+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S M:&%R92]M<&DO9C#@V M("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO;&%M-C$O M:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N M+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U? M4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O M87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(] M." M8R N+B\N+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O M87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(] M." M8R N+B\N+B\N+B]S:&%R92]M<&DO9C#@V("U) M+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U0 M25]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+413 M2$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO9C&-L7V8N8PIG8V,@+4\@+41,24Y56" M1&EX.#8@+4DO:&]M92]A#@V("U)+VAO;64O87)Q M=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M M8R N+B\N+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@ M+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO M9C#@V("U)+VAO;64O87)Q=6DO M;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N M+B\N+B\N+B]S:&%R92]M<&DO9C&-L7V8N8PIG8V,@+4\@+41,24Y56" M1&EX M.#8@+4DO:&]M92]AF5?9BYC"F=C8R M3R M1$Q)3E58("U$ M:7@X-B M22]H;VUE+V%R<75I+VQA;38Q+V@@+4124$E?4TA-("U$4TA-7U-9 M4U8@+41-4$E?1T52/3@@+6,@+BXO+BXO+BXO#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S M:&%R92]M<&DO9C#@V M("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+413 M2$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@ M+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO M9C#@V("U)+VAO;64O M87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(] M." M8R N+B\N+B\N+B]S:&%R92]M<&DO9C#@V M("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+413 M2$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO9C#@V M("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q M=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M M8R N+B\N+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@ M+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO M9C#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]3 M2$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M M<&DO9C#@V("U)+VAO;64O87)Q=6DO;&%M M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N M+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO M;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N M+B\N+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+413 M2$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO;&%M M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N M+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO M;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N M+B\N+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S M:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+413 M2$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]3 M2$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M M<&DO9C#@V("U) M+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1$U0 M25]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO;&%M-C$O M:" M1%)025]32$T@+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N M+B]S:&%R92]M<&DO9C#@V M("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U? M4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO9C#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@ M+4132$U?4UE35B M1$U025]'15(]." M8R N+B\N+B\N+B]S:&%R92]M<&DO M9C7-V&-L+F\@ M9V9R964N;R!G:6YC;"YO(&=I;G1E&-L+F\@9W)I;F-L+F\@9W-I>F4N M;R!G=')A;FMS+F\@9W5N:6]N+F\@:6)S96YD+F\@:6-C2YO('=A:71S;VUE+F\@=W1I8VLN;R!W M=&EM92YO($U022YO(&%B;W)T7V8N;R!A8V-E<'1?9BYO(&%D9')EF5?9BYO(&-S:7IE7V8N;R!C69R965?9BYO(&UP:6Q?:61?9BYO M(&UP:6Q?%]F M+F\@=&AV96-T;W)?9BYO('1I;F1E>%]F+F\@=&QB7V8N;R!T;W!O=&5S=%]F M+F\@='-I>F5?9BYO('1S=')U8W1?9BYO('1U8E]F+F\@='9E8W1OF4N;PIA("T@ M9W1R86YK7,N;PIA("T@:6YF;V=E=&YT:"YO"F$@+2!I M;F9O9V5T=FQE;BYO"F$@+2!I;F9O6-R96%T92YO"F$@+2!K M97EF2YO"F$@+2!T97-T8V%N8V5L+F\* M82 M('1E'1E;G0N;PIA("T@=&9R964N;PIA("T@ M=&AI;F1E>"YO"F$@+2!T:'9E8W1O6-R96%T M95]F+F\*82 M(&ME>69R965?9BYO"F$@+2!M<&EL7VED7V8N;PIA("T@;7!I M;%]S:6=N86Q?9BYO"F$@+2!M<&EL7W-P87=N7V8N;PIA("T@;7!I;%]TF5?9BYO"F$@+2!P8V]N=')O;%]F+F\* M82 M('!O%]F+F\*82 M('1H=F5C M=&]R7V8N;PIA("T@=&EN9&5X7V8N;PIA("T@=&QB7V8N;PIA("T@=&]P;W1E M2!@+VAO;64O87)Q=6DO;&%M M-C$O;&EB+V]T8BUO8FHO;7!I)PH)=#H*;6%K95LR73H@16YT97)I;F<@9&ER M96-T;W)Y(& O:&]M92]A#@V M("U)+VAO;64O87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO+BXO+BXO#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO M+BXO;W1B+W0O971C+V9A:6PN8PIG8V,@+4\@+41,24Y56" M1&EX.#8@+4DO M:&]M92]A#@V("U)+VAO;64O87)Q=6DO;&%M M-C$O:" @+6,@+BXO+BXO+BXO+BXO#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO+BXO;W1B+W0O M971C+VEE964N8PIG8V,@+4\@+41,24Y56" M1&EX.#8@+4DO:&]M92]A7,O:V5R;F5L+VMI<')O8RYC"F=C8R M3R M M1$Q)3E58("U$:7@X-B M22]H;VUE+V%R<75I+VQA;38Q+V@@("UC("XN+RXN M+RXN+RXN+W-H87)E+V5T8R]L86UU;FEX+F,*9V-C("U/("U$3$E.55@@+41I M>#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO+BXO;W1B+W0O M971C+VQD=&EM97(N8PIG8V,@+4\@+41,24Y56" M1&EX.#8@+4DO:&]M92]A M#@V("U)+VAO;64O87)Q=6DO;&%M M-C$O:" @+6,@+BXO+BXO+BXO;W1B+W1R:6QL:75M+VQP871T86-H+F,*9V-C M("U/("U$3$E.55@@+41I>#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" @+6,@ M+BXO+BXO+BXO+BXO7!E+F\@8V]M8FEN M92YO(&9A:6PN;R!F972!@+VAO;64O87)Q M=6DO;&%M-C$O;&EB+V]T8BUO8FHO="]F&ET+F,*9V-C("U/("U$3$E.55@@+41I>#@V("U) M+VAO;64O87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO+BXO+BXO"YC"F=C8R M3R M1$Q)3E58("U$:7@X-B M22]H;VUE+V%R<75I M+VQA;38Q+V@@("UC("XN+RXN+RXN+RXN+W-H87)E+V9R97$O#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO+BXO+BXO"YC"F=C8R M3R M1$Q)3E58("U$:7@X-B M22]H;VUE+V%R<75I M+VQA;38Q+V@@("UC("XN+RXN+RXN+RXN+W-H87)E+V9R97$O#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" @ M+6,@+BXO+BXO+BXO+BXO&ET+F\@"YO(')F#@V("U) M+VAO;64O87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO+BXO;W1B+W0O:W)E<2]C M;&EE;G1I;RYC"F=C8R M3R M1$Q)3E58("U$:7@X-B M22]H;VUE+V%R<75I M+VQA;38Q+V@@("UC("XN+RXN+RXN+V]T8B]T+VMR97$O8V]U=&5R+F,*9V-C M("U/("U$3$E.55@@+41I>#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" @+6,@ M+BXO+BXO+BXO;W1B+W0O:W)E<2]K8W)E871E+F,*9V-C("U/("U$3$E.55@@ M+41I>#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO+BXO;W1B M+W0O:W)E<2]K871T86-H+F,*9V-C("U/("U$3$E.55@@+41I>#@V("U)+VAO M;64O87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO+BXO;W1B+W0O:W)E<2]K9&5T M86-H+F,*9V-C("U/("U$3$E.55@@+41I>#@V("U)+VAO;64O87)Q=6DO;&%M M-C$O:" @+6,@+BXO+BXO+BXO+BXO#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO+BXO+BXO M#@V("U)+VAO;64O87)Q=6DO;&%M M-C$O:" @+6,@+BXO+BXO+BXO+BXO#@V("U)+VAO M;64O87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO+BXO+BXO#@V("U)+VAO;64O87)Q=6DO M;&%M-C$O:" @+6,@+BXO+BXO+BXO+BXO#@V("U)+VAO;64O87)Q=6DO;&%M-C$O M:" @+6,@+BXO+BXO+BXO+BXO#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO+BXO M+BXO#@V("U)+VAO;64O M87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO+BXO+BXO#@V("U)+VAO;64O M87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO+BXO+BXO#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO+BXO M+BXO#@V("U)+VAO;64O87)Q M=6DO;&%M-C$O:" @+6,@+BXO+BXO+BXO+BXO#@V("U)+VAO M;64O87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO+BXO+BXO2!@+VAO;64O87)Q M=6DO;&%M-C$O;&EB+V]T8BUO8FHO="]T;V]L#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO+BXO+BXO M#@V("U) M+VAO;64O87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO+BXO+BXO#@V("U)+VAO;64O87)Q M=6DO;&%M-C$O:" @+6,@+BXO+BXO+BXO+BXO#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO+BXO M+BXO"YO"F$@+2!F2!@+VAO;64O87)Q=6DO;&%M-C$O M;&EB+V]T8BUO8FHO="<*"71R:6QL:75M.@IM86ME6S)=.B!%;G1E#@V("U)+VAO;64O87)Q M=6DO;&%M-C$O:" M22XN+RXN+RXN+W-H87)E+V)O;W0@("UC("XN+RXN+RXN M+W-H87)E+W1R:6QL:75M+W)B9G!A#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M22XN+RXN+RXN+W-H87)E+V)O M;W0@("UC("XN+RXN+RXN+W-H87)E+W1R:6QL:75M+W)B9FQO;VLN8PIG8V,@ M+4\@+41,24Y56" M1&EX.#8@+4DO:&]M92]A#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M22XN M+RXN+RXN+W-H87)E+V)O;W0@("UC("XN+RXN+RXN+W-H87)E+W1R:6QL:75M M+W)B9G=I<&4N8PIG8V,@+4\@+41,24Y56" M1&EX.#8@+4DO:&]M92]A#@V("U)+VAO;64O M87)Q=6DO;&%M-C$O:" M22XN+RXN+RXN+W-H87)E+V)O;W0@("UC("XN+RXN M+RXN+W-H87)E+W1R:6QL:75M+W)G970N8PIG8V,@+4\@+41,24Y56" M1&EX M.#8@+4DO:&]M92]A#@V("U) M+VAO;64O87)Q=6DO;&%M-C$O:" M22XN+RXN+RXN+W-H87)E+V)O;W0@("UC M("XN+RXN+RXN+W-H87)E+W1R:6QL:75M+W)P9V\N8PIG8V,@+4\@+41,24Y5 M6" M1&EX.#8@+4DO:&]M92]A#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M22XN+RXN+RXN+W-H87)E M+V)O;W0@("UC("XN+RXN+RXN+W-H87)E+W1R:6QL:75M+W)P=V%I="YC"F=C M8R M3R M1$Q)3E58("U$:7@X-B M22]H;VUE+V%R<75I+VQA;38Q+V@@+4DN M+B\N+B\N+B]S:&%R92]B;V]T(" M8R N+B\N+B\N+B]S:&%R92]R#@V M("U)+VAO;64O87)Q=6DO;&%M-C$O:" M22XN+RXN+RXN+W-H87)E+V)O;W0@ M("UC("XN+RXN+RXN+W-H87)E+W1R:6QL:75M+W)T#@V("U)+VAO;64O87)Q=6DO;&%M M-C$O:" M22XN+RXN+RXN+W-H87)E+V)O;W0@("UC("XN+RXN+RXN+W-H87)E M+W1R:6QL:75M+W)T"YO(%1224PN;PIA("T@ M87!P7VUG;70N;PIA("T@87-C7W!A#@V("U) M+VAO;64O87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO+BXO#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" @ M+6,@+BXO+BXO+BXO#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO M+BXO#@V("U)+VAO;64O87)Q=6DO;&%M-C$O M:" @+6,@+BXO+BXO+BXO#@V("U)+VAO;64O M87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO+BXO2!@+VAO;64O87)Q=6DO;&%M-C$O;&EB+V]T8BUO8FHG"G1O;VQS+6]B:CH* M;6%K95LQ73H@16YT97)I;F<@9&ER96-T;W)Y(& O:&]M92]A2!@+VAO;64O87)Q=6DO;&%M-C$O M=&]O;',M;V)J+VAC8R<*"6AF-S2!@+VAO;64O87)Q=6DO;&%M-C$O=&]O;',M;V)J+VAF-S#@V("U)+VAO;64O87)Q M=6DO;&%M-C$O:" M22XN+RXN+W-H87)E+V)O;W0@+41$1490/5PB+VAO;64O M87)Q=6DO;&%M-C%<(B M1$1%1E1&-S<]7")G-S=<(B M1%-94TQ)0E,])R(B M)R M8R N+B\N+B]S:&%R92]B;V]T+W!R;V-?2!@+VAO;64O87)Q=6DO;&%M-C$O=&]O;',M M;V)J+VQA;6)O;W0G"F=C8R M3R M1$Q)3E58("U$:7@X-B M22XN+RXN+W-H M87)E+V)O;W0@+4DO:&]M92]A2!@+VAO;64O87)Q=6DO;&%M-C$O=&]O;',M;V)J M+W)E8V]N)PIG8V,@+4\@+41,24Y56" M1&EX.#8@+4DO:&]M92]A2!@+VAO;64O87)Q M=6DO;&%M-C$O=&]O;',M;V)J+W1K:6QL)PH)=VEP93H*;6%K95LR73H@16YT M97)I;F<@9&ER96-T;W)Y(& O:&]M92]A#@V("U)+VAO;64O87)Q=6DO;&%M M-C$O:" M22XN+RXN+W-H87)E+V)O;W0@+41$14900DA/4U0]7"(O:&]M92]A M#@V M("U)+VAO;64O87)Q=6DO;&%M-C$O:" M22XN+RXN+W-H87)E+V)O;W0@+41$ M14900DA/4U0]7"(O:&]M92]A&5C+F\@;&%M;F5T+F\@=VEP M92YO("U,+VAO;64O87)Q=6DO;&%M-C$O;&EB("UL87)G2!@+VAO;64O87)Q=6DO;&%M-C$O=&]O;',M;V)J M+W=I<&4G"FUA:V5;,5TZ($QE879I;F<@9&ER96-T;W)Y(& O:&]M92]A2!@+VAO;64O87)Q=6DO;&%M-C$O;W1B+6]B:B<*"6)F8W1L M.@IM86ME6S)=.B!%;G1E#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO;W1B M+V1O;VTO9&]O;2YC"F=C8R @+6\@9&]O;2!D;V]M+F\@+4PO:&]M92]A2!@+VAO;64O87)Q M=6DO;&%M-C$O;W1B+6]B:B]F8W1L)PIG8V,@+4\@+41,24Y56" M1&EX.#8@ M+4DO:&]M92]A#@V("U)+VAO;64O87)Q=6DO;&%M M-C$O:" @+6,@+BXO+BXO;W1B+V9S=&%T92]F2!@+VAO;64O87)Q=6DO;&%M-C$O M;W1B+6]B:B]L86UC;&5A;B<*9V-C("U/("U$3$E.55@@+41I>#@V("U)+VAO M;64O87)Q=6DO;&%M-C$O:" M1%)025]32$T@+4132$U?4UE35B M1%)-/22!@+VAO;64O87)Q=6DO;&%M-C$O;W1B+6]B:B]L86UC;&5A;B<*"6QA;6=R M;W2!@+VAO;64O87)Q=6DO M;&%M-C$O;W1B+6]B:B]L86UG#@V("U)+VAO;64O87)Q=6DO;&%M M-C$O:" M22XN+RXN+W-H87)E+V)O;W0@+4124T@]7"(O=7-R+V)I;B]R&5C+F,*9V-C("U/("U$3$E. M55@@+41I>#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M22XN+RXN+W-H87)E M+V)O;W0@+4124T@]7"(O=7-R+V)I;B]R#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M22XN+RXN+W-H87)E+V)O M;W0@+4124T@]7"(O=7-R+V)I;B]R&5C+F,*9V-C("U/("U$3$E.55@@+41I>#@V("U)+VAO;64O87)Q M=6DO;&%M-C$O:" M22XN+RXN+W-H87)E+V)O;W0@+4124T@]7"(O=7-R+V)I M;B]R&5C+F\@;&%M2!@+VAO M;64O87)Q=6DO;&%M-C$O;W1B+6]B:B]L;V%D9V\G"F=C8R M3R M1$Q)3E58 M("U$:7@X-B M22]H;VUE+V%R<75I+VQA;38Q+V@@+4DN+B\N+B]S:&%R92]B M;V]T(" M8R N+B\N+B]O=&(O;&]A9&=O+VQO861G;RYC"F=C8R @+6\@;&]A M9&=O(&QO861G;RYO("U,+VAO;64O87)Q=6DO;&%M-C$O;&EB("UL=')I;&QI M=6T@+6QA2!@+VAO M;64O87)Q=6DO;&%M-C$O;W1B+6]B:B]L;V%D9V\G"@EM<&EM2!@+VAO;64O87)Q=6DO;&%M-C$O;W1B M+6]B:B]M<&EM#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" @+6,@ M+BXO+BXO#@V("U)+VAO;64O M87)Q=6DO;&%M-C$O:" @+6,@+BXO+BXO7!E M+F,*9V-C("U/("U$3$E.55@@+41I>#@V("U)+VAO;64O87)Q=6DO;&%M-C$O M:" @+6,@+BXO+BXO2!@+VAO;64O87)Q=6DO;&%M-C$O;W1B+6]B M:B]S=&%T92<*"7-Y7,O;&%M9"]C M;W5T97(N8PIG8V,@+4\@+41,24Y56" M1&EX.#8@+4DO:&]M92]A#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M22XN M+RXN+RXN+V]T8B]S>7,O9FEL960@+4143U]$3$]?04-+/34P,# P," M1%1/ M7T1,3U])1$Q%/38P("U$5$]?1$Q/7T535$E-051%/3,P("U$1$]-05A215-% M3D0],3 P("U$4E!)7U-(32 M1%-(35]365-6("U$4DT])R(O8FEN+W)M("UF M(B<@+6,@+BXO+BXO+BXO;W1B+W-Y#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M22XN+RXN+RXN+V]T8B]S M>7,O9FEL960@+4143U]$3$]?04-+/34P,# P," M1%1/7T1,3U])1$Q%/38P M("U$5$]?1$Q/7T535$E-051%/3,P("U$1$]-05A215-%3D0],3 P("U$4E!) M7U-(32 M1%-(35]365-6("U$4DT])R(O8FEN+W)M("UF(B<@+6,@+BXO+BXO M+BXO;W1B+W-Y7,O9&QI7VEN970O9&E?8F]O=&%G96YT+F,*9V-C("U/("U$3$E.55@@ M+41I>#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M22XN+RXN+RXN+V]T8B]S M>7,O9FEL960@+4143U]$3$]?04-+/34P,# P," M1%1/7T1,3U])1$Q%/38P M("U$5$]?1$Q/7T535$E-051%/3,P("U$1$]-05A215-%3D0],3 P("U$4E!) M7U-(32 M1%-(35]365-6("U$4DT])R(O8FEN+W)M("UF(B<@+6,@+BXO+BXO M+BXO;W1B+W-Y#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M22XN+RXN+RXN+V]T M8B]S>7,O9FEL960@+4143U]$3$]?04-+/34P,# P," M1%1/7T1,3U])1$Q% M/38P("U$5$]?1$Q/7T535$E-051%/3,P("U$1$]-05A215-%3D0],3 P("U$ M4E!)7U-(32 M1%-(35]365-6("U$4DT])R(O8FEN+W)M("UF(B<@+6,@+BXO M+BXO+BXO;W1B+W-Y#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M22XN+RXN+RXN+V]T M8B]S>7,O9FEL960@+4143U]$3$]?04-+/34P,# P," M1%1/7T1,3U])1$Q% M/38P("U$5$]?1$Q/7T535$E-051%/3,P("U$1$]-05A215-%3D0],3 P("U$ M4E!)7U-(32 M1%-(35]365-6("U$4DT])R(O8FEN+W)M("UF(B<@+6,@+BXO M+BXO+BXO;W1B+W-Y7,O9&QO7VEN970O9&QO7VEN970N8PIG8V,@+4\@+41, M24Y56" M1&EX.#8@+4DO:&]M92]A#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M22XN+RXN+RXN+V]T8B]S>7,O M9FEL960@+4143U]$3$]?04-+/34P,# P," M1%1/7T1,3U])1$Q%/38P("U$ M5$]?1$Q/7T535$E-051%/3,P("U$1$]-05A215-%3D0],3 P("U$4E!)7U-( M32 M1%-(35]365-6("U$4DT])R(O8FEN+W)M("UF(B<@+6,@+BXO+BXO+BXO M;W1B+W-Y7,O;&]A9&0O;&]A9&0N8PIG8V,@+4\@+41,24Y56" M1&EX.#8@+4DO:&]M M92]A7,O:V5N>6%D M+W!Q9&5T86-H+F,*9V-C("U/("U$3$E.55@@+41I>#@V("U)+VAO;64O87)Q M=6DO;&%M-C$O:" M22XN+RXN+RXN+V]T8B]S>7,O9FEL960@+4143U]$3$]? M04-+/34P,# P," M1%1/7T1,3U])1$Q%/38P("U$5$]?1$Q/7T535$E-051% M/3,P("U$1$]-05A215-%3D0],3 P("U$4E!)7U-(32 M1%-(35]365-6("U$ M4DT])R(O8FEN+W)M("UF(B<@+6,@+BXO+BXO+BXO;W1B+W-Y#@V("U)+VAO;64O87)Q=6DO M;&%M-C$O:" M22XN+RXN+RXN+V]T8B]S>7,O9FEL960@+4143U]$3$]?04-+ M/34P,# P," M1%1/7T1,3U])1$Q%/38P("U$5$]?1$Q/7T535$E-051%/3,P M("U$1$]-05A215-%3D0],3 P("U$4E!)7U-(32 M1%-(35]365-6("U$4DT] M)R(O8FEN+W)M("UF(B<@+6,@+BXO+BXO+BXO;W1B+W-Y#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M M22XN+RXN+RXN+V]T8B]S>7,O9FEL960@+4143U]$3$]?04-+/34P,# P," M M1%1/7T1,3U])1$Q%/38P("U$5$]?1$Q/7T535$E-051%/3,P("U$1$]-05A2 M15-%3D0],3 P("U$4E!)7U-(32 M1%-(35]365-6("U$4DT])R(O8FEN+W)M M("UF(B<@+6,@+BXO+BXO+BXO;W1B+W-Y7,O9FEL960O9F9A8V4N8PIG8V,@+4\@+41, M24Y56" M1&EX.#8@+4DO:&]M92]A#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M22XN+RXN+RXN+V]T8B]S M>7,O9FEL960@+4143U]$3$]?04-+/34P,# P," M1%1/7T1,3U])1$Q%/38P M("U$5$]?1$Q/7T535$E-051%/3,P("U$1$]-05A215-%3D0],3 P("U$4E!) M7U-(32 M1%-(35]365-6("U$4DT])R(O8FEN+W)M("UF(B<@+6,@+BXO+BXO M+BXO;W1B+W-Y#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M22XN+RXN+RXN+V]T8B]S>7,O M9FEL960@+4143U]$3$]?04-+/34P,# P," M1%1/7T1,3U])1$Q%/38P("U$ M5$]?1$Q/7T535$E-051%/3,P("U$1$]-05A215-%3D0],3 P("U$4E!)7U-( M32 M1%-(35]365-6("U$4DT])R(O8FEN+W)M("UF(B<@+6,@+BXO+BXO+BXO M;W1B+W-Y7,O9FEL960O9G%L7,O M9FEL960O9G%O<&5N+F,*9V-C("U/("U$3$E.55@@+41I>#@V("U)+VAO;64O M87)Q=6DO;&%M-C$O:" M22XN+RXN+RXN+V]T8B]S>7,O9FEL960@+4143U]$ M3$]?04-+/34P,# P," M1%1/7T1,3U])1$Q%/38P("U$5$]?1$Q/7T535$E- M051%/3,P("U$1$]-05A215-%3D0],3 P("U$4E!)7U-(32 M1%-(35]365-6 M("U$4DT])R(O8FEN+W)M("UF(B<@+6,@+BXO+BXO+BXO;W1B+W-Y"YC"F=C8R M3R M1$Q)3E58("U$:7@X-B M22]H;VUE+V%R<75I M+VQA;38Q+V@@+4DN+B\N+B\N+B]O=&(O7,O9FEL960O9G%S M=&%T92YC"F=C8R M3R M1$Q)3E58("U$:7@X-B M22]H;VUE+V%R<75I+VQA M;38Q+V@@+4DN+B\N+B\N+B]O=&(O7,O9FEL960O9G%W7,O9FEL960O9G%O<&5N9F0N M8PIG8V,@+4\@+41,24Y56" M1&EX.#8@+4DO:&]M92]A#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M22XN+RXN+RXN M+V]T8B]S>7,O9FEL960@+4143U]$3$]?04-+/34P,# P," M1%1/7T1,3U]) M1$Q%/38P("U$5$]?1$Q/7T535$E-051%/3,P("U$1$]-05A215-%3D0],3 P M("U$4E!)7U-(32 M1%-(35]365-6("U$4DT])R(O8FEN+W)M("UF(B<@+6,@ M+BXO+BXO+BXO;W1B+W-Y&ET+F,*9V-C("U/("U$3$E.55@@ M+41I>#@V("U)+VAO;64O87)Q=6DO;&%M-C$O:" M22XN+RXN+RXN+V]T8B]S M>7,O9FEL960@+4143U]$3$]?04-+/34P,# P," M1%1/7T1,3U])1$Q%/38P M("U$5$]?1$Q/7T535$E-051%/3,P("U$1$]-05A215-%3D0],3 P("U$4E!) M7U-(32 M1%-(35]365-6("U$4DT])R(O8FEN+W)M("UF(B<@+6,@+BXO+BXO M+BXO;W1B+W-Y#@V M("U)+VAO;64O87)Q=6DO;&%M-C$O:" M22XN+RXN+RXN+V]T8B]S>7,O9FEL M960@+4143U]$3$]?04-+/34P,# P," M1%1/7T1,3U])1$Q%/38P("U$5$]? M1$Q/7T535$E-051%/3,P("U$1$]-05A215-%3D0],3 P("U$4E!)7U-(32 M M1%-(35]365-6("U$4DT])R(O8FEN+W)M("UF(B<@+6,@+BXO+BXO+BXO;W1B M+W-Y#@V("U)+VAO M;64O87)Q=6DO;&%M-C$O:" M22XN+RXN+RXN+V]T8B]S>7,O9FEL960@+414 M3U]$3$]?04-+/34P,# P," M1%1/7T1,3U])1$Q%/38P("U$5$]?1$Q/7T53 M5$E-051%/3,P("U$1$]-05A215-%3D0],3 P("U$4E!)7U-(32 M1%-(35]3 M65-6("U$4DT])R(O8FEN+W)M("UF(B<@+6,@+BXO+BXO+BXO;W1B+W-Y7,O;&%M9"]T7,O:6]D+VEO9"YC"F=C8R M3R M M1$Q)3E58("U$:7@X-B M22]H;VUE+V%R<75I+VQA;38Q+V@@+4DN+B\N+B\N M+B]O=&(O7,O;&%M9"]I;V0N:6YT+F,*9V-C(" M;R!L86UD M(&-O=71E2!@+VAO;64O87)Q=6DO;&%M-C$O M;W1B+6]B:B]S>7,O;&%M9"<*;6%K95LR73H@3&5A=FEN9R!D:7)E8W1O2!@+VAO;64O87)Q=6DO M;&%M-C$O2!@+VAO;64O87)Q .=6DO;&%M-C$O;6%N)PI= end From mpi-comm-human@mcs.anl.gov Tue Mar 2 21:07:28 1999 Return-Path: Received: from antares.mcs.anl.gov (mcs.anl.gov [140.221.9.6]) by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id VAA28714; Tue, 2 Mar 1999 21:07:28 -0500 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id UAA14235 for mpi-comm-out; Tue, 2 Mar 1999 20:04:55 -0600 Received: from miller.cs.uwm.edu (fethiye@miller.cs.uwm.edu [129.89.143.22]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id UAA14229 for ; Tue, 2 Mar 1999 20:04:49 -0600 Received: from localhost (fethiye@localhost) by miller.cs.uwm.edu (8.9.1a/8.9.1a) with SMTP id UAA13982; Tue, 2 Mar 1999 20:04:45 -0600 (CST) Date: Tue, 2 Mar 1999 20:04:45 -0600 (CST) From: Fethiye Akbulut To: Bill Saphir cc: mpi-comm@mcs.anl.gov Subject: MPI_Iprobe? In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk Hi all, OK, since MPI_Abort does put the errors to stdout and it is not what we want I am trying another suggested way: using MPI_Iprobe. Here, I still have questions... In Dr.Pacheco's Book, it is written that MPI_Iprobe function searches for an incoming message matching three of the given parameters. And when it does match, then the flag is turned and we should use MPI_Recv to receive the message sent. So far, it is great. But, in my situation, since I want to kill the other processes when I am done, I don't want to wait for them to come to the same stage as I am. So, where should I put this MPI_Iprobe and MPI_Recv funstions? Do they have to follow each other? Thank you in advance, Fethiye From mpi-comm-human@mcs.anl.gov Thu Mar 4 18:57:15 1999 Return-Path: Received: from antares.mcs.anl.gov (mcs.anl.gov [140.221.9.6]) by netlib2.cs.utk.edu with ESMTP (cf v2.9t-netlib) id SAA13893; Thu, 4 Mar 1999 18:57:14 -0500 Received: (from listserv@localhost) by antares.mcs.anl.gov (8.6.10/8.6.10) id RAA20649 for mpi-comm-out; Thu, 4 Mar 1999 17:53:50 -0600 Received: from MPI-Softtech.Com (mpi.MPI-SoftTech.Com [208.152.187.97]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP id UAA15052 for ; Tue, 2 Mar 1999 20:49:34 -0600 Received: from localhost (tony@localhost); by MPI-Softtech.Com (8.9.1/8.9.1/MPI-Softtech/evision: 1.3 $) with ESMTP; id UAA23364; Tue, 2 Mar 1999 20:48:40 -0600 (CST) Date: Tue, 2 Mar 1999 20:48:40 -0600 (CST) From: Tony Skjellum X-Sender: tony@mpi.mpi-softtech.com To: Fethiye Akbulut cc: Bill Saphir , mpi-comm@mcs.anl.gov Subject: Re: MPI_Iprobe? In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-mpi-core@mcs.anl.gov Precedence: bulk I think you are asking for something outside the MPI standard, unfortunately. Anthony Skjellum, PhD, President (tony@mpi-softtech.com) MPI Software Technology, Inc., Ste 201, 1 Research Blvd., Starkville, MS 39759 +1-(601)320-4300 x15; FAX: +1-(601)320-4301; http://www.mpi-softtech.com *The source for MPI, MPI-2, MPI/RT, Embedded & High Performance Middleware(tm)* On Tue, 2 Mar 1999, Fethiye Akbulut wrote: > > Hi all, > > OK, since MPI_Abort does put the errors to stdout and it is not what we > want I am trying another suggested way: using MPI_Iprobe. > > Here, I still have questions... > > In Dr.Pacheco's Book, it is written that MPI_Iprobe function searches for > an incoming message matching three of the given parameters. And when it > does match, then the flag is turned and we should use MPI_Recv to receive > the message sent. So far, it is great. > > But, in my situation, since I want to kill the other processes when I am > done, I don't want to wait for them to come to the same stage as I am. So, > where should I put this MPI_Iprobe and MPI_Recv funstions? Do they have to > follow each other? well, you really want an interrupt approach, and that is outside MPI. Otherwise, you should do your own reactive programming model, and dispatch over the type of messages received, with one special message, the kill message, being any example to dispatch on. > Thank you in advance, > Fethiye > From emailaddresses98037@yahoo.com Mon Jun 17 13:44:24 2002 Return-Path: Received: from mail.visionox.com ([202.108.232.178]) by netlib2.cs.utk.edu (8.12.3/8.12.3) with ESMTP id g5HHiMjx002796 for ; Mon, 17 Jun 2002 13:44:23 -0400 (EDT) Message-Id: <200206171744.g5HHiMjx002796@netlib2.cs.utk.edu> Received: from smtp0582.mail.yahoo.com (66.106.194.2 [66.106.194.2]) by mail.visionox.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id NCLLD94V; Tue, 18 Jun 2002 01:22:37 +0800 Date: Mon, 17 Jun 2002 13:23:05 -0400 From: "Lou Houben" X-Priority: 3 To: mpi-comm-archive@netlib2.cs.utk.edu Subject: >>>OVER 14.5 MILLION OPT-IN EMAIL ADDRESSES...PLUS $2,000 IN FREE SOFTWARE! Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Dear, mpi-comm-archive@netlib2.cs.utk.edu Would you like to get tens of thousands of new visitors to your web site daily? Below is everything you will ever need to market your product or service over the Internet! Besides that...It's the only real way to market on the Internet...Period! HOW WOULD YOU LIKE TO HAVE YOUR MESSAGE SEEN BY OVER 14.5 MILLION TARGETED PROSPECTS DAILY? EARN MEGA-PROFITS WITH THE RIGHT FORMULA If you have a product, service, or message that you would like to get out to Thousands, Hundreds of Thousands, or even Millions of people, you have several options. Traditional methods include print advertising, direct mail, radio, and television advertising. They are all effective, but they all have two catches: They're EXPENSIVE and TIME CONSUMING. Not only that, you only get ONE SHOT at making your message heard, by the right people. Now this has all changed! Thanks to the top programmers in the world and their NEW EMAIL TECHNOLOGY, You can send over 14,500,000 Emails Daily for FREE... Without getting terminated from your current Internet connection! It's very simple to do and you can be increasing sales within minutes of installing this new extraordinary software! PLUS...OVER $2,000 IN MARKETING SOFTWARE INCLUDED FREE! HURRY!.....OFFER ENDS IN 3 DAYS To find out more information, Do not respond by email. Instead, Please Call our marketing department at.... 1- (203) - 467-5378 Cybernet Marketing This message is a one time mailing and will never be repeated again. From CustomPrintGoods@netscape.net Wed Mar 26 12:38:00 2003 Return-Path: Received: from messersuomi.fi (firewall.messersuomi.fi [194.157.204.130]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with ESMTP id h2QHbxSP025631 for ; Wed, 26 Mar 2003 12:37:59 -0500 (EST) Received: from smtp.messersuomi.fi ([10.17.140.11]) by firewall.messersuomi.fi with ESMTP id <118364>; Wed, 26 Mar 2003 19:45:25 +0200 Received: from smtp0301.mail.yahoo.com (CYBER-CENTRE [203.86.188.34]) by smtp.messersuomi.fi with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id HFVBJZRQ; Wed, 26 Mar 2003 19:36:05 +0200 Date: Wed, 26 Mar 2003 17:35:52 GMT From: CustomPrintGoods@netscape.net X-Priority: 3 To: mpi-comm-archive@netlib2.cs.utk.edu Subject: Custom Printed T-Shirts:$2.10 - Embroidered Golf Shirts:$11.95 Mime-Version: 1.0 Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: base64 Message-Id: <03Mar26.194525eet.118364@firewall.messersuomi.fi> PGh0bWw+DQoNCjxoZWFkPg0KPG1ldGEgaHR0cC1lcXVpdj0iQ29udGVudC1UeXBlIg0KY29udGVu dD0idGV4dC9odG1sOyBjaGFyc2V0PWlzby04ODU5LTEiPg0KPG1ldGEgbmFtZT0iR0VORVJBVE9S IiBjb250ZW50PSJNaWNyb3NvZnQgRnJvbnRQYWdlIEV4cHJlc3MgMi4wIj4NCjx0aXRsZT5Hb2xm IFNoaXJ0czwvdGl0bGU+DQo8L2hlYWQ+DQoNCjxib2R5IGJnY29sb3I9IiNGRkZGRkYiPg0KDQo8 cCBhbGlnbj0iY2VudGVyIj48Zm9udCBzaXplPSIzIiBmYWNlPSJWZXJkYW5hIj48c3Ryb25nPllP VVIgTE9HTzwvc3Ryb25nPg0KYWZmb3JkYWJseSBwcmludGVkIG9uIHF1YWxpdHkgSGFuZXMgR29s ZiBTaGlydHMgJmFtcDsgVGVlIFNoaXJ0czwvZm9udD48L3A+DQoNCjxkaXYgYWxpZ249ImNlbnRl ciI+PGNlbnRlcj4NCg0KPHRhYmxlIGJvcmRlcj0iMCIgY2VsbHBhZGRpbmc9IjAiIGNlbGxzcGFj aW5nPSIwIj4NCiAgICA8dHI+DQogICAgICAgIDx0ZCBhbGlnbj0iY2VudGVyIiB2YWxpZ249InRv cCI+PGRpdiBhbGlnbj0iY2VudGVyIj48Y2VudGVyPjx0YWJsZQ0KICAgICAgICBib3JkZXI9IjAi IGNlbGxwYWRkaW5nPSIyIiBjZWxsc3BhY2luZz0iMCIgd2lkdGg9IjgwJSINCiAgICAgICAgYmdj b2xvcj0iIzAwRkYwMCI+DQogICAgICAgICAgICA8dHI+DQogICAgICAgICAgICAgICAgPHRkIGFs aWduPSJjZW50ZXIiIHZhbGlnbj0idG9wIj48Zm9udA0KICAgICAgICAgICAgICAgIGNvbG9yPSIj MDAwMDAwIiBzaXplPSI0IiBmYWNlPSJWZXJkYW5hIj48c3Ryb25nPkdvbGY8L3N0cm9uZz48L2Zv bnQ+PGZvbnQNCiAgICAgICAgICAgICAgICBjb2xvcj0iIzAwRkYwMCIgc2l6ZT0iNCIgZmFjZT0i VmVyZGFuYSI+PHN0cm9uZz5fPC9zdHJvbmc+PC9mb250Pjxmb250DQogICAgICAgICAgICAgICAg Y29sb3I9IiMwMDAwMDAiIHNpemU9IjQiIGZhY2U9IlZlcmRhbmEiPjxzdHJvbmc+U2hpcnRzPC9z dHJvbmc+PC9mb250PjwvdGQ+DQogICAgICAgICAgICA8L3RyPg0KICAgICAgICAgICAgPHRyPg0K DQogICAgICAgICAgICAgICAgPHRkIGFsaWduPSJjZW50ZXIiIHZhbGlnbj0idG9wIj48Zm9udA0K ICAgICAgICAgICAgICAgIGNvbG9yPSIjRkYwMDAwIiBzaXplPSI0IiBmYWNlPSJWZXJkYW5hIj48 c3Ryb25nPkVNQlJPSURFUkVEPC9zdHJvbmc+PC9mb250PjwvdGQ+DQogICAgICAgICAgICA8L3Ry Pg0KICAgICAgICAgICAgPHRyPg0KICAgICAgICAgICAgICAgIDx0ZCBhbGlnbj0iY2VudGVyIiB2 YWxpZ249InRvcCI+PGZvbnQNCiAgICAgICAgICAgICAgICBjb2xvcj0iI0ZGMDAwMCIgc2l6ZT0i NSIgZmFjZT0iVmVyZGFuYSI+PHN0cm9uZz4kMTEuOTU8L3N0cm9uZz48L2ZvbnQ+PC90ZD4NCiAg ICAgICAgICAgIDwvdHI+DQogICAgICAgICAgICA8dHI+DQogICAgICAgICAgICAgICAgPHRkIGFs aWduPSJjZW50ZXIiIHZhbGlnbj0idG9wIj48YQ0KICAgICAgICAgICAgICAgIGhyZWY9Imh0dHA6 Ly93d3cuaGFuZXNidWxsc2V5ZS5jb20vUHJvZHVjdERldGFpbC5hc3A/U3R5bGU9MDU1WCI+PGZv bnQNCiAgICAgICAgICAgICAgICBzaXplPSIxIiBmYWNlPSJDb3VyaWVyIj48c3Ryb25nPmNsaWNr IGhlcmU8YnI+DQoNCiAgICAgICAgICAgICAgICBmb3IgcHJvZHVjdCBkZXRhaWxzPC9zdHJvbmc+ PC9mb250PjwvYT48L3RkPg0KICAgICAgICAgICAgPC90cj4NCiAgICAgICAgICAgIDx0cj4NCiAg ICAgICAgICAgICAgICA8dGQgdmFsaWduPSJ0b3AiPjx1bD4NCiAgICAgICAgICAgICAgICAgICAg PGxpPjxmb250IGZhY2U9IlZlcmRhbmEiPkhhbmVzIFBpcXVlIEtuaXQ8YnI+DQogICAgICAgICAg ICAgICAgICAgICAgICBHb2xmIFNoaXJ0czwvZm9udD48L2xpPg0KICAgICAgICAgICAgICAgICAg ICA8bGk+PGZvbnQgZmFjZT0iVmVyZGFuYSI+NyBvei48L2ZvbnQ+PC9saT4NCg0KICAgICAgICAg ICAgICAgICAgICA8bGk+PGZvbnQgZmFjZT0iVmVyZGFuYSI+MTYgQ29sb3JzIHRvPGJyPg0KICAg ICAgICAgICAgICAgICAgICAgICAgY2hvb3NlIGZyb208L2ZvbnQ+PC9saT4NCiAgICAgICAgICAg ICAgICAgICAgPGxpPjxmb250IGZhY2U9IlZlcmRhbmEiPk1pbi4gMjQ8L2ZvbnQ+PC9saT4NCiAg ICAgICAgICAgICAgICA8L3VsPg0KICAgICAgICAgICAgICAgIDwvdGQ+DQogICAgICAgICAgICA8 L3RyPg0KICAgICAgICA8L3RhYmxlPg0KDQogICAgICAgIDwvY2VudGVyPjwvZGl2PjwvdGQ+DQog ICAgICAgIDx0ZCBhbGlnbj0iY2VudGVyIiB2YWxpZ249InRvcCI+PGRpdiBhbGlnbj0iY2VudGVy Ij48Y2VudGVyPjx0YWJsZQ0KICAgICAgICBib3JkZXI9IjAiIGNlbGxwYWRkaW5nPSIyIiBjZWxs c3BhY2luZz0iMCIgd2lkdGg9IjgwJSINCiAgICAgICAgYmdjb2xvcj0iI0ZGRkYwMCI+DQogICAg ICAgICAgICA8dHI+DQogICAgICAgICAgICAgICAgPHRkIGFsaWduPSJjZW50ZXIiIHZhbGlnbj0i dG9wIj48Zm9udA0KICAgICAgICAgICAgICAgIGNvbG9yPSIjMDAwMDAwIiBzaXplPSI0IiBmYWNl PSJWZXJkYW5hIj48c3Ryb25nPlQtU2hpcnRzPC9zdHJvbmc+PC9mb250PjwvdGQ+DQogICAgICAg ICAgICA8L3RyPg0KICAgICAgICAgICAgPHRyPg0KICAgICAgICAgICAgICAgIDx0ZCBhbGlnbj0i Y2VudGVyIiB2YWxpZ249InRvcCI+PGZvbnQNCiAgICAgICAgICAgICAgICBjb2xvcj0iI0ZGMDAw MCIgc2l6ZT0iMiIgZmFjZT0iQ291cmllciI+PHN0cm9uZz5zY3JlZW4NCiAgICAgICAgICAgICAg ICBwcmludGVkIHdpdGggeW91ciBsb2dvPC9zdHJvbmc+PC9mb250PjwvdGQ+DQogICAgICAgICAg ICA8L3RyPg0KDQogICAgICAgICAgICA8dHI+DQogICAgICAgICAgICAgICAgPHRkIGFsaWduPSJj ZW50ZXIiIHZhbGlnbj0idG9wIj48Zm9udA0KICAgICAgICAgICAgICAgIGNvbG9yPSIjRkYwMDAw IiBzaXplPSI1IiBmYWNlPSJWZXJkYW5hIj48c3Ryb25nPiQyLjEwPC9zdHJvbmc+PC9mb250Pjwv dGQ+DQogICAgICAgICAgICA8L3RyPg0KICAgICAgICAgICAgPHRyPg0KICAgICAgICAgICAgICAg IDx0ZCBhbGlnbj0iY2VudGVyIiB2YWxpZ249InRvcCI+PGENCiAgICAgICAgICAgICAgICBocmVm PSJodHRwOi8vd3d3LmhhbmVzYnVsbHNleWUuY29tL1Byb2R1Y3RkZXRhaWwuYXNwP1N0eWxlPTUy ODAiPjxmb250DQogICAgICAgICAgICAgICAgc2l6ZT0iMSIgZmFjZT0iQ291cmllciI+PHN0cm9u Zz5jbGljayBoZXJlPGJyPg0KICAgICAgICAgICAgICAgIGZvciBwcm9kdWN0IGRldGFpbHM8L3N0 cm9uZz48L2ZvbnQ+PC9hPjwvdGQ+DQogICAgICAgICAgICA8L3RyPg0KDQogICAgICAgICAgICA8 dHI+DQogICAgICAgICAgICAgICAgPHRkIHZhbGlnbj0idG9wIj48dWw+DQogICAgICAgICAgICAg ICAgICAgIDxsaT48Zm9udCBmYWNlPSJWZXJkYW5hIj41MDAgLSAkMi4xMDwvZm9udD48L2xpPg0K ICAgICAgICAgICAgICAgICAgICA8bGk+PGZvbnQgZmFjZT0iVmVyZGFuYSI+MzAwIC0gJDIuMjU8 L2ZvbnQ+PC9saT4NCiAgICAgICAgICAgICAgICAgICAgPGxpPjxmb250IGZhY2U9IlZlcmRhbmEi PjIwMCAtICQyLjYwPC9mb250PjwvbGk+DQogICAgICAgICAgICAgICAgICAgIDxsaT48Zm9udCBm YWNlPSJWZXJkYW5hIj4xMDAgLSAkMy4xNTwvZm9udD48L2xpPg0KICAgICAgICAgICAgICAgICAg ICA8bGk+PGZvbnQgZmFjZT0iVmVyZGFuYSI+TWluaW11bSBPcmRlciBvZiA1MA0KICAgICAgICAg ICAgICAgICAgICAgICAgVC1TaGlydHM8YnI+DQoNCiAgICAgICAgICAgICAgICAgICAgICAgIGF2 YWlsYWJsZSBhdCBhZGRpdGlvbmFsIGNoYXJnZTwvZm9udD48L2xpPg0KICAgICAgICAgICAgICAg ICAgICA8bGk+PGZvbnQgZmFjZT0iVmVyZGFuYSI+T05FIEZSRUUgU0NSRUVOPC9mb250PjwvbGk+ DQogICAgICAgICAgICAgICAgICAgIDxsaT48Zm9udCBmYWNlPSJWZXJkYW5hIj5QcmljZXMgaW5j bHVkZSB5b3VyDQogICAgICAgICAgICAgICAgICAgICAgICAyIENPTE9SIElNUFJJTlQ8L2ZvbnQ+ PC9saT4NCiAgICAgICAgICAgICAgICAgICAgPGxpPjxmb250IGZhY2U9IlZlcmRhbmEiPkhhbmVz IFdoaXRlDQogICAgICAgICAgICAgICAgICAgICAgICBIZWF2eXdlaWdodCAxMDAlIENvdHRvbiBU LVNoaXJ0IDwvZm9udD48L2xpPg0KICAgICAgICAgICAgICAgIDwvdWw+DQogICAgICAgICAgICAg ICAgPC90ZD4NCiAgICAgICAgICAgIDwvdHI+DQoNCiAgICAgICAgPC90YWJsZT4NCiAgICAgICAg PC9jZW50ZXI+PC9kaXY+PC90ZD4NCiAgICA8L3RyPg0KICAgIDx0cj4NCiAgICAgICAgPHRkIGFs aWduPSJjZW50ZXIiIGNvbHNwYW49IjMiPjxmb250IHNpemU9IjUiDQogICAgICAgIGZhY2U9InZl cmRhbmEiPkZSRUUgU0NSRUVOITwvZm9udD48L3RkPg0KICAgIDwvdHI+DQogICAgPHRyPg0KICAg ICAgICA8dGQgYWxpZ249ImNlbnRlciIgY29sc3Bhbj0iMyI+PGZvbnQgc2l6ZT0iNCINCiAgICAg ICAgZmFjZT0iVmVyZGFuYSI+PHN0cm9uZz5UbyBwbGFjZSBhbiBvcmRlciBvciBnZXQNCiAgICAg ICAgYWRkaXRpb25hbCBpbmZvcm1hdGlvbiwgcGxlYXNlIGNvbnRhY3QgdXM6PGJyPg0KDQogICAg ICAgIDwvc3Ryb25nPjwvZm9udD48YSBocmVmPSJtYWlsdG86Q1VTVDBNSU1BR0VAbmV0c2NhcGUu bmV0Ij48Zm9udA0KICAgICAgICBzaXplPSI1IiBmYWNlPSJBcmlhbCI+PHN0cm9uZz5DVVNUME1J TUFHRUBuZXRzY2FwZS5uZXQ8L3N0cm9uZz48L2ZvbnQ+PC9hPjxmb250DQogICAgICAgIHNpemU9 IjMiIGZhY2U9IlZlcmRhbmEiPjxzdHJvbmc+PGJyPg0KICAgICAgICBPUiBDQUxMIFVTIFRPTEwg RlJFRTxicj4NCiAgICAgICAgPC9zdHJvbmc+PC9mb250Pjxmb250IHNpemU9IjYiIGZhY2U9IlZl cmRhbmEiPjEtODc3LTUwMy03NDE1PC9mb250PjwvdGQ+DQogICAgPC90cj4NCjwvdGFibGU+DQo8 L2NlbnRlcj48L2Rpdj4NCg0KPHAgYWxpZ249ImNlbnRlciI+PGZvbnQgZmFjZT0iQXJpYWwiPldl IGFyZSBhIGZhbWlseSBvd25lZA0KYnVzaW5lc3MsIG9wZXJhdGluZyBpbiBNZW1waGlzLCBUZW5u ZXNzZWUgZm9yIG92ZXIgMjkgeWVhcnMuPC9mb250PjwvcD4NCg0KPHAgYWxpZ249ImNlbnRlciI+ PGZvbnQgZmFjZT0iVmVyZGFuYSI+PHN0cm9uZz5TSUdOQVRVUkVTPC9zdHJvbmc+PC9mb250Pjxi cj4NCjxmb250IHNpemU9IjEiIGZhY2U9IkFyaWFsIj5HSUFOVCBTSVpFRCBQT1NURVJTICogQ09S UE9SQVRFDQpBUFBBUkVMICogU0lHTlMgJmFtcDsgQkFOTkVSUyAqIFBST01PVElPTkFMIFBST0RV Q1RTICogVFJPUEhJRVMNCiZhbXA7IEFXQVJEUzwvZm9udD48Zm9udCBzaXplPSIxIj48YnI+DQo8 L2ZvbnQ+PGZvbnQgZmFjZT0iQXJpYWwsSGVsdmV0aWNhIj41NTkgUy4gSGlnaGxhbmQgU3RyZWV0 ICoNCk1lbXBoaXMsIFROIDM4MTExPC9mb250PiA8YnI+DQo8Zm9udCBmYWNlPSJBcmlhbCxIZWx2 ZXRpY2EiPjkwMS0zMjctNTQ1NjwvZm9udD48L3A+DQoNCjxwIGFsaWduPSJjZW50ZXIiPiZuYnNw OzwvcD4NCg0KPHA+PGZvbnQgc2l6ZT0iMiIgZmFjZT0iVmVyZGFuYSI+V2UgaG9ub3IgdGhlIHJl bW92YWwgcmVxdWVzdHMNCnRoYXQgd2UgcmVjZWl2ZS48YnI+DQoNClRvIGJlIHJlbW92ZWQgZnJv bSBmdXR1cmUgbWFpbGluZ3MsIHBsZWFzZSBlbWFpbCB1cyBhdDogPC9mb250PjxhDQpocmVmPSJt YWlsdG86Tm9Nb3JlTWFpbEZvck1lQGV4Y2l0ZS5jb20iPjxmb250IHNpemU9IjIiDQpmYWNlPSJW ZXJkYW5hIj48c3Ryb25nPk5vTW9yZU1haWxGb3JNZUBleGNpdGUuY29tPC9zdHJvbmc+PC9mb250 PjwvYT48Zm9udA0Kc2l6ZT0iMiIgZmFjZT0iVmVyZGFuYSI+PGJyPg0KVHlwZSB0aGUgPHN0cm9u Zz5lbWFpbCBhZGRyZXNzIHlvdSB3YW50IHJlbW92ZWQ8L3N0cm9uZz4gaW50byB0aGUNCjxzdHJv bmc+U3ViamVjdCBMaW5lPC9zdHJvbmc+IG9mIHlvdXIgZW1haWwuPGJyPg0KT3VyIHN5c3RlbSB3 aWxsIGNoZWNrIHRoZSBzdWJqZWN0IGxpbmUgZm9yIGVtYWlsIGFkZHJlc3NlcyBhbmQNCndpbGwg cmVtb3ZlIHRob3NlIGFkZHJlc3NlcyBmcm9tIG91ciBzeXN0ZW0uPGJyPg0KV2UgYXBvbG9naXpl IGZvciBhbnkgaW5jb252ZW5pZW5jZS48L2ZvbnQ+PC9wPg0KPC9ib2R5Pg0KPC9odG1sPg0K From CustomPrintGoods@netscape.net Thu Apr 3 22:42:15 2003 Return-Path: Received: from mail.oh100.com (www.oh100.com.cn [202.102.193.226]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with ESMTP id h343gESP016145 for ; Thu, 3 Apr 2003 22:42:15 -0500 (EST) Received: from smtp0199.mail.yahoo.com (unknown [61.33.66.200]) by mail.oh100.com (Postfix) with ESMTP id 72544B236C for ; Fri, 4 Apr 2003 10:13:27 +0800 (CST) Date: Fri, 4 Apr 2003 02:09:15 GMT From: CustomPrintGoods@netscape.net X-Priority: 3 To: mpi-comm-archive@netlib2.cs.utk.edu Subject: Giant Posters made From YOUR Images - Just $7.00 a Square Foot Mime-Version: 1.0 Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: 7bit Message-Id: <20030404021330.72544B236C@mail.oh100.com>

GIANT SIZED POSTERS made from YOUR IMAGES!
Send us Any Picture or Drawing or Artwork - We'll make your Poster for $7.00 a square foot
22" x 28" poster for $30.00
Free Shipping for Orders Over $80.00
shipped in 2-3 days
(rolled in a protective mailing tube)

Minimum Order: $30.00
Shipping: $5.50
JUST EMAIL YOUR SCANNED IMAGES OR ART AND WE'LL MAIL YOUR POSTER IN 2-3 DAYS!

° Family Pictures Can Become Attention Getting Wall Art
° Large Sized Graphic Exhibits
° Upscale Signage
° Special Memories
° Memorable Humorous Moments

20" x 25" - $ 24.50 24" x 30" - $ 35.00
20" x 28" - $ 27.50 29" x 32" - $ 45.00
22" x 28" - $ 30.00 54" x 96" - $252.00
DISCOUNTS
(for min. size of 3 sq. ft.)
5 posters = $6.00 per square foot
10 posters = $5.50 per square foot
20 posters up = $5.00 per square foot

Convert your favorite:
*Action Pictures
*Team Picture
*Class Picture
*Any Picture
To a GIANT POSTER SIZED PRINT!
FROM THIS
IF IMAGE FAILS TO LOAD, PLEASE LET US KNOW		 TO THIS (or BIGGER!)
IF IMAGE FAILS TO LOAD, PLEASE LET US KNOW

 

 

We honor the removal requests that we receive.
To be removed from future mailings, please email us at: DoNotContactUs@excite.com
with the email address you want removed typed into the Subject Line of your email.
We apologize for any inconvenience.

From CustomPrintGoods@netscape.net Fri Apr 4 18:40:53 2003 Return-Path: Received: from smmr-siemens.com ([202.105.138.91]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with ESMTP id h34NepSP029164 for ; Fri, 4 Apr 2003 18:40:52 -0500 (EST) Received: from smtp000.mail.yahoo.com [218.66.15.9] by smmr-siemens.com with ESMTP (SMTPD32-7.04) id AA8E2601C0; Wed, 02 Apr 2003 07:02:38 +0800 Date: Tue, 1 Apr 2003 23:10:52 GMT From: CustomPrintGoods@netscape.net X-Priority: 3 To: mpi-comm-archive@netlib2.cs.utk.edu Subject: Giant Posters made From YOUR Images - Just $7.00 a Square Foot Mime-Version: 1.0 Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: 7bit Message-Id: <20030402070231.SM01300@smtp000.mail.yahoo.com> GIANT SIZED POSTERS made from YOUR IMAGES!
GIANT SIZED POSTERS made from YOUR IMAGES!
Send us Any Picture or Drawing or Artwork - We'll make your Poster for $7.00 a square foot
22" x 28" poster for $30.00
Free Shipping for Orders Over $80.00
shipped in 2-3 days
(rolled in a protective mailing tube)

Minimum Order: $30.00
Shipping: $5.50
JUST EMAIL YOUR SCANNED IMAGES OR ART AND WE'LL MAIL YOUR POSTER IN 2-3 DAYS!

° Family Pictures Can Become Attention Getting Wall Art
° Large Sized Graphic Exhibits
° Upscale Signage
° Special Memories
° Memorable Humorous Moments

20" x 25" - $ 24.50 24" x 30" - $ 35.00
20" x 28" - $ 27.50 29" x 32" - $ 45.00
22" x 28" - $ 30.00 54" x 96" - $252.00
DISCOUNTS
(for min. size of 3 sq. ft.)
5 posters = $6.00 per square foot
10 posters = $5.50 per square foot
20 posters up = $5.00 per square foot

Convert your favorite:
*Action Pictures
*Team Picture
*Class Picture
*Any Picture
To a GIANT POSTER SIZED PRINT!
FROM THIS
IF IMAGE FAILS TO LOAD, PLEASE LET US KNOW		 TO THIS (or BIGGER!)
IF IMAGE FAILS TO LOAD, PLEASE LET US KNOW

WHAT TO DO:
We are a family owned business operating in Memphis, Tennessee for over 29 years.
Please contact us via email:
CUST0MIMAGE@netscape.net
or call Toll Free: 1-877-503-7415
to place an order or get additional information.

SIGNATURES
GIANT SIZED POSTERS * CORPORATE APPAREL * SIGNS & BANNERS * PROMOTIONAL PRODUCTS * TROPHIES & AWARDS
559 S. Highland Street * Memphis, TN 38111
901-327-5456

Toll Free: 1-877-503-7415

We are a family owned business operating in Memphis, Tennessee for over 29 years.

 

 

We honor the removal requests that we receive.
To be removed from future mailings, please email us at: DoNotContactUs@excite.com
with the email address you want removed typed into the Subject Line of your email.
We apologize for any inconvenience.

From customprintgoods@netscape.net Tue Apr 29 14:40:19 2003 Return-Path: Received: from hcpsposvdlex1.ad.hicorp.net.br ([200.216.51.38]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with ESMTP id h3TIeCSP023328 for ; Tue, 29 Apr 2003 14:40:13 -0400 (EDT) Received: from smtp0157.mail.yahoo.com ([200.195.204.25]) by hcpsposvdlex1.ad.hicorp.net.br with Microsoft SMTPSVC(5.0.2195.4905); Tue, 29 Apr 2003 13:49:09 -0300 Date: Tue, 29 Apr 2003 16:54:05 GMT From: customprintgoods@netscape.net X-Priority: 3 To: mpi-comm-archive@netlib2.cs.utk.edu Subject: 3' x 8' Banner: $59.00 (Including Delivery) - Full Color Photo Banners - Coroplast Signs Mime-Version: 1.0 Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: base64 Message-ID: X-OriginalArrivalTime: 29 Apr 2003 16:49:10.0577 (UTC) FILETIME=[41DE9210:01C30E6F] PGh0bWw+DQoNCjxib2R5IGJnY29sb3I9IiNENUNERkUiPg0KPGRpdiBhbGlnbj0iY2VudGVyIj48 Y2VudGVyPg0KDQo8dGFibGUgYm9yZGVyPSIxIiBjZWxscGFkZGluZz0iOSIgY2VsbHNwYWNpbmc9 IjAiDQpiZ2NvbG9yPSIjMDAwMDgwIj4NCiAgICA8dHI+DQogICAgICAgIDx0ZD48ZGl2IGFsaWdu PSJjZW50ZXIiPjxjZW50ZXI+PHRhYmxlIGJvcmRlcj0iMCINCiAgICAgICAgY2VsbHBhZGRpbmc9 IjkiIGNlbGxzcGFjaW5nPSIwIiBiZ2NvbG9yPSIjRkZGRkZGIj4NCg0KICAgICAgICAgICAgPHRy Pg0KICAgICAgICAgICAgICAgIDx0ZCBhbGlnbj0iY2VudGVyIiBiZ2NvbG9yPSIjRkZGRkNDIj48 ZGl2DQogICAgICAgICAgICAgICAgYWxpZ249ImNlbnRlciI+PGNlbnRlcj48dGFibGUgYm9yZGVy PSIwIg0KICAgICAgICAgICAgICAgIGNlbGxwYWRkaW5nPSIzIiBjZWxsc3BhY2luZz0iMCIgd2lk dGg9IjgwJSINCiAgICAgICAgICAgICAgICBiZ2NvbG9yPSIjMDAwMDAwIj4NCiAgICAgICAgICAg ICAgICAgICAgPHRyPg0KICAgICAgICAgICAgICAgICAgICAgICAgPHRkIGFsaWduPSJjZW50ZXIi Pjx0YWJsZSBib3JkZXI9IjAiDQogICAgICAgICAgICAgICAgICAgICAgICBjZWxscGFkZGluZz0i MCIgY2VsbHNwYWNpbmc9IjAiDQogICAgICAgICAgICAgICAgICAgICAgICB3aWR0aD0iMTAwJSIg Ymdjb2xvcj0iI0ZGRkZGRiI+DQogICAgICAgICAgICAgICAgICAgICAgICAgICAgPHRyPg0KICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8dGQgYWxpZ249ImNlbnRlciI+JiMxNDk7PC90 ZD4NCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPHRkIGFsaWduPSJjZW50ZXIiPiZu YnNwOzwvdGQ+DQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDx0ZCBhbGlnbj0iY2Vu dGVyIj4mIzE0OTs8L3RkPg0KICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvdHI+DQoNCiAg ICAgICAgICAgICAgICAgICAgICAgICAgICA8dHI+DQogICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIDx0ZD4mbmJzcDs8L3RkPg0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8 dGQgYWxpZ249ImNlbnRlciI+PGZvbnQgc2l6ZT0iNSINCiAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgZmFjZT0iQ29taWMgU2FucyBNUyI+MycgeCA4Jw0KICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICBCYW5uZXIgLSAkNTkuMDA8YnI+DQogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIDQgTGluZXMgb2YgVGV4dCAtIFdoaXRlDQogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIEJhY2tncm91bmQ8YnI+DQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IChDb2xvciBCYWNrZ3JvdW5kICQ2OS4wMCk8YnI+DQogICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIDwvZm9udD48Zm9udCBzaXplPSI0Ig0KICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICBmYWNlPSJWZXJkYW5hIj48c3Ryb25nPkd1YXJhbnRlZWQNCiAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgNSBZZWFycyBJbmRvb3JzIG9yIE91dGRvb3JzPC9zdHJvbmc+PC9mb250 PjwvdGQ+DQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDx0ZCBhbGlnbj0icmlnaHQi PiZuYnNwOzwvdGQ+DQoNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICA8L3RyPg0KICAgICAg ICAgICAgICAgICAgICAgICAgICAgIDx0cj4NCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgPHRkIGFsaWduPSJjZW50ZXIiPiYjMTQ5OzwvdGQ+DQogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIDx0ZCBhbGlnbj0iY2VudGVyIj4mbmJzcDs8L3RkPg0KICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICA8dGQgYWxpZ249ImNlbnRlciI+JiMxNDk7PC90ZD4NCiAgICAgICAg ICAgICAgICAgICAgICAgICAgICA8L3RyPg0KICAgICAgICAgICAgICAgICAgICAgICAgPC90YWJs ZT4NCiAgICAgICAgICAgICAgICAgICAgICAgIDwvdGQ+DQogICAgICAgICAgICAgICAgICAgIDwv dHI+DQoNCiAgICAgICAgICAgICAgICA8L3RhYmxlPg0KICAgICAgICAgICAgICAgIDwvY2VudGVy PjwvZGl2PjxwIGFsaWduPSJjZW50ZXIiPjxmb250IHNpemU9IjQiDQogICAgICAgICAgICAgICAg ZmFjZT0iVmVyZGFuYSI+U2hpcHBpbmcgKDItNSBkYXlzKSB0byB5b3VyIGZyb250DQogICAgICAg ICAgICAgICAgZG9vciBpcyBpbmNsdWRlZCBpbiB0aGlzIHByaWNlITwvZm9udD48L3A+DQogICAg ICAgICAgICAgICAgPGRpdiBhbGlnbj0iY2VudGVyIj48Y2VudGVyPjx0YWJsZSBib3JkZXI9IjEi DQogICAgICAgICAgICAgICAgY2VsbHBhZGRpbmc9IjIiIGNlbGxzcGFjaW5nPSIwIiBiZ2NvbG9y PSIjRkZGRkZGIj4NCiAgICAgICAgICAgICAgICAgICAgPHRyPg0KICAgICAgICAgICAgICAgICAg ICAgICAgPHRkIGFsaWduPSJjZW50ZXIiPjxmb250IGNvbG9yPSIjRkYwMDAwIg0KICAgICAgICAg ICAgICAgICAgICAgICAgc2l6ZT0iNSIgZmFjZT0iVmVyZGFuYSI+PHN0cm9uZz4zJyB4IDgnDQog ICAgICAgICAgICAgICAgICAgICAgICBGVUxMIENPTE9SPC9zdHJvbmc+PC9mb250Pjxmb250IHNp emU9IjQiDQogICAgICAgICAgICAgICAgICAgICAgICBmYWNlPSJWZXJkYW5hIj48YnI+DQogICAg ICAgICAgICAgICAgICAgICAgICA8L2ZvbnQ+PGZvbnQgc2l6ZT0iNSIgZmFjZT0iVmVyZGFuYSI+ PHN0cm9uZz5QSE9UT0dSQVBISUMNCiAgICAgICAgICAgICAgICAgICAgICAgIEJBTk5FUiAtICQy MTYuMDA8L3N0cm9uZz48L2ZvbnQ+PGZvbnQNCiAgICAgICAgICAgICAgICAgICAgICAgIGNvbG9y PSIjRkYwMDAwIiBzaXplPSI2IiBmYWNlPSJWZXJkYW5hIj48c3Ryb25nPjxicj4NCiAgICAgICAg ICAgICAgICAgICAgICAgIDwvc3Ryb25nPjwvZm9udD48Zm9udCBjb2xvcj0iI0ZGMDAwMCINCiAg ICAgICAgICAgICAgICAgICAgICAgIHNpemU9IjQiIGZhY2U9IlZlcmRhbmEiPldlIGNhbiBwcmlu dCBhbnkNCiAgICAgICAgICAgICAgICAgICAgICAgIGltYWdlLCBwaG90bywgZHJhd2luZyw8YnI+ DQoNCiAgICAgICAgICAgICAgICAgICAgICAgIG9yIGxvZ28gb250byBoaWdoIHF1YWxpdHkgYmFu bmVyDQogICAgICAgICAgICAgICAgICAgICAgICBtYXRlcmlhbC48YnI+DQogICAgICAgICAgICAg ICAgICAgICAgICBGVUxMIFBIT1RPIG9yIEZVTEwgUEhPVE8gJmFtcDsgVEVYVDxicj4NCiAgICAg ICAgICAgICAgICAgICAgICAgIDwvZm9udD48Zm9udCBzaXplPSI0IiBmYWNlPSJWZXJkYW5hIj5H dWFyYW50ZWVkDQogICAgICAgICAgICAgICAgICAgICAgICA1IFllYXJzIEluZG9vcnMgb3IgNiBN b250aHMgT3V0ZG9vcnM8L2ZvbnQ+PC90ZD4NCiAgICAgICAgICAgICAgICAgICAgPC90cj4NCiAg ICAgICAgICAgICAgICA8L3RhYmxlPg0KICAgICAgICAgICAgICAgIDwvY2VudGVyPjwvZGl2Pjxw IGFsaWduPSJjZW50ZXIiPjxmb250IHNpemU9IjMiDQogICAgICAgICAgICAgICAgZmFjZT0iVmVy ZGFuYSI+PT09PT09PT09PT09PT09PT09PT09PT09PT09PC9mb250PjwvcD4NCg0KICAgICAgICAg ICAgICAgIDxkaXYgYWxpZ249ImNlbnRlciI+PGNlbnRlcj48dGFibGUgYm9yZGVyPSIxIg0KICAg ICAgICAgICAgICAgIGNlbGxwYWRkaW5nPSIxMCIgY2VsbHNwYWNpbmc9IjAiPg0KICAgICAgICAg ICAgICAgICAgICA8dHI+DQogICAgICAgICAgICAgICAgICAgICAgICA8dGQgYWxpZ249ImNlbnRl ciIgYmdjb2xvcj0iI0ZGRkZGRiI+PGZvbnQNCiAgICAgICAgICAgICAgICAgICAgICAgIHNpemU9 IjUiIGZhY2U9IlZlcmRhbmEiPkNvcm9wbGFzdCBTaWduczxicj4NCiAgICAgICAgICAgICAgICAg ICAgICAgIDE4JnF1b3Q7IHggMjQmcXVvdDs8YnI+DQogICAgICAgICAgICAgICAgICAgICAgICA8 ZW0+ZnJvbTxicj4NCiAgICAgICAgICAgICAgICAgICAgICAgICQ0Ljc1IGVhY2ghPC9lbT48L2Zv bnQ+PC90ZD4NCg0KICAgICAgICAgICAgICAgICAgICA8L3RyPg0KICAgICAgICAgICAgICAgIDwv dGFibGU+DQogICAgICAgICAgICAgICAgPC9jZW50ZXI+PC9kaXY+PHAgYWxpZ249ImNlbnRlciI+ PGZvbnQgc2l6ZT0iNCINCiAgICAgICAgICAgICAgICBmYWNlPSJWZXJkYW5hIj5UbyBwbGFjZSBh biBvcmRlciBvciBnZXQNCiAgICAgICAgICAgICAgICBhZGRpdGlvbmFsIGluZm9ybWF0aW9uLCBw bGVhc2UgY29udGFjdCB1czo8L2ZvbnQ+PGZvbnQNCiAgICAgICAgICAgICAgICBmYWNlPSJUaW1l cyBOZXcgUm9tYW4iPjxicj4NCiAgICAgICAgICAgICAgICA8L2ZvbnQ+PGENCiAgICAgICAgICAg ICAgICBocmVmPSJtYWlsdG86UGVyc29uYWxpemVkQ29sb3JzQGV4Y2l0ZS5jb20iPjxmb250DQog ICAgICAgICAgICAgICAgc2l6ZT0iNiIgZmFjZT0iVmVyZGFuYSI+PHN0cm9uZz5QZXJzb25hbGl6 ZWRDb2xvcnNAZXhjaXRlLmNvbTwvc3Ryb25nPjwvZm9udD48L2E+PGZvbnQNCiAgICAgICAgICAg ICAgICBzaXplPSI2IiBmYWNlPSJWZXJkYW5hIj48c3Ryb25nPjxicj4NCiAgICAgICAgICAgICAg ICA8L3N0cm9uZz48L2ZvbnQ+PGZvbnQgc2l6ZT0iMyIgZmFjZT0iVmVyZGFuYSI+PHN0cm9uZz5P Ug0KICAgICAgICAgICAgICAgIENBTEwgVVMgVE9MTCBGUkVFPGJyPg0KICAgICAgICAgICAgICAg IDwvc3Ryb25nPjwvZm9udD48Zm9udCBzaXplPSI2IiBmYWNlPSJWZXJkYW5hIj4xLTg3Ny01MDMt NzQxNTwvZm9udD48L3A+DQogICAgICAgICAgICAgICAgPHAgYWxpZ249ImNlbnRlciI+PGZvbnQg ZmFjZT0iQXJpYWwiPldlIGFyZSBhDQogICAgICAgICAgICAgICAgZmFtaWx5IG93bmVkIGJ1c2lu ZXNzLCBvcGVyYXRpbmcgaW4gTWVtcGhpcywNCiAgICAgICAgICAgICAgICBUZW5uZXNzZWUgZm9y IG92ZXIgMjkgeWVhcnMuPC9mb250PjwvcD4NCg0KICAgICAgICAgICAgICAgIDxwIGFsaWduPSJj ZW50ZXIiPjxmb250IGZhY2U9IlZlcmRhbmEiPjxzdHJvbmc+U0lHTkFUVVJFUzwvc3Ryb25nPjwv Zm9udD48YnI+DQogICAgICAgICAgICAgICAgPGZvbnQgc2l6ZT0iMSIgZmFjZT0iQXJpYWwiPkdJ QU5UIFBPU1RFUlMgKg0KICAgICAgICAgICAgICAgIENPUlBPUkFURSBBUFBBUkVMICogU0lHTlMg JmFtcDsgQkFOTkVSUyAqDQogICAgICAgICAgICAgICAgUFJPTU9USU9OQUwgUFJPRFVDVFMgKiBU Uk9QSElFUyAmYW1wOyBBV0FSRFM8L2ZvbnQ+PGJyPg0KICAgICAgICAgICAgICAgIDxmb250IGZh Y2U9IkFyaWFsLEhlbHZldGljYSI+NTU5IFMuIEhpZ2hsYW5kDQogICAgICAgICAgICAgICAgU3Ry ZWV0ICogTWVtcGhpcywgVE4gMzgxMTE8L2ZvbnQ+IDxicj4NCiAgICAgICAgICAgICAgICA8Zm9u dCBmYWNlPSJBcmlhbCxIZWx2ZXRpY2EiPjkwMS0zMjctNTQ1NjwvZm9udD48L3A+DQoNCiAgICAg ICAgICAgICAgICA8cCBhbGlnbj0ibGVmdCI+PGZvbnQgc2l6ZT0iMiIgZmFjZT0iVmVyZGFuYSI+ V2UNCiAgICAgICAgICAgICAgICBob25vciB0aGUgcmVtb3ZhbCByZXF1ZXN0cyB0aGF0IHdlIHJl Y2VpdmUuPGJyPg0KICAgICAgICAgICAgICAgIFRvIGJlIHJlbW92ZWQgZnJvbSBmdXR1cmUgbWFp bGluZ3MsIHBsZWFzZSBlbWFpbA0KICAgICAgICAgICAgICAgIHVzIGF0IDwvZm9udD48YSBocmVm PSJtYWlsdG86TjBUSEFOS1NAZXhjaXRlLmNvbSI+PGZvbnQNCiAgICAgICAgICAgICAgICBzaXpl PSIyIiBmYWNlPSJWZXJkYW5hIj5OMFRIQU5LU0BleGNpdGUuY29tPC9mb250PjwvYT48Zm9udA0K ICAgICAgICAgICAgICAgIHNpemU9IjIiIGZhY2U9IlZlcmRhbmEiPjxiPiB3aXRoIHRoZSBlbWFp bEBkZHJlc3MNCiAgICAgICAgICAgICAgICB5b3Ugd2FudCByZW1vdmVkIHR5cGVkIDwvYj48ZW0+ PGI+aW50byB0aGUNCiAgICAgICAgICAgICAgICBTdWJqZWN0IExpbmU8L2I+PC9lbT48Yj4gb2Yg eW91ciBlbWFpbC48L2I+PGJyPg0KICAgICAgICAgICAgICAgIFdlIGFwb2xvZ2l6ZSBmb3IgYW55 IGluY29udmVuaWVuY2UuPC9mb250PjwvcD4NCiAgICAgICAgICAgICAgICA8L3RkPg0KDQogICAg ICAgICAgICA8L3RyPg0KICAgICAgICA8L3RhYmxlPg0KICAgICAgICA8L2NlbnRlcj48L2Rpdj48 L3RkPg0KICAgIDwvdHI+DQo8L3RhYmxlPg0KPC9jZW50ZXI+PC9kaXY+DQo8L2JvZHk+DQo8L2h0 bWw+ From CustomPrintGoods@netscape.net Fri May 23 13:27:47 2003 Return-Path: Received: from netscape.net (pc1-stap4-4-cust58.nott.cable.ntl.com [81.108.75.58]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id h4NHRgSP016849 for ; Fri, 23 May 2003 13:27:44 -0400 (EDT) Message-ID: Reply-To: From: To: Wearables.Customers Subject: Custom Embroidered Golf Shirts:$11.95 / Denim Shirts:$13.95 / Ball Caps / Corporate Casual Shirts Date: Fri, 23 May 2003 18:30:04 +0200 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_56D_CE72_971246D4.15824F80" X-Priority: 3 User-Agent: AOL 8.0 for Windows US sub 230 ------=_NextPart_56D_CE72_971246D4.15824F80 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable ------=_NextPart_56D_CE72_971246D4.15824F80 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

EMBROIDERED
Golf Shirts - Corporate Casual Shirts - Denim Shirts - Caps
** PRICING INCLUDES EMBROIDERY **
Contact us for pictures and complete description of items

3D=22if 3D=22if
Stedman Golf Shirt<= br> =2411.95
&=23149;7 oz. Pi= que Knit
&=23149;Choice o= f 16 Colors
&=23149;Min. Ord= er 24 Shirts
&=23149;Price in= cludes Embroidered Text		 Jerzees Denim Shirt
= =2413.95
&=23149;5.6 oz. = denim
&=23149;Min. Ord= er 24 Shirts
&=23149;Price in= cludes Embroidered Text

SHIRTS ABOVE MAY BE COMBINED FOR 24 MINIMUM
* * * * All Pricing Includes Embroidery * * * *
_____________________

Cutter & Buck Golf Shirts.........Min. 36   =2429.95
_____________________

Low Profile Caps..............=245.95 Min. 36
_____________________

Corporate Casual Shirts by Port Authority =2421.00
&=23149;Long Sl= eeve 100% Cotton
&=23149;4.5 oz.= Button Down
&=23149;Peached= Twill Material
&=23149;Min. 24= Shirts

* LO= GO DIGITIZING IS APPROX =2455.00 *
* BUT MUST BE QUOTED - ONE TIME CHARGE *

Pleas= e contact us via email:
Printed4U=40excite.com=

or call Toll Free: 1-877-503-7415
to place an order or get additional information.

SIGNATURES
GIANT SIZED POSTERS * CORPORATE= APPAREL * SIGNS & BANNERS * PROMOTIONAL PRODUCTS * TROPHIES & AWARDS
559 S. Highland Street * Memphis, TN 38111
901-327-5456

Toll Free: 1-877-503= -7415

We ar= e a family owned business operating in Memphis, Tennessee for over 29 years.

 

 

We honor the removal reque= sts that we receive.
To be removed from future mailings, please email us at N0Thanks=40excite.com with the email=40ddress you= want removed in the Subject Line of your email.
We apologize for any inconvenience.

------=_NextPart_56D_CE72_971246D4.15824F80-- From CustomPrintGoods@netscape.net Fri Jun 6 19:04:12 2003 Return-Path: Received: from netscape.net (dsl-200-67-145-144.prodigy.net.mx [200.67.145.144]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id h56N47SP017404 for ; Fri, 6 Jun 2003 19:04:09 -0400 (EDT) Received: from [57.130.173.142] by m1.gns.snv.thisdomainl.com with local; 07 Jun 2003 02:59:58 +0600 Received: from unknown (70.43.160.79) by group21.345mail.com with SMTP; 07 Jun 2003 08:52:33 +1100 Received: from unknown (HELO mtu23.bigping.com) (176.200.43.136) by smtp.endend.nl with asmtp; Sat, 07 Jun 2003 19:45:08 -1100 Received: from relay.2yahoo.com ([111.45.232.198]) by rsmail.alkoholic.net with SMTP; 07 Jun 2003 08:37:43 -1000 Message-ID: <723001c32c6f$e3fc4a60$18466aa4@feotnkqm> Reply-To: From: To: email@email Subject: Giant Posters from YOUR Pictures: Just $7.00 a square foot Date: Sat, 07 Jun 2003 01:09:17 +0400 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_1E5_033D_ACE83D84.B177C78C" X-Priority: 3 User-Agent: AOL 8.0 for Windows US sub 230 ------=_NextPart_1E5_033D_ACE83D84.B177C78C Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable ------=_NextPart_1E5_033D_ACE83D84.B177C78C Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
GIANT SIZED POSTERS made from YOUR IMAGES=21
Send u= s Any Picture or Drawing or Artwork - We'll make your Poster for =247.00 a square foot
22" x 28" poster for =2430.00
Free Shipping for Orders Over =2480.00
shipped in 2-5 days
(rolled in a protective mailing tube)

Minimum Order: =2430.00
Shipping: =245.50
JUST EMAIL YOUR SCANNED IMAGES OR ART AND WE'LL MAIL YOUR POSTER IN 2-5 DAYS=21

=B0 Family Pictures Can Become Attention Getting Wall Art
=B0 Large Sized Graphic Exhibits
=B0 Upscale Signage
=B0 Special Memories
=B0 Memorable Humorous Moments

20" x 25" - =24 24.50 24" x 30" - =24 35.00
20" x 28" - =24 27.50 29" x 32" - =24 45.00
22" x 28" - =24 30.00 54" x 96" - =24252.00
DISCOUNTS
(for min. size of 3 sq. ft.)
5 posters =3D <= strong>=246.00 per square foot
10 posters =3D <= strong>=245.50 per square foot
20 posters up =3D<= /strong> <= strong>=245.00 per square foot

Convert your favorite:
*Action Pictur= es
*Team Picture
*Class Picture
*Any Picture
To a G= IANT POSTER SIZED PRINT=21
FROM = THIS
3D=22IF		 TO TH= IS (or BIGGER=21)
3D=22IF

WHAT TO DO:
W= e are a family owned business operating in Memphis, Tennessee for over 29 years.
Please contact us via email:
Printed4U=40exc= ite.com

or call Toll Free: 1-877-503-7415
to place an order or get additional information.

= SIGNATURES
GIANT SIZED POSTERS= * CORPORATE APPAREL * SIGNS & BANNERS * PROMOTIONAL PRODUCTS * TROPHIES & AWARDS
559 S. Highland Street * Memphis, TN 38111
901-327-5456

Toll Fre= e: 1-877-503-7415

We are a family owned business operating in Memphis, Tennessee for over 29 years.

 

 

We honor the removal reque= sts that we receive.
To be removed from future mailings, please email us at: N0THANKS=40excite.com with the email address you= want removed in the Subjec= t Line of your email.= Our system checks the subject line for email addresses and will remove those addresses from our database.
We apologize for any inconvenience.

------=_NextPart_1E5_033D_ACE83D84.B177C78C-- From CustomPrintGoods@netscape.net Wed Jun 11 16:00:17 2003 Return-Path: Received: from netscape.net (58.Red-80-36-75.pooles.rima-tde.net [80.36.75.58]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id h5BJxvSP006150 for ; Wed, 11 Jun 2003 16:00:08 -0400 (EDT) Received: from [142.239.112.41] by mail.naihautsui.co.kr with esmtp; 10 Jun 2003 23:59:43 +0900 Received: from 142.91.161.245 ([142.91.161.245]) by qrx.quickslick.com with SMTP; 11 Jun 2003 08:56:18 -1000 Received: from [168.239.117.227] by relay-x.misswldrs.com with asmtp; 10 Jun 2003 22:52:53 +0900 Received: from 151.113.236.159 ([151.113.236.159]) by m1.gns.snv.thisdomainl.com with esmtp; 11 Jun 2003 07:49:28 +1000 Received: from unknown (90.197.173.144) by smtp.mixedthings.net with local; Wed, 11 Jun 2003 17:46:03 +0200 Message-ID: Reply-To: From: To: RECIPIENTS Subject: 3' x 8' Banner:$59.00 - includes delivery / Full Color Photo Banners / Coroplast Signs Date: Wed, 11 Jun 2003 16:37:25 -0300 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_5A6_9F93_BEB67C41.12F248B6" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook, Build 10.0.4510 X-MimeOLE: Produced By Microsoft MimeOLE V10.0.4510 This is a multi-part message in MIME format. ------=_NextPart_5A6_9F93_BEB67C41.12F248B6 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable ------=_NextPart_5A6_9F93_BEB67C41.12F248B6 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
&=23149;   &=23149;
  3' x 8' Banner - =2459.00
4 Lines of Text - White Background
(Color Background =2469.00)
Guaranteed= 5 Years Indoors or Outdoors		  
&=23149;   &=23149;

Shipping (2-5 days) to your front door is included in this price=21

3' = x 8' FULL COLOR
PHOTOGRAPHIC BANNER - =24216.00
We can prin= t any image, photo, drawing,
or logo onto high quality banner material.
FULL PHOTO or FULL PHOTO & TEXT
Guaranteed 5 Years Indoors or 6 Months Outdoors

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

Coroplast S= igns
18" x 24"
from
=244.75 each=21

To place an order or get additional information, please contact us:
CUSTOMPRINT= ED=40excite.com

OR CALL US TOLL FREE
1-877-503-7415

We are= a family owned business, operating in Memphis, Tennessee for over 29 years.

SIGNATURES
GIANT POSTERS *= CORPORATE APPAREL * SIGNS & BANNERS * PROMOTIONAL PRODUCTS * TROPHIES & AWARDS
= 559 S. Highland Street * Memphis, TN 38111
901-327-5456<= /p>

We honor the removal requests that we receive.
To be removed from future mailings, please email us at N0THANKS=40excite.c= om with your email= address (or the email address you want removed) typed into the Subject Line of your email. Our system checks the subject line for addresses and removes those address from our database. Thank you.
We apologize for any inconvenience.
------=_NextPart_5A6_9F93_BEB67C41.12F248B6-- From CustomPrintGoods@netscape.net Mon Jun 30 20:39:09 2003 Return-Path: Received: from 202.157.186.178 ([202.157.186.178]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with ESMTP id h610d7SP027439 for ; Mon, 30 Jun 2003 20:39:07 -0400 (EDT) Received: from smtp0432.mail.yahoo.com ([12.31.17.134]) by 202.157.186.178 ; Wed, 25 Jun 2003 17:12:33 +0800 Date: Wed, 25 Jun 2003 08:32:06 GMT From: CustomPrintGoods@netscape.net X-Priority: 3 To: mpi-comm-archive@netlib2.cs.utk.edu Subject: Giant Posters from YOUR Images: Just $7.00 a square foot Mime-Version: 1.0 Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: base64 X-Rcpt-To: Message-ID: <105653235707@202.157.186.178> PGh0bWw+DQoNCjxoZWFkPg0KPHRpdGxlPlBPU1RFUiBNQUlMT1VUPC90aXRsZT4NCjwvaGVhZD4N Cg0KPGJvZHkgYmdjb2xvcj0iI0ZGRkZGRiI+DQo8ZGl2IGFsaWduPSJjZW50ZXIiPjxjZW50ZXI+ DQoNCjx0YWJsZSBib3JkZXI9IjAiIGNlbGxwYWRkaW5nPSIwIiBjZWxsc3BhY2luZz0iMCIgd2lk dGg9IjEwMCUiPg0KICAgIDx0cj4NCiAgICAgICAgPHRkIGFsaWduPSJjZW50ZXIiIGNvbHNwYW49 IjIiPjxkaXYgYWxpZ249ImNlbnRlciI+PGNlbnRlcj48dGFibGUNCiAgICAgICAgYm9yZGVyPSIw IiBjZWxscGFkZGluZz0iNSIgY2VsbHNwYWNpbmc9IjAiPg0KICAgICAgICAgICAgPHRyPg0KDQog ICAgICAgICAgICAgICAgPHRkIGFsaWduPSJjZW50ZXIiIGNvbHNwYW49IjIiPjxmb250DQogICAg ICAgICAgICAgICAgY29sb3I9IiNGRjAwMDAiIHNpemU9IjQiIGZhY2U9IlZlcmRhbmEiPkdJQU5U DQogICAgICAgICAgICAgICAgU0laRUQgUE9TVEVSUyBtYWRlIGZyb20gWU9VUiBJTUFHRVMhPGJy Pg0KICAgICAgICAgICAgICAgIDwvZm9udD48Zm9udCBzaXplPSIyIiBmYWNlPSJWZXJkYW5hIj5T ZW5kIHVzIDxlbT5Bbnk8L2VtPg0KICAgICAgICAgICAgICAgIFBpY3R1cmUgb3IgRHJhd2luZyBv ciBBcnR3b3JrIC0gV2UnbGwgbWFrZSB5b3VyDQogICAgICAgICAgICAgICAgUG9zdGVyIGZvciAk Ny4wMCBhIHNxdWFyZSBmb290PC9mb250PjwvdGQ+DQogICAgICAgICAgICA8L3RyPg0KICAgICAg ICAgICAgPHRyPg0KICAgICAgICAgICAgICAgIDx0ZCBhbGlnbj0iY2VudGVyIj48ZGl2IGFsaWdu PSJjZW50ZXIiPjxjZW50ZXI+PHRhYmxlDQogICAgICAgICAgICAgICAgYm9yZGVyPSIyIiBjZWxs cGFkZGluZz0iMiIgY2VsbHNwYWNpbmc9IjAiDQogICAgICAgICAgICAgICAgYmdjb2xvcj0iI0ZG RkZBQSIgYm9yZGVyY29sb3I9IiNGRjAwMDAiPg0KICAgICAgICAgICAgICAgICAgICA8dHI+DQoN CiAgICAgICAgICAgICAgICAgICAgICAgIDx0ZCBhbGlnbj0iY2VudGVyIj48Zm9udCBjb2xvcj0i IzgwMDAwMCINCiAgICAgICAgICAgICAgICAgICAgICAgIHNpemU9IjUiIGZhY2U9IlZlcmRhbmEi PjIyJnF1b3Q7IHgNCiAgICAgICAgICAgICAgICAgICAgICAgIDI4JnF1b3Q7IHBvc3RlciBmb3Ig JDMwLjAwPGJyPg0KICAgICAgICAgICAgICAgICAgICAgICAgPC9mb250Pjxmb250IHNpemU9IjQi IGZhY2U9IlZlcmRhbmEiPkZyZWUNCiAgICAgICAgICAgICAgICAgICAgICAgIFNoaXBwaW5nIGZv ciBPcmRlcnMgT3ZlciAkODAuMDA8YnI+DQogICAgICAgICAgICAgICAgICAgICAgICA8L2ZvbnQ+ PGZvbnQgY29sb3I9IiMwMDAwODAiIHNpemU9IjMiDQogICAgICAgICAgICAgICAgICAgICAgICBm YWNlPSJWZXJkYW5hIj48c3Ryb25nPnNoaXBwZWQgaW4gMi01DQogICAgICAgICAgICAgICAgICAg ICAgICBkYXlzPC9zdHJvbmc+PC9mb250Pjxmb250IHNpemU9IjQiDQogICAgICAgICAgICAgICAg ICAgICAgICBmYWNlPSJWZXJkYW5hIj48c3Ryb25nPjxicj4NCiAgICAgICAgICAgICAgICAgICAg ICAgIDwvc3Ryb25nPjwvZm9udD48Zm9udCBzaXplPSIzIg0KICAgICAgICAgICAgICAgICAgICAg ICAgZmFjZT0iVmVyZGFuYSI+KHJvbGxlZCBpbiBhIHByb3RlY3RpdmUNCiAgICAgICAgICAgICAg ICAgICAgICAgIG1haWxpbmcgdHViZSk8L2ZvbnQ+PHA+PGZvbnQgc2l6ZT0iMyINCiAgICAgICAg ICAgICAgICAgICAgICAgIGZhY2U9IlZlcmRhbmEiPjxzdHJvbmc+TWluaW11bSBPcmRlcjoNCiAg ICAgICAgICAgICAgICAgICAgICAgICQzMC4wMDxicj4NCg0KICAgICAgICAgICAgICAgICAgICAg ICAgU2hpcHBpbmc6ICQ3LjUwPC9zdHJvbmc+PC9mb250PjwvcD4NCiAgICAgICAgICAgICAgICAg ICAgICAgIDwvdGQ+DQogICAgICAgICAgICAgICAgICAgIDwvdHI+DQogICAgICAgICAgICAgICAg PC90YWJsZT4NCiAgICAgICAgICAgICAgICA8L2NlbnRlcj48L2Rpdj48L3RkPg0KICAgICAgICAg ICAgICAgIDx0ZCBhbGlnbj0iY2VudGVyIj48Zm9udCBzaXplPSIyIiBmYWNlPSJWZXJkYW5hIj5K VVNUDQogICAgICAgICAgICAgICAgRU1BSUwgWU9VUiBTQ0FOTkVEIElNQUdFUyBPUiBBUlQgQU5E IFdFJ0xMIE1BSUwNCiAgICAgICAgICAgICAgICBZT1VSIFBPU1RFUiBJTiAyLTUgREFZUyE8L2Zv bnQ+PHA+PGZvbnQgc2l6ZT0iMiINCiAgICAgICAgICAgICAgICBmYWNlPSJWZXJkYW5hIj6wIEZh bWlseSBQaWN0dXJlcyBDYW4gQmVjb21lDQogICAgICAgICAgICAgICAgQXR0ZW50aW9uIEdldHRp bmcgV2FsbCBBcnQ8YnI+DQogICAgICAgICAgICAgICAgsCBMYXJnZSBTaXplZCBHcmFwaGljIEV4 aGliaXRzPGJyPg0KDQogICAgICAgICAgICAgICAgsCBVcHNjYWxlIFNpZ25hZ2U8YnI+DQogICAg ICAgICAgICAgICAgsCBTcGVjaWFsIE1lbW9yaWVzPGJyPg0KICAgICAgICAgICAgICAgILAgTWVt b3JhYmxlIEh1bW9yb3VzIE1vbWVudHM8L2ZvbnQ+PC9wPg0KICAgICAgICAgICAgICAgIDxkaXYg YWxpZ249ImNlbnRlciI+PGNlbnRlcj48dGFibGUgYm9yZGVyPSIxIg0KICAgICAgICAgICAgICAg IGNlbGxzcGFjaW5nPSIwIiBiZ2NvbG9yPSIjRjRGNEY0Ij4NCiAgICAgICAgICAgICAgICAgICAg PHRyPg0KICAgICAgICAgICAgICAgICAgICAgICAgPHRkIGFsaWduPSJjZW50ZXIiIHZhbGlnbj0i dG9wIj48Zm9udA0KICAgICAgICAgICAgICAgICAgICAgICAgc2l6ZT0iMiIgZmFjZT0iVmVyZGFu YSI+MjAmcXVvdDsgeA0KICAgICAgICAgICAgICAgICAgICAgICAgMjUmcXVvdDsgLSAkIDI0LjUw PC9mb250PjwvdGQ+DQoNCiAgICAgICAgICAgICAgICAgICAgICAgIDx0ZCBhbGlnbj0iY2VudGVy IiB2YWxpZ249InRvcCI+PGZvbnQNCiAgICAgICAgICAgICAgICAgICAgICAgIHNpemU9IjIiIGZh Y2U9IlZlcmRhbmEiPjI0JnF1b3Q7IHgNCiAgICAgICAgICAgICAgICAgICAgICAgIDMwJnF1b3Q7 IC0gJCAzNS4wMDwvZm9udD48L3RkPg0KICAgICAgICAgICAgICAgICAgICA8L3RyPg0KICAgICAg ICAgICAgICAgICAgICA8dHI+DQogICAgICAgICAgICAgICAgICAgICAgICA8dGQgYWxpZ249ImNl bnRlciIgdmFsaWduPSJ0b3AiPjxmb250DQogICAgICAgICAgICAgICAgICAgICAgICBzaXplPSIy IiBmYWNlPSJWZXJkYW5hIj4yMCZxdW90OyB4DQogICAgICAgICAgICAgICAgICAgICAgICAyOCZx dW90OyAtICQgMjcuNTA8L2ZvbnQ+PC90ZD4NCg0KICAgICAgICAgICAgICAgICAgICAgICAgPHRk IGFsaWduPSJjZW50ZXIiIHZhbGlnbj0idG9wIj48Zm9udA0KICAgICAgICAgICAgICAgICAgICAg ICAgc2l6ZT0iMiIgZmFjZT0iVmVyZGFuYSI+MjkmcXVvdDsgeA0KICAgICAgICAgICAgICAgICAg ICAgICAgMzImcXVvdDsgLSAkIDQ1LjAwPC9mb250PjwvdGQ+DQogICAgICAgICAgICAgICAgICAg IDwvdHI+DQogICAgICAgICAgICAgICAgICAgIDx0cj4NCiAgICAgICAgICAgICAgICAgICAgICAg IDx0ZCBhbGlnbj0iY2VudGVyIiB2YWxpZ249InRvcCI+PGZvbnQNCiAgICAgICAgICAgICAgICAg ICAgICAgIHNpemU9IjIiIGZhY2U9IlZlcmRhbmEiPjIyJnF1b3Q7IHgNCiAgICAgICAgICAgICAg ICAgICAgICAgIDI4JnF1b3Q7IC0gJCAzMC4wMDwvZm9udD48L3RkPg0KDQogICAgICAgICAgICAg ICAgICAgICAgICA8dGQgYWxpZ249ImNlbnRlciIgdmFsaWduPSJ0b3AiPjxmb250DQogICAgICAg ICAgICAgICAgICAgICAgICBzaXplPSIyIiBmYWNlPSJWZXJkYW5hIj41NCZxdW90OyB4DQogICAg ICAgICAgICAgICAgICAgICAgICA5NiZxdW90OyAtICQyNTIuMDA8L2ZvbnQ+PC90ZD4NCiAgICAg ICAgICAgICAgICAgICAgPC90cj4NCiAgICAgICAgICAgICAgICA8L3RhYmxlPg0KICAgICAgICAg ICAgICAgIDwvY2VudGVyPjwvZGl2PjwvdGQ+DQogICAgICAgICAgICA8L3RyPg0KICAgICAgICA8 L3RhYmxlPg0KDQogICAgICAgIDwvY2VudGVyPjwvZGl2PjwvdGQ+DQogICAgPC90cj4NCiAgICA8 dHI+DQogICAgICAgIDx0ZCBhbGlnbj0iY2VudGVyIiB2YWxpZ249InRvcCI+PGRpdiBhbGlnbj0i Y2VudGVyIj48Y2VudGVyPjx0YWJsZQ0KICAgICAgICBib3JkZXI9IjEiIGNlbGxwYWRkaW5nPSI0 IiBjZWxsc3BhY2luZz0iMCINCiAgICAgICAgYmdjb2xvcj0iI0RDRkZCOSI+DQogICAgICAgICAg ICA8dHI+DQogICAgICAgICAgICAgICAgPHRkIGFsaWduPSJjZW50ZXIiPjxmb250IGZhY2U9IlZl cmRhbmEiPjxzdHJvbmc+RElTQ09VTlRTPGJyPg0KICAgICAgICAgICAgICAgIChmb3IgbWluLiBz aXplIG9mIDMgc3EuIGZ0Lik8L3N0cm9uZz48L2ZvbnQ+PGRpdg0KICAgICAgICAgICAgICAgIGFs aWduPSJjZW50ZXIiPjxjZW50ZXI+PHRhYmxlIGJvcmRlcj0iMCINCiAgICAgICAgICAgICAgICBj ZWxscGFkZGluZz0iMiIgY2VsbHNwYWNpbmc9IjAiIGJnY29sb3I9IiNGRkZGOTciPg0KICAgICAg ICAgICAgICAgICAgICA8dHI+DQoNCiAgICAgICAgICAgICAgICAgICAgICAgIDx0ZCBhbGlnbj0i cmlnaHQiPjxmb250IHNpemU9IjIiDQogICAgICAgICAgICAgICAgICAgICAgICBmYWNlPSJWZXJk YW5hIj48c3Ryb25nPjUgcG9zdGVycyA9PC9zdHJvbmc+PC9mb250PjwvdGQ+DQogICAgICAgICAg ICAgICAgICAgICAgICA8dGQ+PGZvbnQgc2l6ZT0iMiIgZmFjZT0iVmVyZGFuYSI+PHN0cm9uZz4k Ni4wMA0KICAgICAgICAgICAgICAgICAgICAgICAgcGVyIHNxdWFyZSBmb290PC9zdHJvbmc+PC9m b250PjwvdGQ+DQogICAgICAgICAgICAgICAgICAgIDwvdHI+DQogICAgICAgICAgICAgICAgICAg IDx0cj4NCiAgICAgICAgICAgICAgICAgICAgICAgIDx0ZCBhbGlnbj0icmlnaHQiPjxmb250IHNp emU9IjIiDQogICAgICAgICAgICAgICAgICAgICAgICBmYWNlPSJWZXJkYW5hIj48c3Ryb25nPjEw IHBvc3RlcnMgPTwvc3Ryb25nPjwvZm9udD48L3RkPg0KICAgICAgICAgICAgICAgICAgICAgICAg PHRkPjxmb250IHNpemU9IjIiIGZhY2U9IlZlcmRhbmEiPjxzdHJvbmc+JDUuNTANCiAgICAgICAg ICAgICAgICAgICAgICAgIHBlciBzcXVhcmUgZm9vdDwvc3Ryb25nPjwvZm9udD48L3RkPg0KICAg ICAgICAgICAgICAgICAgICA8L3RyPg0KDQogICAgICAgICAgICAgICAgICAgIDx0cj4NCiAgICAg ICAgICAgICAgICAgICAgICAgIDx0ZCBhbGlnbj0icmlnaHQiPjxmb250IHNpemU9IjIiDQogICAg ICAgICAgICAgICAgICAgICAgICBmYWNlPSJWZXJkYW5hIj48c3Ryb25nPjIwIHBvc3RlcnMgdXAg PTwvc3Ryb25nPjwvZm9udD48L3RkPg0KICAgICAgICAgICAgICAgICAgICAgICAgPHRkPjxmb250 IHNpemU9IjIiIGZhY2U9IlZlcmRhbmEiPjxzdHJvbmc+JDUuMDANCiAgICAgICAgICAgICAgICAg ICAgICAgIHBlciBzcXVhcmUgZm9vdDwvc3Ryb25nPjwvZm9udD48L3RkPg0KICAgICAgICAgICAg ICAgICAgICA8L3RyPg0KICAgICAgICAgICAgICAgIDwvdGFibGU+DQogICAgICAgICAgICAgICAg PC9jZW50ZXI+PC9kaXY+PC90ZD4NCiAgICAgICAgICAgIDwvdHI+DQogICAgICAgIDwvdGFibGU+ DQoNCiAgICAgICAgPC9jZW50ZXI+PC9kaXY+PHAgYWxpZ249ImNlbnRlciI+PGZvbnQgc2l6ZT0i MyINCiAgICAgICAgZmFjZT0iVmVyZGFuYSI+Q29udmVydCB5b3VyIGZhdm9yaXRlOjxicj4NCiAg ICAgICAgPC9mb250Pjxmb250IHNpemU9IjIiIGZhY2U9IlZlcmRhbmEiPipBY3Rpb24gUGljdHVy ZXM8YnI+DQogICAgICAgICpUZWFtIFBpY3R1cmU8YnI+DQogICAgICAgICpDbGFzcyBQaWN0dXJl PGJyPg0KICAgICAgICAqQW55IFBpY3R1cmU8YnI+DQogICAgICAgIDwvZm9udD48Zm9udCBzaXpl PSIzIiBmYWNlPSJWZXJkYW5hIj48c3Ryb25nPlRvIGEgR0lBTlQNCiAgICAgICAgUE9TVEVSIFNJ WkVEIFBSSU5UITwvc3Ryb25nPjwvZm9udD48L3A+DQoNCiAgICAgICAgPC90ZD4NCiAgICAgICAg PHRkIHZhbGlnbj0iYm90dG9tIj48ZGl2IGFsaWduPSJjZW50ZXIiPjxjZW50ZXI+PHRhYmxlDQog ICAgICAgIGJvcmRlcj0iMCIgY2VsbHBhZGRpbmc9IjUiIGNlbGxzcGFjaW5nPSIwIj4NCiAgICAg ICAgICAgIDx0cj4NCiAgICAgICAgICAgICAgICA8dGQgdmFsaWduPSJ0b3AiPjxmb250IGZhY2U9 IlZlcmRhbmEiPkZST00gVEhJUzxicj4NCiAgICAgICAgICAgICAgICA8aW1nDQogICAgICAgICAg ICAgICAgc3JjPSJodHRwOi8vd3d3Lmdlb2NpdGllcy5jb20vemlwX3ppcF8xL3Bvc3RlcjEuanBn Ig0KICAgICAgICAgICAgICAgIGFsdD0iSUYgSU1BR0UgRkFJTFMgVE8gTE9BRCwgUExFQVNFIExF VCBVUyBLTk9XIg0KICAgICAgICAgICAgICAgIHdpZHRoPSIxNDEiIGhlaWdodD0iMTU5Ij48L2Zv bnQ+PC90ZD4NCiAgICAgICAgICAgICAgICA8dGQgdmFsaWduPSJ0b3AiPjxmb250IGZhY2U9IlZl cmRhbmEiPlRPIFRISVMgKG9yDQogICAgICAgICAgICAgICAgQklHR0VSISk8YnI+DQogICAgICAg ICAgICAgICAgPGltZw0KICAgICAgICAgICAgICAgIHNyYz0iaHR0cDovL3d3dy5nZW9jaXRpZXMu Y29tL2N1emJhay9wb3N0ZXIyLmpwZyINCiAgICAgICAgICAgICAgICBhbHQ9IklGIElNQUdFIEZB SUxTIFRPIExPQUQsIFBMRUFTRSBMRVQgVVMgS05PVyINCiAgICAgICAgICAgICAgICB3aWR0aD0i MTgyIiBoZWlnaHQ9IjIzNCI+PC9mb250PjwvdGQ+DQogICAgICAgICAgICA8L3RyPg0KDQogICAg ICAgIDwvdGFibGU+DQogICAgICAgIDwvY2VudGVyPjwvZGl2PjwvdGQ+DQogICAgPC90cj4NCjwv dGFibGU+DQo8L2NlbnRlcj48L2Rpdj48ZGl2IGFsaWduPSJjZW50ZXIiPjxjZW50ZXI+DQoNCjx0 YWJsZSBib3JkZXI9IjAiIGNlbGxwYWRkaW5nPSIwIiBjZWxsc3BhY2luZz0iMCIgd2lkdGg9IjEw MCUiDQpiZ2NvbG9yPSIjRkVFOUQzIj4NCiAgICA8dHI+DQogICAgICAgIDx0ZD48YmxvY2txdW90 ZT4NCiAgICAgICAgICAgIDxwIGFsaWduPSJjZW50ZXIiPjxmb250IHNpemU9IjQiIGZhY2U9IlZl cmRhbmEiPjxzdHJvbmc+V0hBVA0KICAgICAgICAgICAgVE8gRE86PGJyPg0KDQogICAgICAgICAg ICA8L3N0cm9uZz48L2ZvbnQ+PGZvbnQgc2l6ZT0iMiIgZmFjZT0iVmVyZGFuYSI+V2UgYXJlDQog ICAgICAgICAgICBhIGZhbWlseSBvd25lZCBidXNpbmVzcyBvcGVyYXRpbmcgaW4gTWVtcGhpcywN CiAgICAgICAgICAgIFRlbm5lc3NlZSBmb3Igb3ZlciAyOSB5ZWFycy48L2ZvbnQ+PGZvbnQgc2l6 ZT0iNCINCiAgICAgICAgICAgIGZhY2U9IlZlcmRhbmEiPjxzdHJvbmc+PGJyPg0KICAgICAgICAg ICAgPC9zdHJvbmc+UGxlYXNlIGNvbnRhY3QgdXMgdmlhIGVtYWlsOjxicj4NCiAgICAgICAgICAg IDwvZm9udD48YSBocmVmPSJtYWlsdG86U2hpcnRzU2lnbnNNb3JlQG5ldHNjYXBlLm5ldCI+PGZv bnQNCiAgICAgICAgICAgIHNpemU9IjUiIGZhY2U9IlZlcmRhbmEiPjxzdHJvbmc+U2hpcnRzU2ln bnNNb3JlQG5ldHNjYXBlLm5ldDwvc3Ryb25nPjwvZm9udD48L2E+PC9wPg0KICAgICAgICAgICAg PHAgYWxpZ249ImNlbnRlciI+PGZvbnQgc2l6ZT0iNCIgZmFjZT0iVmVyZGFuYSI+PHN0cm9uZz5v cg0KICAgICAgICAgICAgY2FsbCBUb2xsIEZyZWU6IDEtODc3LTUwMy03NDE1PGJyPg0KICAgICAg ICAgICAgPC9zdHJvbmc+dG8gcGxhY2UgYW4gb3JkZXIgb3IgZ2V0IGFkZGl0aW9uYWwNCiAgICAg ICAgICAgIGluZm9ybWF0aW9uLjwvZm9udD48L3A+DQogICAgICAgICAgICA8cCBhbGlnbj0iY2Vu dGVyIj48Zm9udCBmYWNlPSJWZXJkYW5hIj48c3Ryb25nPlNJR05BVFVSRVM8L3N0cm9uZz48L2Zv bnQ+PGJyPg0KDQogICAgICAgICAgICA8Zm9udCBzaXplPSIxIiBmYWNlPSJBcmlhbCI+R0lBTlQg U0laRUQgUE9TVEVSUyAqDQogICAgICAgICAgICBDT1JQT1JBVEUgQVBQQVJFTCAqIFNJR05TICZh bXA7IEJBTk5FUlMgKiBQUk9NT1RJT05BTA0KICAgICAgICAgICAgUFJPRFVDVFMgKiBUUk9QSElF UyAmYW1wOyBBV0FSRFM8L2ZvbnQ+PGZvbnQgc2l6ZT0iMSI+PGJyPg0KICAgICAgICAgICAgPC9m b250Pjxmb250IGZhY2U9IkFyaWFsLEhlbHZldGljYSI+NTU5IFMuIEhpZ2hsYW5kDQogICAgICAg ICAgICBTdHJlZXQgKiBNZW1waGlzLCBUTiAzODExMTwvZm9udD4gPGJyPg0KICAgICAgICAgICAg PGZvbnQgZmFjZT0iQXJpYWwsSGVsdmV0aWNhIj45MDEtMzI3LTU0NTY8L2ZvbnQ+PC9wPg0KICAg ICAgICAgICAgPHAgYWxpZ249ImNlbnRlciI+PGZvbnQgZmFjZT0iVmVyZGFuYSI+VG9sbCBGcmVl Og0KICAgICAgICAgICAgMS04NzctNTAzLTc0MTU8L2ZvbnQ+PC9wPg0KDQogICAgICAgICAgICA8 cCBhbGlnbj0iY2VudGVyIj48Zm9udCBzaXplPSIyIiBmYWNlPSJWZXJkYW5hIj5XZQ0KICAgICAg ICAgICAgYXJlIGEgZmFtaWx5IG93bmVkIGJ1c2luZXNzIG9wZXJhdGluZyBpbiBNZW1waGlzLA0K ICAgICAgICAgICAgVGVubmVzc2VlIGZvciBvdmVyIDI5IHllYXJzLjwvZm9udD48L3A+DQogICAg ICAgIDwvYmxvY2txdW90ZT4NCiAgICAgICAgPC90ZD4NCiAgICA8L3RyPg0KPC90YWJsZT4NCjwv Y2VudGVyPjwvZGl2Pg0KDQo8cD4mbmJzcDs8L3A+DQoNCjxwPiZuYnNwOzwvcD4NCg0KPHA+PGZv bnQgc2l6ZT0iMiIgZmFjZT0iVmVyZGFuYSI+V2UgaG9ub3IgdGhlIHJlbW92YWwgcmVxdWVzdHMN CnRoYXQgd2UgcmVjZWl2ZS48YnI+DQoNClRvIGJlIHJlbW92ZWQgZnJvbSBmdXR1cmUgbWFpbGlu Z3MsIHBsZWFzZSBlbWFpbCB1cyBhdCA8L2ZvbnQ+PGENCmhyZWY9Im1haWx0bzpJQW1EaXNpbnRl cmVzdGVkQG5ldHNjYXBlLm5ldCI+PGZvbnQgc2l6ZT0iMiINCmZhY2U9IlZlcmRhbmEiPjxzdHJv bmc+SUFtRGlzaW50ZXJlc3RlZEBuZXRzY2FwZS5uZXQ8L3N0cm9uZz48L2ZvbnQ+PC9hPjxmb250 DQpzaXplPSIyIiBmYWNlPSJWZXJkYW5hIj48c3Ryb25nPiB3aXRoIHRoZSBlbWFpbCBhZGRyZXNz IHlvdSB3YW50DQpyZW1vdmVkIDwvc3Ryb25nPjxlbT48c3Ryb25nPmluIHRoZSA8L3N0cm9uZz48 L2VtPjxlbT48c3Ryb25nPjx1PlN1YmplY3QNCkxpbmU8L3U+PC9zdHJvbmc+PC9lbT48ZW0+PHN0 cm9uZz4gb2YgeW91ciBlbWFpbDwvc3Ryb25nPjwvZW0+PHN0cm9uZz4uPC9zdHJvbmc+DQpPdXIg c3lzdGVtIGNoZWNrcyB0aGUgc3ViamVjdCBsaW5lIGZvciBlbWFpbCBhZGRyZXNzZXMgYW5kIHdp bGwNCnJlbW92ZSB0aG9zZSBhZGRyZXNzZXMgZnJvbSBvdXIgZGF0YWJhc2UuPGJyPg0KV2UgYXBv bG9naXplIGZvciBhbnkgaW5jb252ZW5pZW5jZS48L2ZvbnQ+PC9wPg0KPC9ib2R5Pg0KPC9odG1s Pg== From CustomPrintGoods@netscape.net Wed Jul 9 14:13:32 2003 Return-Path: Received: from netscape.net (cs177030.pp.htv.fi [213.243.177.30]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id h69IDMSP020471 for ; Wed, 9 Jul 2003 14:13:25 -0400 (EDT) Message-ID: <7b3c01c34639$c9b7ec20$35739420@ehreejehmfik> Reply-To: CustomPrintGoods@netscape.net From: CustomPrintGoods@netscape.net To: "email@email" Subject: 3' x 8' banner $59 - 5 yr. Guarantee Date: Wed, 09 Jul 2003 06:47:30 -1000 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_92E_F3B2_4E8CE680.A9ADFBA3" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook, Build 10.0.4510 X-MimeOLE: Produced By Microsoft MimeOLE V10.0.4510 This is a multi-part message in MIME format. ------=_NextPart_92E_F3B2_4E8CE680.A9ADFBA3 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable ------=_NextPart_92E_F3B2_4E8CE680.A9ADFBA3 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
•   •
  3' x 8' banner $59 - 5 yr. Guarantee
4 Lines of Text - White Background
(Color Background $85.00)
Guaranteed 5 Years Indoors or Outdoors 		 
•   •

Shipping (2-5 days) to your front door is included in this price!
(Continental U.S. Only)

3' x 8' FULL COLOR
PHOTOGRAPHIC BANNER - $150.00
3D"An<= strong>
We can print any image, photo, drawing,
or logo onto high quality banner material.
FULL PHOTO or FULL PHOTO & TEXT
Guaran= teed 5 Years Indoors or 6 Months Outdoors

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

Coroplast Signs
= 18&quo= t; x 24" from $4.75 each! (Min 100)
50 =3D $5.75

To place an order or get additional information, please contact us:
ShirtsSignsMore@net= scape.net

<= strong>OR CALL US TOLL FREE
1-877= -503-7415

We are a family owned business, operating in Memphis, Tennessee for over 29 years.


= SIGNATURES
GIANT POSTERS * CORPORATE APPAREL * SIGNS & BANNERS * PROMOTIONAL PRODUCTS * TROPHIES & AWARDS
559 S. Highland Street * Memphis, TN 38111
901-327-5456

We honor the removal requests that we receive. To be removed from future mailings, please email us at IAmDisinterested@netscape.n= et with your email address (or the email address you want removed) typed into the Subject Line of your email. Our system checks the subject line for addresses and removes those addresses. Please allow 48 hours for your address to be removed from our data base. Thank you.
We apologize for any inconvenience.
------=_NextPart_92E_F3B2_4E8CE680.A9ADFBA3-- From CustomPrintGoods@netscape.net Wed Jul 9 14:38:08 2003 Return-Path: Received: from netscape.net ([213.231.82.15]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id h69IbiSP021465 for ; Wed, 9 Jul 2003 14:37:51 -0400 (EDT) Message-ID: <62eb01c34645$97159e50$c7972bbd@sff> Reply-To: CustomPrintGoods@netscape.net From: CustomPrintGoods@netscape.net To: "email@email" Subject: 3' x 8' banner $59 - 5 yr. Guarantee Date: Wed, 09 Jul 2003 14:11:59 -0400 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_C34_02E0_79489A34.282FB701" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2314.1300 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300 This is a multi-part message in MIME format. ------=_NextPart_C34_02E0_79489A34.282FB701 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable ------=_NextPart_C34_02E0_79489A34.282FB701 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
•   •
  3' x 8' banner $59 - 5 yr. Guarantee
4 Lines of Text - White Background
(Color Background $85.00)
Guaranteed 5 Years Indoors or Outdoors 		 
•   •

Shipping (2-5 days) to your front door is included in this price!
(Continental U.S. Only)

3' x 8' FULL COLOR
PHOTOGRAPHIC BANNER - $150.00
3D"An<= strong>
We can print any image, photo, drawing,
or logo onto high quality banner material.
FULL PHOTO or FULL PHOTO & TEXT
Guaran= teed 5 Years Indoors or 6 Months Outdoors

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

Coroplast Signs
= 18&quo= t; x 24" from $4.75 each! (Min 100)
50 =3D $5.75

To place an order or get additional information, please contact us:
ShirtsSignsMore@net= scape.net

<= strong>OR CALL US TOLL FREE
1-877= -503-7415

We are a family owned business, operating in Memphis, Tennessee for over 29 years.


= SIGNATURES
GIANT POSTERS * CORPORATE APPAREL * SIGNS & BANNERS * PROMOTIONAL PRODUCTS * TROPHIES & AWARDS
559 S. Highland Street * Memphis, TN 38111
901-327-5456

We honor the removal requests that we receive. To be removed from future mailings, please email us at IAmDisinterested@netscape.n= et with your email address (or the email address you want removed) typed into the Subject Line of your email. Our system checks the subject line for addresses and removes those addresses. Please allow 48 hours for your address to be removed from our data base. Thank you.
We apologize for any inconvenience.
------=_NextPart_C34_02E0_79489A34.282FB701-- From CustomPrintGoods@netscape.net Fri Jul 11 18:29:58 2003 Return-Path: Received: from netscape.net ([213.231.82.15]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id h6BMTcSP014341 for ; Fri, 11 Jul 2003 18:29:47 -0400 (EDT) Message-ID: Reply-To: CustomPrintGoods@netscape.net From: CustomPrintGoods@netscape.net To: "email@email" Subject: POSTERS from YOUR PICTURES - $30 Date: Fri, 11 Jul 2003 23:12:05 +0100 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_950_D6CD_BC0552A0.0FC5CCF5" X-Priority: 3 User-Agent: AOL 7.0 for Windows US sub 118 ------=_NextPart_950_D6CD_BC0552A0.0FC5CCF5 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable ------=_NextPart_950_D6CD_BC0552A0.0FC5CCF5 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
=
POSTERS from YOUR PICTURES - $30
We are a family owned business in Memphis, TN .... since 1974
OUR SPECIALTY IS COMMERCIAL WORK
S= end us Any Picture or Drawing or Artwork - We'll make your Poster for $7.00 a square foot
22" x 28" poster for $30.00
Free Shipping for Orders Over $80.00
(to the Continental United States only)
shipped in 2-5 days
(rolled in a protective mailing tube)

Minimum Order: $30.00
Shipping: $7.50
(to the Continental United States)
= JUST EMAIL YOUR SCANNED IMAGES OR ART AND WE'LL MAIL YOUR POSTER IN 2-5 DAYS!

=B0 Family Pictures Can Become Attention Getting Wall Art
=B0 Large Sized Graphic Exhibits
=B0 Upscale Signage
=B0 Special Memories
=B0 Memorable Humorous Moments

20" x 25" - $ 24.50 24" x 30" - $ 35.00
20" x 28" - $ 27.50 29" x 32" - $ 45.00
22" x 28" - $ 30.00 54" x 96" - $252.00
DIS= COUNTS
(for min. size of 3 sq. ft.)
5 posters =3D= $= 6.00 per square foot
10 posters =3D $= 5.50 per square foot
20 posters up =3D $= 5.00 per square foot

Convert your favorite:
*Action Pictures
*Team Picture
*Class Picture
*Any Picture
To a GIANT POSTER SIZED PRINT!

FROM THIS
= 3D"BEFORE		 TO THIS (or BIGGER!)
3D"POSTER

WHAT TO DO:
We are a family owned business operating in Memphis, Tennessee for over 29 years.
Please contact us via email:
ShirtsSignsMore@netscap= e.net

or call Toll Free: 1-877-503-7415
to place an order or get additional information.

SIGNATUR= ES
GIANT SIZED POSTERS * CORPORATE APPAREL * SIGNS & BANNERS * PROMOTIONAL PRODUCTS * TROPHIES & AWARDS 559 S. Highland Street * Memphis, TN 38111
901-327-5456

Toll Free: 1-877-503-7415

We are a family owned business operating in Memphis, Tennessee for over 29 years.

 

 

We honor the removal requests that we receive.
To be removed from future mailings, please email us at IAmDisinterested@netscape.net<= /a> with the email address you want removed in the Subjec= t Line of your email.= Our system checks the subject line for email addresses and will remove those addresses. Please allow 48 hours for your address to be removed from our database.
We apologize for any inconvenience.

------=_NextPart_950_D6CD_BC0552A0.0FC5CCF5-- From customprintgoods@netscape.net Wed Aug 13 14:02:50 2003 Return-Path: Received: from netscape.net (93.Red-80-35-120.pooles.rima-tde.net [80.35.120.93]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id h7DI2V8b027376 for ; Wed, 13 Aug 2003 14:02:44 -0400 (EDT) Message-ID: <4b3a01c361b8$7e3c1790$4137e4b7@kiark> Reply-To: customprintgoods@netscape.net From: customprintgoods@netscape.net To: "email@email" Subject: T-SHIRTS $2.10 - WITH IMPRINT Date: Wed, 13 Aug 2003 18:32:31 +0200 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_7D6_767C_8EC48218.2373C81A" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2314.1300 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300 This is a multi-part message in MIME format. ------=_NextPart_7D6_767C_8EC48218.2373C81A Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable ------=_NextPart_7D6_767C_8EC48218.2373C81A Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

T-SHIRTS $2.10 - WITH IMPRINT

=
Golf Shirts= T-Shirts
EMBROIDERED
Price= s Include Your Company Logo		 screen printed
with your logo
$10.95$2.10
  • Hanes Pique Knit Golf Shirts
  • 7 oz.
  • 16 Colors to choose from
  • Min. 24
  • Contact Us for Digitizing
Cutter & Buck Golf Shirts - $24.95
Min. 36
  • 500 - $2.10
  • 300 - $2.25
  • 200 - $2.60
  • 100 - $3.15
  • Minimum Order of 50 T-Shirts
    available at additional charge
  • ONE FREE SCREEN
  • Prices include your 2 COLOR IMPRINT
  • Hanes White Heavyweight
    100% Cotton T-Shirt

FREE SCREEN

To place an order or get additional information, please contact us:
ShirtsSignsMore@netscape.net

OR CALL US TOLL FREE
1-877-503-7415=

We are a family owned business, operating in Memphis, Tennessee for over 29 years.

=


SIGNATURES
GIANT SIZED POSTERS * CORPORATE APPAREL * SIGNS & BANNERS * PROMOTIONAL PRODUCTS * TROPHIES & AWARDS
559 S. Highland Street * Memphis, TN 38111
901-327-5456

 

 

We honor the removal requests that we receive.
To be removed from future mailings, please email us at IAmDisinterested@netscape.net with your email address (or the email address you want removed) typed in the Subj= ect Line of your email. Our system checks the subject line for addresses and removes them from our database. Please allow 48 hours for your address to be removed. We apologize for any inconvenience.

------=_NextPart_7D6_767C_8EC48218.2373C81A-- From customprintgoods@netscape.net Thu Sep 18 14:51:06 2003 Return-Path: Received: from netscape.net ([200.228.94.4]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id h8IIoxRY014573 for ; Thu, 18 Sep 2003 14:51:02 -0400 (EDT) Message-ID: <31ad01c37e0f$17ed6930$dc8c2d85@w> Reply-To: customprintgoods@netscape.net From: customprintgoods@netscape.net To: "email@email" Subject: T-SHIRTS $2.10 - WITH IMPRINT Date: Thu, 18 Sep 2003 20:02:58 +0200 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_74A_E52A_E1C56DE3.99E75B73" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4133.2400 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 This is a multi-part message in MIME format. ------=_NextPart_74A_E52A_E1C56DE3.99E75B73 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable ------=_NextPart_74A_E52A_E1C56DE3.99E75B73 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

T-SHIRTS $2.10 - WITH IMPRINT
We are a family owned business in Memphis, TN .... since 1974

=
Golf Shirts= T-Shirts
EMBROIDERED
Price= s Include Your Company Logo		 screen printed
with your logo
$10.95$2.10
  • Hanes Pique Knit Golf Shirts
  • 7 oz.
  • 16 Colors to choose from
  • Min. 24
  • Contact Us for Digitizing
Cutter & Buck Golf Shirts - $24.95
Min. 36
  • 500 - $2.10
  • 300 - $2.25
  • 200 - $2.60
  • 100 - $3.15
  • Minimum Order of 50 T-Shirts
    available at additional charge
  • ONE FREE SCREEN
  • Prices include your 2 COLOR IMPRINT
  • Hanes White Heavyweight
    100% Cotton T-Shirt

FREE SCREEN

To place an order or get additional information, please contact us:
ShirtsSignsMore@netscape.net

OR CALL US TOLL FREE
1-877-503-7415=

We are a family owned business, operating in Memphis, Tennessee for over 29 years.

=


SIGNATURES
GIANT SIZED POSTERS * CORPORATE APPAREL * SIGNS & BANNERS * PROMOTIONAL PRODUCTS * TROPHIES & AWARDS
559 S. Highland Street * Memphis, TN 38111
901-327-5456

 

 

We honor the removal requests that we receive.
To be removed from future mailings, please email us at IAmDisinterested@netscape.net with your email address (or the email address you want removed) typed in the Subj= ect Line of your email. Our system checks the subject line for addresses and removes them from our database. Please allow 48 hours for your address to be removed. We apologize for any inconvenience.

------=_NextPart_74A_E52A_E1C56DE3.99E75B73-- From Promoter@2911.net Mon Sep 22 18:31:05 2003 Return-Path: Received: from 2911.net ([218.86.51.27]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id h8MMV2RY028741 for ; Mon, 22 Sep 2003 18:31:04 -0400 (EDT) Message-Id: <200309222231.h8MMV2RY028741@netlib2.cs.utk.edu> From: "Betty" To: Subject: Promoting Your Business Sender: "Betty" Mime-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-1" Date: Tue, 23 Sep 2003 06:18:39 +0800 Reply-To: "Betty" Content-Transfer-Encoding: 8bit Email Marketing is one of the most effective and inexpensive ways to promote your products and services. We offer a complete Email Marketing solution with quality service and the lowest prices. The result is that you will enjoy more success. 1. Targeted Email Addresses We can supply targeted email addresses according to your requirements, which are compiled only on your order. We will customize your customer email addresses. * We have millions of email addresses in a wide variety of categories. 2. Targeted Mailing If you are worried about any complications or consequences with sending out targeted emails, or want to avoid the work of sending out targeted emails. We will do it for you! We can send your email message to your targeted customers. * We can Bullet-Proof your Web Site. We also offer a wide variety of marketing software. For more details, you can refer to: www.marketingboosting.com Our services will help you get more business opportunities. Regards! Betty Jones Customer Support www.marketingboosting.com Sales@marketingboosting.com ************************************************************************ Receiving this email because you registered to receive special offers from one of our partners. If you would prefer not to receive future email, click: Http://optin.garyshawkey.com/optin/r.php3 ************************************************************************ From customprintgoods@netscape.net Fri Sep 26 21:29:36 2003 Return-Path: Received: from netscape.net (MG021150.user.veloxzone.com.br [200.165.21.150]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id h8R1SYRY024921 for ; Fri, 26 Sep 2003 21:28:53 -0400 (EDT) Message-ID: <17ae01c38486$4491dd40$917d5929@h> Reply-To: customprintgoods@netscape.net From: customprintgoods@netscape.net To: "email@email.com" Subject: POSTERS from YOUR PICTURES - $30 Date: Fri, 26 Sep 2003 22:31:10 -0100 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_529_2B2C_8D7B1D08.744D5B2B" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2919.6700 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 This is a multi-part message in MIME format. ------=_NextPart_529_2B2C_8D7B1D08.744D5B2B Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable ------=_NextPart_529_2B2C_8D7B1D08.744D5B2B Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
=
POSTERS from YOUR PICTURES- $30
We are a family owned business in Memphis, TN .... since 1974
OUR SPECIALTY IS COMMERCIAL WORK
S= end us Any Picture or Drawing or Artwork - We'll make your Poster for $7.00 a square foot
22" x 28" poster for $30.00
Free Shipping for Orders Over $80.00
(to the Continental United States only)
shipped in 2-5 days
(rolled in a protective mailing tube)

Minimum Order: $30.00
Shipping: $7.50
(to the Continental United States)
= JUST EMAIL YOUR SCANNED IMAGES OR ART AND WE'LL MAIL YOUR POSTER IN 2-5 DAYS!

=B0 Family Pictures Can Become Attention Getting Wall Art
=B0 Large Sized Graphic Exhibits
=B0 Upscale Signage
=B0 Special Memories
=B0 Memorable Humorous Moments

20" x 25" - $ 24.50 24" x 30" - $ 35.00
20" x 28" - $ 27.50 29" x 32" - $ 45.00
22" x 28" - $ 30.00 54" x 96" - $252.00

LAMINATION: Just $1.50 Per Square Foot
$10 Minimum
DIS= COUNTS
(for min. size of 3 sq. ft.)
5 posters =3D= $= 6.00 per square foot
10 posters =3D $= 5.50 per square foot
20 posters up =3D $= 5.00 per square foot

Convert your favorite:
*Action Pictures
*Team Picture
*Class Picture
*Any Picture
To a GIANT POSTER SIZED PRINT!

FROM THIS
= 3D"BEFORE		 TO THIS (or BIGGER!)
3D"POSTER

WHAT TO DO:
We are a family owned business operating in Memphis, Tennessee for over 29 years.
Please contact us via email:
ShirtsSignsMore@netscape.net

or call Toll Free: 1-877-503-7415
to place an order or get additional information.


3D"SIGNATURES
SIGNATURES
GIANT SIZED POSTERS * CORPORATE APPAREL * SIGNS & BANNERS * PROMOTIONAL PRODUCTS * TROPHIES & AWARDS
559 S. Highland Street * Memphis, TN 38111
901-327-5456

Toll Free: 1-877-503-7415

We are a family owned business operating in Memphis, Tennessee for over 29 years.

 

We honor the removal requests that we receive.
To be removed from future mailings, please email us at IAmDisinterested@netscape.net<= /a> with your email address (or the email address you want removed) in the Subject Line of your email.= Our system checks the subject line for email addresses and will remove those addresses from our database. Please allow up to 48 hours for your email address to be successfully removed.
We apologize for any inconvenience.

------=_NextPart_529_2B2C_8D7B1D08.744D5B2B-- From customprintgoods@netscape.net Fri Sep 26 22:29:09 2003 Return-Path: Received: from netscape.net ([61.98.102.85]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id h8R2T4RY025516 for ; Fri, 26 Sep 2003 22:29:06 -0400 (EDT) Message-ID: <2e5001c38491$4e9fb680$cf78e074@grigntbmkah> Reply-To: customprintgoods@netscape.net From: customprintgoods@netscape.net To: "email@email" Subject: T-SHIRTS $2.10 - WITH IMPRINT Date: Sat, 27 Sep 2003 00:50:12 +0000 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_5F0_4596_FAD39018.54B0652C" X-Priority: 3 User-Agent: AOL 7.0 for Windows US sub 118 ------=_NextPart_5F0_4596_FAD39018.54B0652C Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable ------=_NextPart_5F0_4596_FAD39018.54B0652C Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

T-SHIRTS $2.10 - WITH IMPRINT
We are a family owned business in Memphis, TN .... since 1974

=
Golf Shirts= T-Shirts
EMBROIDERED
Price= s Include Your Company Logo		 screen printed
with your logo
$10.95$2.10
  • Hanes Pique Knit Golf Shirts
  • 7 oz.
  • 16 Colors to choose from
  • Min. 24
  • Contact Us for Digitizing
Cutter & Buck Golf Shirts - $24.95
Min. 36
  • 500 - $2.10
  • 300 - $2.25
  • 200 - $2.60
  • 100 - $3.15
  • Minimum Order of 50 T-Shirts
    available at additional charge
  • ONE FREE SCREEN
  • Prices include your 2 COLOR IMPRINT
  • Hanes White Heavyweight
    100% Cotton T-Shirt

FREE SCREEN

OR CALL US TOLL FREE
1-877-503-7415=

We are a family owned business, operating in Memphis, Tennessee for over 29 years.

=


SIGNATURES
GIANT SIZED POSTERS * CORPORATE APPAREL * SIGNS & BANNERS * PROMOTIONAL PRODUCTS * TROPHIES & AWARDS
559 S. Highland Street * Memphis, TN 38111
901-327-5456

 

 

We honor the removal requests that we receive.
To be removed from future mailings, please email us at IAmDisinterested@netscape.net with your email address (or the email address you want removed) typed in the Subj= ect Line of your email. Our system checks the subject line for addresses and removes them from our database. Please allow 48 hours for your address to be removed. We apologize for any inconvenience.

------=_NextPart_5F0_4596_FAD39018.54B0652C-- From customprintgoods@netscape.net Wed Oct 8 14:25:19 2003 Return-Path: Received: from netscape.net ([61.99.4.171]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id h98IPERY006520 for ; Wed, 8 Oct 2003 14:25:16 -0400 (EDT) Message-ID: Reply-To: customprintgoods@netscape.net From: customprintgoods@netscape.net To: "Advertising" Subject: BANNERS...3X8.....$59....14 oz. vinyl material Date: Wed, 08 Oct 2003 10:42:52 -0600 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_9CE_7B6C_1F92C38D.D04567BD" X-Priority: 3 User-Agent: AOL 7.0 for Windows US sub 118 ------=_NextPart_9CE_7B6C_1F92C38D.D04567BD Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable ------=_NextPart_9CE_7B6C_1F92C38D.D04567BD Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Signs & Banners
We are a family owned business in Memphis, TN ... since 1974
3' x 8' FULL COLOR
PHOT= OGRAPHIC BANNER - $150.00
3D"FULL We can print any image, photo, drawing,
or logo onto high quality banner material.
FULL PHOTO or FULL PHOTO & TEXT
Guaranteed 5= Years Indoors or 3 Years Outdoors

=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D

&= #8226;   &= #8226;
  3' x 8' banner $59 - 5 yr. Guarantee
4 Lines of Text - White Background
= 14 oz. vinyl material
(Color Background $85.00)
Guaranteed 5 Years Indoors or Outdoors<= /strong>		  
&= #8226;   &= #8226;

Shipp= ing (2-5 days) to your front door is included in this price!
(Continental U.S. = Only)

Coroplast Signs
18" x 24" fr= om $4.75 each! (Min 100)
50 =3D $5.75

To pl= ace an order or get additional information, please contact us:
BannersSignsMore@netscape.net

OR CALL US TOLL FREE
1-877-503-7415

We are a family owned business, operating in Memphis, Tennessee for over 29 years.
3D"Signatures
SIGNATURES
GIANT POSTERS * CORPORATE APPAREL * SIGNS & BANNERS * PROMOTIONAL PRODUCTS * TROPHIES & AWARDS
559 S. Highland Street * Memphis, TN 38111
901-327-5456

We hono= r the removal requests that we receive.
To be removed from future mailings, please email us at IAmDisinterested@netscape.net with your email address (or the email address you= want removed) typed into the Subject Line of your email. Our system checks the subject line for addresses and removes those address from our database. Please allow up to 48 hours for your removal request to be processed. Thank you.
We apologize for any inconvenience.
------=_NextPart_9CE_7B6C_1F92C38D.D04567BD-- From customprintgoods@netscape.net Wed Oct 8 17:06:58 2003 Return-Path: Received: from netscape.net (ogzc@[61.96.202.190]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id h98L6sRY010855 for ; Wed, 8 Oct 2003 17:06:56 -0400 (EDT) Message-ID: <75ed01c38dd1$627434d0$b81966d5@tvxbkhx> Reply-To: customprintgoods@netscape.net From: customprintgoods@netscape.net To: "Purchasing" Subject: Full Color Banners...$150....3x8BANNERS...$59.....3x8......14 oz. material Date: Wed, 08 Oct 2003 14:21:33 -0500 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_4DE_08C7_DE1172F9.63C690E7" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2462.0000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2462.0000 This is a multi-part message in MIME format. ------=_NextPart_4DE_08C7_DE1172F9.63C690E7 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable ------=_NextPart_4DE_08C7_DE1172F9.63C690E7 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Signs & Banners
We are a family owned business in Memphis, TN ... since 1974
3' x 8' FULL COLOR
PHOT= OGRAPHIC BANNER - $150.00
3D"FULL We can print any image, photo, drawing,
or logo onto high quality banner material.
FULL PHOTO or FULL PHOTO & TEXT
Guaranteed 5= Years Indoors or 3 Years Outdoors

=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D

&= #8226;   &= #8226;
  3' x 8' banner $59 - 5 yr. Guarantee
4 Lines of Text - White Background
= 14 oz. vinyl material
(Color Background $85.00)
Guaranteed 5 Years Indoors or Outdoors<= /strong>		  
&= #8226;   &= #8226;

Shipp= ing (2-5 days) to your front door is included in this price!
(Continental U.S. = Only)

Coroplast Signs
18" x 24" fr= om $4.75 each! (Min 100)
50 =3D $5.75

To pl= ace an order or get additional information, please contact us:
BannersSignsMore@netscape.net

OR CALL US TOLL FREE
1-877-503-7415

We are a family owned business, operating in Memphis, Tennessee for over 29 years.
3D"Signatures
SIGNATURES
GIANT POSTERS * CORPORATE APPAREL * SIGNS & BANNERS * PROMOTIONAL PRODUCTS * TROPHIES & AWARDS
559 S. Highland Street * Memphis, TN 38111
901-327-5456

We hono= r the removal requests that we receive.
To be removed from future mailings, please email us at IAmDisinterested@netscape.net with your email address (or the email address you= want removed) typed into the Subject Line of your email. Our system checks the subject line for addresses and removes those address from our database. Please allow up to 48 hours for your removal request to be processed. Thank you.
We apologize for any inconvenience.
------=_NextPart_4DE_08C7_DE1172F9.63C690E7-- From promoplacement@excite.com Thu Oct 30 03:32:16 2003 Return-Path: Received: from nsamail.oslo.nsa.no (nsamail.nsa.no [193.71.12.37]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with ESMTP id h9U8WBRY028448 for ; Thu, 30 Oct 2003 03:32:12 -0500 (EST) Received: from smtp0291.mail.yahoo.com ([200.157.208.11] unverified) by nsamail.oslo.nsa.no with Microsoft SMTPSVC(5.0.2195.6713); Thu, 30 Oct 2003 03:50:45 +0100 Date: Thu, 30 Oct 2003 02:54:57 GMT From: promoplacement@excite.com X-Priority: 3 To: mpi-comm-archive@netlib2.cs.utk.edu Subject: adv:3' x 8' Banner - $59 - Guaranteed Five Years Mime-Version: 1.0 Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: X-OriginalArrivalTime: 30 Oct 2003 02:50:48.0171 (UTC) FILETIME=[9F4E8BB0:01C39E90]
POSTERS from YOUR PICTURES- $30
We are a family owned business in Memphis, TN .... since 1974
OUR SPECIALTY IS COMMERCIAL WORK
Send us Any Picture or Drawing or Artwork - We'll make your Poster for $7.00 a square foot
22" x 28" poster for $30.00
Free Shipping for Orders Over $80.00
(to the Continental United States only)
shipped in 2-5 days
(rolled in a protective mailing tube)

Minimum Order: $30.00
Shipping: $7.50
(to the Continental United States)
JUST EMAIL YOUR SCANNED IMAGES OR ART AND WE'LL MAIL YOUR POSTER IN 2-5 DAYS!

° Family Pictures Can Become Attention Getting Wall Art
° Large Sized Graphic Exhibits
° Upscale Signage
° Special Memories
° Memorable Humorous Moments

20" x 25" - $ 24.50 24" x 30" - $ 35.00
20" x 28" - $ 27.50 29" x 32" - $ 45.00
22" x 28" - $ 30.00 54" x 96" - $252.00

LAMINATION: Just $1.50 Per Square Foot
$10 Minimum
DISCOUNTS
(for min. size of 3 sq. ft.)
5 posters = $6.00 per square foot
10 posters = $5.50 per square foot
20 posters up = $5.00 per square foot

Convert your favorite:
*Action Pictures
*Team Picture
*Class Picture
*Any Picture
To a GIANT POSTER SIZED PRINT!
FROM THIS
EXAMPLE PICTURE - BEFORE - if image doesn't load, please try refresh		 TO THIS (or BIGGER!)
EXAMPLE PICTURE - AFTER - if image doesn't load, please try refresh

WHAT TO DO:
We are a family owned business operating in Memphis, Tennessee for over 29 years.
Please contact us via email:
JumboPosters@netscape.net

or call Toll Free: 1-877-503-7415
to place an order or get additional information.


SIGNATURES LOGO
SIGNATURES
GIANT SIZED POSTERS * CORPORATE APPAREL * SIGNS & BANNERS * PROMOTIONAL PRODUCTS * TROPHIES & AWARDS
559 S. Highland Street * Memphis, TN 38111
901-327-5456

Toll Free: 1-877-503-7415

We are a family owned business operating in Memphis, Tennessee for over 29 years.

 

We honor the removal requests that we receive.
To be removed from future mailings, please email us at IAmDisinterested@netscape.net with your email address (or the email address you want removed) in the Subject Line of your email. Our system checks the subject line for email addresses and will remove those addresses from our database. Please allow up to 48 hours for your email address to be successfully removed.
We apologize for any inconvenience.

From logoprintaihqd@excite.com Fri Nov 14 18:19:11 2003 Return-Path: Received: from excite.com ([61.153.213.14]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id hAENJ5RY027964 for ; Fri, 14 Nov 2003 18:19:08 -0500 (EST) Received: from [19.147.69.173] by smtp.mixedthings.net with QMQP; Fri, 14 Nov 2003 13:19:09 +1000 Received: from unknown (181.51.33.19) by mail.gimmicc.net with NNFMP; Fri, 14 Nov 2003 23:10:13 -1100 Received: from unknown (65.56.151.43) by qrx.quickslick.com with asmtp; Fri, 14 Nov 2003 12:01:17 +1100 Message-ID: <11f201c3aaec$bfb352b0$137b5c38@yexdewkjxa> Reply-To: "" From: "" To: Subject: adv: PHOTO POSTERS; from YOUR pictures..$30 Date: Fri, 14 Nov 2003 22:20:30 +0200 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_E0B_80B6_50913B72.E7D48F35" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2615.200 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200 This is a multi-part message in MIME format. ------=_NextPart_E0B_80B6_50913B72.E7D48F35 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable ------=_NextPart_E0B_80B6_50913B72.E7D48F35 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: base64 PGh0bWw+DQo8Ym9keSBiZ2NvbG9yPSJBenVyZSIgdGV4dD0iSW5kaWdvIj4NCjxwIGFsaWduPSJj ZW50ZXIiPjxmb250IGNvbG9yPSJOYXZ5IiBzaXplPSI0IiBmYWNlPSJWZXJkYW5hIj48c3Ryb25n Pg0KUE9TVEVSUyBmcm9tIFlPVVIgUElDVFVSRVMgJmFtcDsgQVJUIC0gJDMwPC9zdHJvbmc+PC9m b250PjwvcD4NCjxkaXYgYWxpZ249ImNlbnRlciI+PGNlbnRlcj4NCg0KPHRhYmxlIGJvcmRlcj0i MCIgY2VsbHBhZGRpbmc9IjAiIGNlbGxzcGFjaW5nPSIwIiB3aWR0aD0iMTAwJSI+DQogICAgPHRy Pg0KICAgICAgICA8dGQgYWxpZ249ImNlbnRlciIgY29sc3Bhbj0iMyIgd2lkdGg9IjEwMCUiPjxm b250DQogICAgICAgIHNpemU9IjMiIGZhY2U9IlZlcmRhbmEiPldlIGFyZSBhIGZhbWlseSBvd25l ZCBidXNpbmVzcyBpbg0KICAgICAgICBNZW1waGlzLCBUTiAuLi4uIHNpbmNlIDE5NzQ8c3Ryb25n Pjxicj4NCiAgICAgICAgPC9zdHJvbmc+PC9mb250Pjxmb250IGNvbG9yPSIjODAwMDgwIiBzaXpl PSIzIg0KICAgICAgICBmYWNlPSJWZXJkYW5hIj48dT5PVVIgU1BFQ0lBTFRZIElTIENPTU1FUkNJ QUwgV09SSzwvdT48c3Ryb25nPjx1Pjxicj4NCiAgICAgICAgPC91Pjwvc3Ryb25nPjwvZm9udD48 Zm9udCBzaXplPSIzIiBmYWNlPSJWZXJkYW5hIj5TZW5kIHVzDQogICAgICAgIDxlbT5Bbnk8L2Vt PiBQaWN0dXJlIG9yIERyYXdpbmcgb3IgQXJ0d29yazxicj4NCiAgICAgICAgPHN0cm9uZz5XZSds bCBtYWtlIHlvdXIgUG9zdGVyIGZvciAkNy4wMCBhIHNxdWFyZSBmb290PC9zdHJvbmc+PC9mb250 PjwvdGQ+DQogICAgPC90cj4NCiAgICA8dHI+DQogICAgICAgIDx0ZCB3aWR0aD0iMzMlIj48ZGl2 IGFsaWduPSJjZW50ZXIiPjxjZW50ZXI+PHRhYmxlDQogICAgICAgIGJvcmRlcj0iMiIgY2VsbHBh ZGRpbmc9IjIiIGNlbGxzcGFjaW5nPSIwIg0KICAgICAgICBiZ2NvbG9yPSIjRkZGRkFBIiBib3Jk ZXJjb2xvcj0iI0ZGMDAwMCI+DQogICAgICAgICAgICA8dHI+DQogICAgICAgICAgICAgICAgPHRk IGFsaWduPSJjZW50ZXIiPjxmb250IGNvbG9yPSIjODAwMDAwIiBzaXplPSIzIg0KICAgICAgICAg ICAgICAgIGZhY2U9IlZlcmRhbmEiPjxzdHJvbmc+MjImcXVvdDsgeCAyOCZxdW90OyBQb3N0ZXIN CiAgICAgICAgICAgICAgICBmb3IgJDMwLjAwPC9zdHJvbmc+PGJyPg0KICAgICAgICAgICAgICAg IDwvZm9udD48Zm9udCBjb2xvcj0iI0ZGMDAwMCIgc2l6ZT0iMyINCiAgICAgICAgICAgICAgICBm YWNlPSJWZXJkYW5hIj48c3Ryb25nPkZyZWUgU2hpcHBpbmcgZm9yIE9yZGVycw0KICAgICAgICAg ICAgICAgIE92ZXIgJDgwLjAwPGJyPg0KICAgICAgICAgICAgICAgIDwvc3Ryb25nPjwvZm9udD48 Zm9udCBzaXplPSIxIiBmYWNlPSJWZXJkYW5hIj4odG8NCiAgICAgICAgICAgICAgICB0aGUgQ29u dGluZW50YWwgVW5pdGVkIFN0YXRlcyBvbmx5KTwvZm9udD48Zm9udA0KICAgICAgICAgICAgICAg IHNpemU9IjIiIGZhY2U9IlZlcmRhbmEiPjxicj4NCiAgICAgICAgICAgICAgICA8L2ZvbnQ+PGZv bnQgY29sb3I9IiMwMDAwODAiIHNpemU9IjMiDQogICAgICAgICAgICAgICAgZmFjZT0iVmVyZGFu YSI+PHN0cm9uZz5zaGlwcGVkIGluIDItNSBkYXlzPC9zdHJvbmc+PC9mb250Pjxmb250DQogICAg ICAgICAgICAgICAgc2l6ZT0iMyIgZmFjZT0iVmVyZGFuYSI+PHN0cm9uZz48YnI+DQogICAgICAg ICAgICAgICAgPC9zdHJvbmc+PC9mb250Pjxmb250IHNpemU9IjIiIGZhY2U9IlZlcmRhbmEiPihy b2xsZWQNCiAgICAgICAgICAgICAgICBpbiBhIHByb3RlY3RpdmUgbWFpbGluZyB0dWJlKTxicj4N CiAgICAgICAgICAgICAgICA8L2ZvbnQ+PGZvbnQgc2l6ZT0iMyIgZmFjZT0iVmVyZGFuYSI+PHN0 cm9uZz5NaW5pbXVtDQogICAgICAgICAgICAgICAgT3JkZXI6ICQzMC4wMDxicj4NCiAgICAgICAg ICAgICAgICBTaGlwcGluZzogJDcuNTA8YnI+DQogICAgICAgICAgICAgICAgPC9zdHJvbmc+PC9m b250Pjxmb250IHNpemU9IjEiIGZhY2U9IlZlcmRhbmEiPih0bw0KICAgICAgICAgICAgICAgIHRo ZSBDb250aW5lbnRhbCBVbml0ZWQgU3RhdGVzIG9ubHkpPC9mb250PjwvdGQ+DQogICAgICAgICAg ICA8L3RyPg0KICAgICAgICA8L3RhYmxlPg0KICAgICAgICA8L2NlbnRlcj48L2Rpdj48L3RkPg0K ICAgICAgICA8dGQgYWxpZ249ImNlbnRlciIgd2lkdGg9IjMzJSIgYmdjb2xvcj0iI0ZGRkZGRiI+ PGltZw0KICAgICAgICBzcmM9Imh0dHA6Ly93d3cuZ2VvY2l0aWVzLmNvbS9rdWt1ZXR1ZS9waG9s ZGluZ3Bvc3Rlci5naWYiDQogICAgICAgIGFsdD0iZnJvbSBTTkFQU0hPVCB0byBGVUxMIFNJWkVE IFBPU1RFUiIgd2lkdGg9IjQwMCIgaGVpZ2h0PSIxNTAiPjwvdGQ+DQogICAgPC90cj4NCiAgICA8 dHI+DQogICAgICAgIDx0ZCBhbGlnbj0iY2VudGVyIiB3aWR0aD0iMzMlIiBiZ2NvbG9yPSIjRkZG RkZGIj48Zm9udA0KICAgICAgICBzaXplPSIzIiBmYWNlPSJWZXJkYW5hIj5KPHN0cm9uZz5VU1Qg RU1BSUwgWU9VUiBTQ0FOTkVEDQogICAgICAgIElNQUdFUyBPUiBBUlQgQU5EIFdFJ0xMIE1BSUwg WU9VUiBQT1NURVIgSU4gMi01IERBWVMhPC9zdHJvbmc+PC9mb250PjxwPjxmb250DQogICAgICAg IHNpemU9IjIiIGZhY2U9IlZlcmRhbmEiPrAgRmFtaWx5IFBpY3R1cmVzIENhbiBCZWNvbWUNCiAg ICAgICAgQXR0ZW50aW9uIEdldHRpbmcgV2FsbCBBcnQ8YnI+DQogICAgICAgILAgTGFyZ2UgU2l6 ZWQgR3JhcGhpYyBFeGhpYml0czxicj4NCiAgICAgICAgsCBVcHNjYWxlIFNpZ25hZ2U8YnI+DQog ICAgICAgILAgU3BlY2lhbCBNZW1vcmllczxicj4NCiAgICAgICAgsCBNZW1vcmFibGUgSHVtb3Jv dXMgTW9tZW50czwvZm9udD48L3A+DQogICAgICAgIDwvdGQ+DQogICAgICAgIDx0ZCB3aWR0aD0i MzMlIj48ZGl2IGFsaWduPSJjZW50ZXIiPjxjZW50ZXI+PHRhYmxlDQogICAgICAgIGJvcmRlcj0i MSIgY2VsbHNwYWNpbmc9IjAiIGJnY29sb3I9IiNGNEY0RjQiPg0KICAgICAgICAgICAgPHRyPg0K ICAgICAgICAgICAgICAgIDx0ZCBhbGlnbj0ibWlkZGxlIiB2YWxpZ249InRvcCI+PGZvbnQgc2l6 ZT0iMyINCiAgICAgICAgICAgICAgICBmYWNlPSJWZXJkYW5hIj4yMCZxdW90OyB4IDI1JnF1b3Q7 IC0gJCAyNC41MDwvZm9udD48L3RkPg0KICAgICAgICAgICAgICAgIDx0ZCBhbGlnbj0ibWlkZGxl IiB2YWxpZ249InRvcCI+PGZvbnQgc2l6ZT0iMyINCiAgICAgICAgICAgICAgICBmYWNlPSJWZXJk YW5hIj4yNCZxdW90OyB4IDMwJnF1b3Q7IC0gJCAzNS4wMDwvZm9udD48L3RkPg0KICAgICAgICAg ICAgPC90cj4NCiAgICAgICAgICAgIDx0cj4NCiAgICAgICAgICAgICAgICA8dGQgYWxpZ249Im1p ZGRsZSIgdmFsaWduPSJ0b3AiPjxmb250IHNpemU9IjMiDQogICAgICAgICAgICAgICAgZmFjZT0i VmVyZGFuYSI+MjAmcXVvdDsgeCAyOCZxdW90OyAtICQgMjcuNTA8L2ZvbnQ+PC90ZD4NCiAgICAg ICAgICAgICAgICA8dGQgYWxpZ249Im1pZGRsZSIgdmFsaWduPSJ0b3AiPjxmb250IHNpemU9IjMi DQogICAgICAgICAgICAgICAgZmFjZT0iVmVyZGFuYSI+MjkmcXVvdDsgeCAzMiZxdW90OyAtICQg NDUuMDA8L2ZvbnQ+PC90ZD4NCiAgICAgICAgICAgIDwvdHI+DQogICAgICAgICAgICA8dHI+DQog ICAgICAgICAgICAgICAgPHRkIGFsaWduPSJtaWRkbGUiIHZhbGlnbj0idG9wIj48Zm9udCBzaXpl PSIzIg0KICAgICAgICAgICAgICAgIGZhY2U9IlZlcmRhbmEiPjIyJnF1b3Q7IHggMjgmcXVvdDsg LSAkIDMwLjAwPC9mb250PjwvdGQ+DQogICAgICAgICAgICAgICAgPHRkIGFsaWduPSJtaWRkbGUi IHZhbGlnbj0idG9wIj48Zm9udCBzaXplPSIzIg0KICAgICAgICAgICAgICAgIGZhY2U9IlZlcmRh bmEiPjU0JnF1b3Q7IHggOTYmcXVvdDsgLSAkMjUyLjAwPC9mb250PjwvdGQ+DQogICAgICAgICAg ICA8L3RyPg0KICAgICAgICA8L3RhYmxlPg0KICAgICAgICA8L2NlbnRlcj48L2Rpdj48cCBhbGln bj0iY2VudGVyIj48Zm9udCBjb2xvcj0iI0ZGMDAwMCINCiAgICAgICAgc2l6ZT0iMyIgZmFjZT0i VmVyZGFuYSI+PHN0cm9uZz5MQU1JTkFUSU9OOiBKdXN0ICQxLjUwDQogICAgICAgIFBlciBTcXVh cmUgRm9vdDxicj4NCiAgICAgICAgJDEwIE1pbmltdW08L3N0cm9uZz48L2ZvbnQ+PC9wPg0KICAg ICAgICA8ZGl2IGFsaWduPSJjZW50ZXIiPjxjZW50ZXI+PHRhYmxlIGJvcmRlcj0iMSINCiAgICAg ICAgY2VsbHBhZGRpbmc9IjQiIGNlbGxzcGFjaW5nPSIwIiBiZ2NvbG9yPSIjRENGRkI5Ij4NCiAg ICAgICAgICAgIDx0cj4NCiAgICAgICAgICAgICAgICA8dGQgYWxpZ249ImNlbnRlciI+PGZvbnQg c2l6ZT0iMiIgZmFjZT0iVmVyZGFuYSI+PHN0cm9uZz5DT05WRVJUDQogICAgICAgICAgICAgICAg WU9VUjxicj4NCiAgICAgICAgICAgICAgICBGQVZPUklURTo8L3N0cm9uZz48L2ZvbnQ+PGZvbnQg c2l6ZT0iMSINCiAgICAgICAgICAgICAgICBmYWNlPSJWZXJkYW5hIj48YnI+DQogICAgICAgICAg ICAgICAgKkFjdGlvbiBQaWN0dXJlczxicj4NCiAgICAgICAgICAgICAgICAqVGVhbSBQaWN0dXJl PGJyPg0KICAgICAgICAgICAgICAgICpDbGFzcyBQaWN0dXJlPGJyPg0KICAgICAgICAgICAgICAg ICpBbnkgUGljdHVyZTxicj4NCiAgICAgICAgICAgICAgICA8L2ZvbnQ+PGZvbnQgc2l6ZT0iMyIg ZmFjZT0iVmVyZGFuYSI+PHN0cm9uZz5UbyBhDQogICAgICAgICAgICAgICAgR0lBTlQgUE9TVEVS IFNJWkVEIFBSSU5UPC9zdHJvbmc+PC9mb250PjwvdGQ+DQogICAgICAgICAgICAgICAgPHRkIGFs aWduPSJjZW50ZXIiPjxmb250IHNpemU9IjEiIGZhY2U9IlZlcmRhbmEiPjxzdHJvbmc+RElTQ09V TlRTPGJyPg0KICAgICAgICAgICAgICAgIChmb3IgbWluLiBzaXplIG9mIDMgc3EuIGZ0Lik8L3N0 cm9uZz4gPC9mb250PjxkaXYNCiAgICAgICAgICAgICAgICBhbGlnbj0iY2VudGVyIj48Y2VudGVy Pjx0YWJsZSBib3JkZXI9IjAiDQogICAgICAgICAgICAgICAgY2VsbHBhZGRpbmc9IjIiIGNlbGxz cGFjaW5nPSIwIiBiZ2NvbG9yPSIjRkZGRjk3Ij4NCiAgICAgICAgICAgICAgICAgICAgPHRyPg0K ICAgICAgICAgICAgICAgICAgICAgICAgPHRkIGFsaWduPSJjZW50ZXIiPjxmb250IHNpemU9IjEi DQogICAgICAgICAgICAgICAgICAgICAgICBmYWNlPSJWZXJkYW5hIj48c3Ryb25nPjUgcG9zdGVy cyA9PC9zdHJvbmc+PC9mb250PjwvdGQ+DQogICAgICAgICAgICAgICAgICAgICAgICA8dGQgYWxp Z249ImNlbnRlciI+PGZvbnQgc2l6ZT0iMSINCiAgICAgICAgICAgICAgICAgICAgICAgIGZhY2U9 IlZlcmRhbmEiPjxzdHJvbmc+JDYuMDAgcGVyIHNxdWFyZQ0KICAgICAgICAgICAgICAgICAgICAg ICAgZm9vdDwvc3Ryb25nPjwvZm9udD48L3RkPg0KICAgICAgICAgICAgICAgICAgICA8L3RyPg0K ICAgICAgICAgICAgICAgICAgICA8dHI+DQogICAgICAgICAgICAgICAgICAgICAgICA8dGQgYWxp Z249ImNlbnRlciI+PGZvbnQgc2l6ZT0iMSINCiAgICAgICAgICAgICAgICAgICAgICAgIGZhY2U9 IlZlcmRhbmEiPjxzdHJvbmc+MTAgcG9zdGVycyA9PC9zdHJvbmc+PC9mb250PjwvdGQ+DQogICAg ICAgICAgICAgICAgICAgICAgICA8dGQgYWxpZ249ImNlbnRlciI+PGZvbnQgc2l6ZT0iMSINCiAg ICAgICAgICAgICAgICAgICAgICAgIGZhY2U9IlZlcmRhbmEiPjxzdHJvbmc+JDUuNTAgcGVyIHNx dWFyZQ0KICAgICAgICAgICAgICAgICAgICAgICAgZm9vdDwvc3Ryb25nPjwvZm9udD48L3RkPg0K ICAgICAgICAgICAgICAgICAgICA8L3RyPg0KICAgICAgICAgICAgICAgICAgICA8dHI+DQogICAg ICAgICAgICAgICAgICAgICAgICA8dGQgYWxpZ249ImNlbnRlciI+PGZvbnQgc2l6ZT0iMSINCiAg ICAgICAgICAgICAgICAgICAgICAgIGZhY2U9IlZlcmRhbmEiPjxzdHJvbmc+MjAgcG9zdGVycyB1 cCA9PC9zdHJvbmc+PC9mb250PjwvdGQ+DQogICAgICAgICAgICAgICAgICAgICAgICA8dGQgYWxp Z249ImNlbnRlciI+PGZvbnQgc2l6ZT0iMSINCiAgICAgICAgICAgICAgICAgICAgICAgIGZhY2U9 IlZlcmRhbmEiPjxzdHJvbmc+JDUuMDAgcGVyIHNxdWFyZQ0KICAgICAgICAgICAgICAgICAgICAg ICAgZm9vdDwvc3Ryb25nPjwvZm9udD48L3RkPg0KICAgICAgICAgICAgICAgICAgICA8L3RyPg0K ICAgICAgICAgICAgICAgIDwvdGFibGU+DQogICAgICAgICAgICAgICAgPC9jZW50ZXI+PC9kaXY+ PC90ZD4NCiAgICAgICAgICAgIDwvdHI+DQogICAgICAgIDwvdGFibGU+DQogICAgICAgIDwvY2Vu dGVyPjwvZGl2PjwvdGQ+DQogICAgPC90cj4NCjwvdGFibGU+DQo8L2NlbnRlcj48L2Rpdj4NCg0K PHAgYWxpZ249ImNlbnRlciI+PGZvbnQgc2l6ZT0iNSIgZmFjZT0iVmVyZGFuYSI+VG8gcGxhY2Ug YW4gb3JkZXINCm9yIGdldCBhZGRpdGlvbmFsPGJyPg0KaW5mb3JtYXRpb24sIHBsZWFzZSBjb250 YWN0IHVzOjwvZm9udD48Zm9udCBzaXplPSI0Ig0KZmFjZT0iVmVyZGFuYSI+PGJyPg0KPC9mb250 PjxhIGhyZWY9Im1haWx0bzpKdW1ib1Bvc3RlcnNAbmV0c2NhcGUubmV0Ij48Zm9udCBzaXplPSI2 Ig0KZmFjZT0iVmVyZGFuYSI+PHN0cm9uZz5KdW1ib1Bvc3RlcnNAbmV0c2NhcGUubmV0PC9zdHJv bmc+PC9mb250PjwvYT48Zm9udA0Kc2l6ZT0iNiIgZmFjZT0iVmVyZGFuYSI+PHN0cm9uZz48YnI+ DQo8L3N0cm9uZz48L2ZvbnQ+PGZvbnQgc2l6ZT0iNCIgZmFjZT0iVmVyZGFuYSI+PHN0cm9uZz5v ciBjYWxsDQpUb2xsIEZyZWU6IDEtODc3LTUwMy03NDE1PGJyPg0KPC9zdHJvbmc+PC9mb250Pjwv cD4NCjxkaXYgYWxpZ249ImNlbnRlciI+PGNlbnRlcj4NCg0KPHRhYmxlIGJvcmRlcj0iMSIgY2Vs bHBhZGRpbmc9IjUiIGNlbGxzcGFjaW5nPSIwIg0KYmdjb2xvcj0iI0ZGRkZGRiI+DQogICAgPHRy Pg0KICAgICAgICA8dGQgYmdjb2xvcj0iI0ZGRkZGRiI+PHAgYWxpZ249ImNlbnRlciI+PGZvbnQN CiAgICAgICAgZmFjZT0iVmVyZGFuYSI+PHN0cm9uZz4NCjxpbWcgc3JjPSJodHRwOi8vd3d3Lmdl b2NpdGllcy5jb20veHVldHVlL2ZvYnMuZ2lmIiBhbHQ9IlNJR05BVFVSRVMgTE9HTyAtIEEgZmFt aWx5IG93bmVkIGJ1c2luZXNzIHNpbmNlIDE5NzQiIHdpZHRoPSIyMDAiIGhlaWdodD0iMTAwIj4N Cjxicj4NCiAgICAgICAgU0lHTkFUVVJFUzwvc3Ryb25nPjwvZm9udD48YnI+DQogICAgICAgIDxm b250IHNpemU9IjEiIGZhY2U9IkFyaWFsIj5HSUFOVCBTSVpFRCBQT1NURVJTICoNCiAgICAgICAg Q09SUE9SQVRFIEFQUEFSRUwgKiBTSUdOUyAmYW1wOyBCQU5ORVJTICogUFJPTU9USU9OQUwNCiAg ICAgICAgUFJPRFVDVFMgKiBUUk9QSElFUyAmYW1wOyBBV0FSRFM8L2ZvbnQ+PGZvbnQgc2l6ZT0i MSI+PGJyPg0KICAgICAgICA8L2ZvbnQ+PGZvbnQgZmFjZT0iQXJpYWwsSGVsdmV0aWNhIj41NTkg Uy4gSGlnaGxhbmQNCiAgICAgICAgU3RyZWV0ICogTWVtcGhpcywgVE4gMzgxMTE8L2ZvbnQ+IDxi cj4NCiAgICAgICAgPGZvbnQgZmFjZT0iQXJpYWwsSGVsdmV0aWNhIj45MDEtMzI3LTU0NTY8L2Zv bnQ+PC9wPg0KICAgICAgICA8cCBhbGlnbj0iY2VudGVyIj48Zm9udCBmYWNlPSJWZXJkYW5hIj5U b2xsIEZyZWU6DQogICAgICAgIDEtODc3LTUwMy03NDE1PC9mb250Pjxmb250IHNpemU9IjIiIGZh Y2U9IlZlcmRhbmEiPjxicj4NCiAgICAgICAgV2UgYXJlIGEgZmFtaWx5IG93bmVkIGJ1c2luZXNz IG9wZXJhdGluZyBpbiBNZW1waGlzLA0KICAgICAgICBUZW5uZXNzZWUgZm9yIG92ZXIgMjkgeWVh cnMuPC9mb250PjwvcD4NCiAgICAgICAgPC90ZD4NCiAgICA8L3RyPg0KPC90YWJsZT4NCjwvY2Vu dGVyPjwvZGl2Pg0KDQo8cCBhbGlnbj0iY2VudGVyIj4mbmJzcDs8L3A+DQoNCjxwPiZuYnNwOzwv cD4NCg0KPHA+PGZvbnQgc2l6ZT0iMiIgZmFjZT0iVmVyZGFuYSI+V2UgaG9ub3IgdGhlIHJlbW92 YWwgcmVxdWVzdHMNCnRoYXQgd2UgcmVjZWl2ZS48YnI+DQpUbyBiZSByZW1vdmVkIGZyb20gZnV0 dXJlIG1haWxpbmdzLCBwbGVhc2UgZW1haWwgdXMgYXQgPC9mb250PjxhDQpocmVmPSJtYWlsdG86 SUFtRGlzaW50ZXJlc3RlZEBuZXRzY2FwZS5uZXQiPjxmb250IHNpemU9IjIiDQpmYWNlPSJWZXJk YW5hIj48c3Ryb25nPklBbURpc2ludGVyZXN0ZWRAbmV0c2NhcGUubmV0PC9zdHJvbmc+PC9mb250 PjwvYT48Zm9udA0Kc2l6ZT0iMiIgZmFjZT0iVmVyZGFuYSI+PHN0cm9uZz4gd2l0aCB5b3VyIGVt YWlsIGFkZHJlc3MgKG9yIHRoZQ0KZW1haWwgYWRkcmVzcyB5b3Ugd2FudCByZW1vdmVkKSA8L3N0 cm9uZz48ZW0+PHN0cm9uZz5pbiB0aGUgPC9zdHJvbmc+PC9lbT48ZW0+PHN0cm9uZz48dT5TdWJq ZWN0DQpMaW5lPC91Pjwvc3Ryb25nPjwvZW0+PGVtPjxzdHJvbmc+IG9mIHlvdXIgZW1haWw8L3N0 cm9uZz48L2VtPjxzdHJvbmc+Ljwvc3Ryb25nPg0KT3VyIHN5c3RlbSBjaGVja3MgdGhlIHN1Ympl Y3QgbGluZSBmb3IgZW1haWwgYWRkcmVzc2VzIGFuZCB3aWxsDQpyZW1vdmUgdGhvc2UgYWRkcmVz c2VzIGZyb20gb3VyIGRhdGFiYXNlLiBQbGVhc2UgYWxsb3cgdXAgdG8gNDgNCmhvdXJzIGZvciB5 b3VyIGVtYWlsIGFkZHJlc3MgdG8gYmUgc3VjY2Vzc2Z1bGx5IHJlbW92ZWQuPGJyPg0KV2UgYXBv bG9naXplIGZvciBhbnkgaW5jb252ZW5pZW5jZS48L2ZvbnQ+PC9wPg0KDQo8cD48YnI+DQo8YnI+ DQo8YnI+DQo8YnI+DQo8YnI+DQo8YnI+DQo8YnI+DQo8YnI+DQo8L3A+DQoNCjx0YWJsZSBib3Jk ZXI9IjAiIGNlbGxwYWRkaW5nPSIwIiBjZWxsc3BhY2luZz0iMCIgd2lkdGg9IjEwMCUiDQpiZ2Nv bG9yPSIjRkZGRkZGIj4NCiAgICA8dHI+DQogICAgICAgIDx0ZD48Zm9udCBjb2xvcj0iI0ZGRkZG RiI+V3BiZHlncGtjczwvZm9udD48L3RkPg0KICAgIDwvdHI+DQo8L3RhYmxlPg0KPC9ib2R5Pg0K PC9odG1sPg0K ------=_NextPart_E0B_80B6_50913B72.E7D48F35-- From BulkSales@2911.net Mon Nov 24 23:53:20 2003 Return-Path: Received: from 160.36.58.108 ([218.6.2.20]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id hAP4rHRY029799 for ; Mon, 24 Nov 2003 23:53:19 -0500 (EST) Message-ID: From: "Host" To: mpi-comm-archive@netlib2.cs.utk.edu X-MS-QBO: PARZARJROX Date: Tue, 25 Nov 03 12:44:00 Öйú±ê׼ʱ¼ä X-Priority: 1 X-MSMail-Priority: High Subject: Bullet Proof Web Hosting & Server X-Mailer: Microsoft Outlook Express 5.50.4522.1200 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary=WC_MAIL_PaRt_BoUnDaRy_05151998 This is a multi-part message in MIME format. --WC_MAIL_PaRt_BoUnDaRy_05151998 Content-Type: text/plain Content-Transfer-Encoding: 7Bit We offer reliable bulk email friendly web hosting services. You can now havethe peace of mind knowing that your web site is secure during your email marketing campaigns. Bullet-Proof Web Hosting 100% Guaranteed! We guarantee that your site will be 99% uptime. Hosting Features: Your own Web address: www.YourName.com Dedicated 100 M fiber 100MB of disk space Unlimited Data Transfer FTP Support FrontPage Support ASP Support PERL Support PHP Support CGI Support No Setup Fee US$199.00/month We also offer Bullet-Proof dedicated servers: 1. Two IPs 1024MB RAM DDR PIIII / Two CPU 120 GB SCS Intel 82559 LAN /(2U) Unlimited Data Transfer Linux / Windows / FreeBSD Price: No setup fee US$ 599.00/month 2. Dynamic IP 1024MB RAM DDR PIIII / Two CPU 120 GB SCS Intel 82559 LAN /(2U) Unlimited Data Transfer Linux / Windows / FreeBSD Price: No setup fee US$ 999.00/month Allowed Usage You can use the server for any of the following: Direct Bulk Mailing or Proxy Mailing Web Site Hosting Proxy, Relay or Port Scanning We can supply targeted email addresses according to your requirements, Send out Targeted Emails for you We looking forward to serving you in the near future. PS: Please contact us by Emaildata@163.com Cheers! Betty Jones Support Teams mbWNzzISKFtiFcx --WC_MAIL_PaRt_BoUnDaRy_05151998-- From Marketinglist@freemail.soim.com Tue Dec 9 15:33:20 2003 Return-Path: Received: from freemail.soim.com ([218.6.25.209]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id hB9KXEMN004271 for ; Tue, 9 Dec 2003 15:33:17 -0500 (EST) Message-Id: <200312092033.hB9KXEMN004271@netlib2.cs.utk.edu> From: "Info" To: Subject: Marketing Service Sender: "Info" Mime-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-1" Date: Wed, 10 Dec 2003 04:38:04 +0800 Reply-To: "Info" Content-Transfer-Encoding: 8bit Email Marketing is one of the most effective and inexpensive ways to promote your products and services. We offer a complete Email Marketing solution with quality service and the lowest prices. The result is that you will enjoy more success. 1. Targeted Email Addresses We can supply targeted email addresses according to your requirements, which are compiled only on your order. We will customize your customer email addresses. * We have millions of email addresses in a wide variety of categories. 2. Send out Targeted Emails for you If you are worried about any complications or consequences with sending out targeted emails, or want to avoid the work of sending out targeted emails. We will do it for you! We can send your email message to your targeted customers. * We can Bullet-Proof your Web Site. We also offer a wide variety of marketing software. For more details, you can refer to: Http://www.Marketing-Promote.com Looking forward to serving you, and your business needs! Regards! Steve Gellman Marketing Support www.Marketing-Promote.com ************************************************************************ Receiving this email because you registered to receive special offers from one of our partners.To purge your address from our list: http://remove.better-delivery.com ************************************************************************ From promoplacement@excite.com Mon Dec 15 09:05:22 2003 Return-Path: Received: from excite.com (YahooBB218122024045.bbtec.net [218.122.24.45]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id hBFE5IMN022170 for ; Mon, 15 Dec 2003 09:05:20 -0500 (EST) Message-ID: <6d6701c3c2fc$c2f466d0$134396ae@qn> Reply-To: promoplacement@excite.com From: promoplacement@excite.com To: "Human Resources" Subject: adv: 3' x 8' Banner - $59.00 - 5 Year Guarantee Date: Mon, 15 Dec 2003 01:15:35 -1000 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_59D_8551_C40D9ED2.133AC90E" X-Priority: 3 User-Agent: ------=_NextPart_59D_8551_C40D9ED2.133AC90E Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable ------=_NextPart_59D_8551_C40D9ED2.133AC90E Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

= We are a family owned business in Memphis, TN .... since 1974

3' x 8' FULL COLOR PHOTOGRAPHIC BANNER - $150.00
3D"SAMPLE
SAMPLE BANN= ER
We can print any image, photo, drawing,
or logo onto high quality banner material.
FULL PHOTO or FULL PHOTO & TEXT
Guaranteed 5 Years Indoors or 3 Years Outdoors

=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D

•   •
  = 3' x 8' banner $59 - 5 yr. Guarantee
4 Lines of Text - White Background
(Color Background $85.00)
Guaranteed 5 Years Indoors or Outdoors= 		 
•   •

Shipping (2-5 days) to your front door is included in this price!
(Continental U.S. Only)=

=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D

Coroplast Signs
18" x 24" from $4.75 each! (Min 100)
50 =3D $5.75

To place an order= or get additional information, please contact us:
BannersSignsMore@netscape.net

OR CALL US TOLL FREE
1-877-503-7415=

 

=

We are a family owned business, operating in Memphis, Tennessee for over 29 years.
3D"SIGNATURES=
SIGNATURES
GIANT POSTERS * CORPORATE APPAREL * SIGNS & BANNERS * PROMOTIONAL PRODUCTS * TROPHIES & AWARDS
559 S. Highland Street * Memphis, TN 38111
901-327-5456

Vncogiawfb

 

 

 

We honor the removal requests that we receive.
To be removed from future mailings, please email us at IAmDisinterested@netscape.net with your email address (or the email address you want removed) typed into the Subject Line of your email. Our system checks the subject line for addresses and removes those address from our database. Thank you.
We apologize for any inconvenience.

------=_NextPart_59D_8551_C40D9ED2.133AC90E-- From logoplacement@excite.com Tue Dec 16 12:07:45 2003 Return-Path: Received: from excite.com (cpc1-melt1-3-0-cust177.nott.cable.ntl.com [62.254.16.177]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id hBGH7WMN025444 for ; Tue, 16 Dec 2003 12:07:34 -0500 (EST) Received: from unknown (HELO mtu67.syds.piswix.net) (4.141.205.248) by mtu67.syds.piswix.net with smtp; Tue, 16 Dec 2003 08:08:02 +0900 Received: from unknown (HELO mail.gimmicc.net) (119.89.22.190) by smtp18.yenddx.com with SMTP; 16 Dec 2003 17:07:20 +0700 Received: from mailout.endmonthnow.com ([67.166.94.103]) by mx03.listsystemsf.net with SMTP; 17 Dec 2003 00:06:38 -0700 Message-ID: <1dbe01c3c3e8$4f02d300$21308844@jjng> Reply-To: "" From: "" To: "art dept." Subject: adv: CUSTOM POSTERS--$15 from YOUR PICTURES Date: Tue, 16 Dec 2003 18:21:42 +0300 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_82F_C400_C0A75B59.DE3C5E8F" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1106 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 This is a multi-part message in MIME format. ------=_NextPart_82F_C400_C0A75B59.DE3C5E8F Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable ------=_NextPart_82F_C400_C0A75B59.DE3C5E8F Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

CUSTOM POSTERS from YOUR PICTURES & ART - $15
Holiday Special - Offer Expires 12/30/04

=
We are a family owned business in Memphis, TN .... since 1974
Send us Any Picture or Drawing or Artwork
We'll make your Poster for $3.50 a square foot=
22" x 28" Poster for $15.00
Free Shipping for Orders Over $80.00
(to the Continental United States only)
shipped in 2-5 days
(rolled in a protective mailing tube)
Minimum Order: $30.00
Shipping: $7.50
(to the Continental United States only)
3D"from
JUST EMAIL YOUR SCANNED IMAGES OR ART AND WE'LL MAIL YOUR POSTER IN 2-5 DAYS!

=B0 Family Pictures Can Become Attention Getting Wall Art
=B0 Large Sized Graphic Exhibits
=B0 Upscale Signage
=B0 Special Memories
=B0 Memorable Humorous Moments
20" x 25" - $ 12.85 24" x 30" - $ 17.50
20" x 28" - $ 13.75 29" x 32" - $ 22.50
22" x 28" - $ 15.00 54" x 96" - $126.00

LAMINATION: Just $1.50 Per Square Foot

CONVER= T YOUR
FAVORITE:
*Action Pictures
*Team Picture
*Class Picture
*Any Picture
To a GIANT POSTER SIZED PRINT

3D"SIGNATURES
SIGNATURES
GIANT SIZED POSTERS * CORPORATE APPAREL * SIGNS & BANNERS * PROMOTIONAL PRODUCTS * TROPHIES & AWARDS
559 S. Highland Street * Memphis, TN 38111
901-327-5456

Toll Free: 1-877-503-7415
We are a family owned business operating in Memphis, Tennessee for over 29 years.

 

 

We honor the removal requests that we receive.
To be removed from future mailings, please email us at IAmDisinterested@netscape.net<= /a> with your email address (or the email address you want removed) in the Subject Line of your email.= Our system checks the subject line for email addresses and will remove those addresses from our database. Please allow up to 48 hours for your email address to be successfully removed.
We apologize for any inconvenience.








Jkwowlulwu
------=_NextPart_82F_C400_C0A75B59.DE3C5E8F-- From elilzjl.tdf.fq.8saenzj@cuiieue-stwl-tbz.eXcuria.com Mon Feb 16 22:07:23 2004 Return-Path: Received: from cuiieue-stwl-tbz.eXcuria.com (ink.webair.com [216.130.183.68]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i1H37NCn025020 for ; Mon, 16 Feb 2004 22:07:23 -0500 (EST) Message-Id: <200402170307.i1H37NCn025020@netlib2.cs.utk.edu> Date: 2/16/2004 9:08:48 PM To: From: Printer Ink Supplies Reply-to: Subject: Save up to 89% on Ink + No Shipping Cost Save up to 89% on Inkjet, Laser & Copier Supplies Quality Products, with 100% Satisfaction Guarantee Easy, Fast, Affordable Shipping Worldwide Plenty of Payment Options to Meet YOUR Needs! >> SPECIAL: FREE Shipping to US & Canada on Orders over $50 << Visit us on the web at http://www.eXcuria.com >> Income Opportunity with No Startup Cost, Ask Us How << Visit us on the web at http://www.eXcuria.com/DealerInfo/ If you wish to contact us please visit our web site. For instruction on how to be permanently remove from this distribution system go to http://www.eXcuria.com/Remove/ From guaranteedengineinclusion@fsmail.net Mon Mar 1 08:34:04 2004 Return-Path: Received: from GZ-FYOIRVHVXU9Y ([219.129.20.249]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i21DXvCn020179 for ; Mon, 1 Mar 2004 08:34:00 -0500 (EST) Message-Id: <200403011334.i21DXvCn020179@netlib2.cs.utk.edu> X-Priority: 3 (normal) X-MSMail-Priority: Normal X-Mailer: MailingLIST_Email_Sender Importance: Normal Date: Fri, 1 Jun 2001 2:35:56 -0700 Subject: Guaranteed Search Engine Inclusion in only 72 hours To: "mpi-comm-archive@netlib2.cs.utk.edu" From: "Guaranteed SE Inclusion" MIME-Version: 1.0 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: base64 PGh0bWw+DQo8aGVhZD4NCjx0aXRsZT5HdWFyYW50ZWVkIEluY2x1c2lvbiBFbWFpbDwvdGl0bGU+ DQo8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUiIGNvbnRlbnQ9InRleHQvaHRtbDsgY2hh cnNldD1pc28tODg1OS0xIj4NCjwvaGVhZD4NCg0KPGJvZHkgYmdjb2xvcj0iI2Y0ZjBlYyIgdGV4 dD0iIzY2NjY2NiIgbGluaz0iIzMzMzMzMyIgdmxpbms9IiM5OTk5OTkiIGFsaW5rPSIjMzMzMzMz Ij4NCjxkaXYgYWxpZ249ImNlbnRlciI+DQogIDx0YWJsZSB3aWR0aD0iNTUwIiBjZWxsc3BhY2lu Zz0iMCIgY2VsbHBhZGRpbmc9IjAiIGhlaWdodD0iNTAwIiBiZ2NvbG9yPSIjRkZGRkZGIiBiYWNr Z3JvdW5kPSJodHRwOi8vMjE5LjEyOS4yMC4yNDkvZ3VhcmFudGVlZGluY2x1c2lvbi9lbWFpbC9l bWFpbGJnLmpwZyI+DQogICAgPHRyPiANCiAgICAgIDx0ZCBoZWlnaHQ9IjY4IiBjb2xzcGFuPSIy Ij4NCiAgICAgICAgPGRpdiBhbGlnbj0icmlnaHQiPjxmb250IGZhY2U9IlRpbWVzIE5ldyBSb21h biwgVGltZXMsIHNlcmlmIiBzaXplPSI2IiBjb2xvcj0iIzAwMDA2NiI+PHU+R3VhcmFudGVlZCAN CiAgICAgICAgICBTZWFyY2ggRW5naW5lIEluY2x1c2lvbjxmb250IHNpemU9IjIiPiA8L2ZvbnQ+ PC91PjwvZm9udD48L2Rpdj4NCiAgICAgIDwvdGQ+DQogICAgPC90cj4NCiAgICA8dHI+IA0KICAg ICAgPHRkIHdpZHRoPSIxMTYiIGhlaWdodD0iNTAzIiB2YWxpZ249InRvcCI+IA0KICAgICAgICA8 ZGl2IGFsaWduPSJjZW50ZXIiPiANCiAgICAgICAgICA8cD48aW1nIHNyYz0iaHR0cDovLzIxOS4x MjkuMjAuMjQ5L2d1YXJhbnRlZWRpbmNsdXNpb24vZW1haWwvc3BhY2UuZ2lmIiB3aWR0aD0iNSIg aGVpZ2h0PSIxMTAiPjwvcD4NCiAgICAgICAgICA8cD48aW1nIHNyYz0iaHR0cDovLzIxOS4xMjku MjAuMjQ5L2d1YXJhbnRlZWRpbmNsdXNpb24vZW1haWwvc2VhcmNobG9nb3MuZ2lmIiB3aWR0aD0i OTkiIGhlaWdodD0iMzg1IiBhbGlnbj0iYm90dG9tIj48L3A+DQogICAgICAgIDwvZGl2Pg0KICAg ICAgPC90ZD4NCiAgICAgIDx0ZCB3aWR0aD0iNDM0IiBoZWlnaHQ9IjUwMyIgdmFsaWduPSJ0b3Ai PiANCiAgICAgICAgPGRpdiBhbGlnbj0iY2VudGVyIj4gDQogICAgICAgICAgPHA+PGJyPg0KICAg ICAgICAgICAgPGZvbnQgZmFjZT0iQXJpYWwsIEhlbHZldGljYSwgc2Fucy1zZXJpZiIgc2l6ZT0i NSIgY29sb3I9IiMwMDAwNjYiPllvdXIgDQogICAgICAgICAgICBXZWIgU2l0ZSBSYW5rZWQgaW4g b25seSAzIGRheXM8YnI+DQogICAgICAgICAgICA8L2ZvbnQ+PGZvbnQgZmFjZT0iQXJpYWwsIEhl bHZldGljYSwgc2Fucy1zZXJpZiIgc2l6ZT0iNSIgY29sb3I9IiMwMDAwNjYiPmluIA0KICAgICAg ICAgICAgb3ZlciAxMDAgTWFqb3IgU2VhcmNoIEVuZ2luZXM8YnI+DQogICAgICAgICAgICA8YnI+ DQogICAgICAgICAgICA8YSBocmVmPSJodHRwOi8vMjE5LjEyOS4yMC4yNDkvZ3VhcmFudGVlZGlu Y2x1c2lvbi9pbmRleC5odG1sIj48aW1nIHNyYz0iaHR0cDovLzIxOS4xMjkuMjAuMjQ5L2d1YXJh bnRlZWRpbmNsdXNpb24vZW1haWwvYnV0dG9uMV90b2ZpbmRvdXRtb3JlMi5naWYiIHdpZHRoPSIx OTAiIGhlaWdodD0iMjAiIGJvcmRlcj0iMCI+PC9hPjwvZm9udD48L3A+DQogICAgICAgICAgPHAg YWxpZ249ImNlbnRlciI+PGZvbnQgZmFjZT0iQXJpYWwsIEhlbHZldGljYSwgc2Fucy1zZXJpZiIg c2l6ZT0iNSIgY29sb3I9IiMwMDAwNjYiPjxiPklUUyANCiAgICAgICAgICAgIEdVQVJBTlRFRUQu Li4uPC9iPjwvZm9udD48L3A+DQogICAgICAgICAgPHRhYmxlIHdpZHRoPSI4NSUiIGJvcmRlcj0i MCIgY2VsbHNwYWNpbmc9IjAiIGNlbGxwYWRkaW5nPSIwIj4NCiAgICAgICAgICAgIDx0cj4gDQog ICAgICAgICAgICAgIDx0ZCB3aWR0aD0iNDEiIGhlaWdodD0iMzAiPjxpbWcgc3JjPSJodHRwOi8v MjE5LjEyOS4yMC4yNDkvZ3VhcmFudGVlZGluY2x1c2lvbi9lbWFpbC9hcnJvd3MuanBnIiB3aWR0 aD0iMjUiIGhlaWdodD0iMjEiPjwvdGQ+DQogICAgICAgICAgICAgIDx0ZCB3aWR0aD0iMzI2IiBo ZWlnaHQ9IjMwIj48Yj48Zm9udCBmYWNlPSJBcmlhbCwgSGVsdmV0aWNhLCBzYW5zLXNlcmlmIiBz aXplPSIyIiBjb2xvcj0iIzAwMDA2NiI+V0UgDQogICAgICAgICAgICAgICAgUEFZIFRIRSBTRUFS Q0ggRU5HSU5FUyBGT1IgWU9VPC9mb250PjwvYj48L3RkPg0KICAgICAgICAgICAgPC90cj4NCiAg ICAgICAgICAgIDx0cj4gDQogICAgICAgICAgICAgIDx0ZCB3aWR0aD0iNDEiIGhlaWdodD0iMzAi PjxpbWcgc3JjPSJodHRwOi8vMjE5LjEyOS4yMC4yNDkvZ3VhcmFudGVlZGluY2x1c2lvbi9lbWFp bC9hcnJvd3MuanBnIiB3aWR0aD0iMjUiIGhlaWdodD0iMjEiPjwvdGQ+DQogICAgICAgICAgICAg IDx0ZCB3aWR0aD0iMzI2IiBoZWlnaHQ9IjMwIj48Yj48Zm9udCBmYWNlPSJBcmlhbCwgSGVsdmV0 aWNhLCBzYW5zLXNlcmlmIiBzaXplPSIyIiBjb2xvcj0iIzAwMDA2NiI+WU9VUiANCiAgICAgICAg ICAgICAgICBTSVRFIElTIFJBTktFRCBJTiA3MiBIT1VSUzwvZm9udD48L2I+PC90ZD4NCiAgICAg ICAgICAgIDwvdHI+DQogICAgICAgICAgICA8dHI+IA0KICAgICAgICAgICAgICA8dGQgd2lkdGg9 IjQxIiBoZWlnaHQ9IjMwIj48aW1nIHNyYz0iaHR0cDovLzIxOS4xMjkuMjAuMjQ5L2d1YXJhbnRl ZWRpbmNsdXNpb24vZW1haWwvYXJyb3dzLmpwZyIgd2lkdGg9IjI1IiBoZWlnaHQ9IjIxIj48L3Rk Pg0KICAgICAgICAgICAgICA8dGQgd2lkdGg9IjMyNiIgaGVpZ2h0PSIzMCI+PGI+PGZvbnQgZmFj ZT0iQXJpYWwsIEhlbHZldGljYSwgc2Fucy1zZXJpZiIgc2l6ZT0iMiIgY29sb3I9IiMwMDAwNjYi PllPVVIgDQogICAgICAgICAgICAgICAgU0lURSBJUyBDSEVDS0VEIEVWRVJZIDQ4IEhPVVJTPC9m b250PjwvYj48L3RkPg0KICAgICAgICAgICAgPC90cj4NCiAgICAgICAgICAgIDx0cj4gDQogICAg ICAgICAgICAgIDx0ZCB3aWR0aD0iNDEiIGhlaWdodD0iMzAiPjxpbWcgc3JjPSJodHRwOi8vMjE5 LjEyOS4yMC4yNDkvZ3VhcmFudGVlZGluY2x1c2lvbi9lbWFpbC9hcnJvd3MuanBnIiB3aWR0aD0i MjUiIGhlaWdodD0iMjEiPjwvdGQ+DQogICAgICAgICAgICAgIDx0ZCB3aWR0aD0iMzI2IiBoZWln aHQ9IjMwIj48Yj48Zm9udCBmYWNlPSJBcmlhbCwgSGVsdmV0aWNhLCBzYW5zLXNlcmlmIiBzaXpl PSIyIiBjb2xvcj0iIzAwMDA2NiI+WU9VUiANCiAgICAgICAgICAgICAgICBTSVRFIFJBTktFRCBG T1IgMTIgTU9OVEhTPC9mb250PjwvYj48L3RkPg0KICAgICAgICAgICAgPC90cj4NCiAgICAgICAg ICAgIDx0cj4gDQogICAgICAgICAgICAgIDx0ZCB3aWR0aD0iNDEiIGhlaWdodD0iMzAiPjxpbWcg c3JjPSJodHRwOi8vMjE5LjEyOS4yMC4yNDkvZ3VhcmFudGVlZGluY2x1c2lvbi9lbWFpbC9hcnJv d3MuanBnIiB3aWR0aD0iMjUiIGhlaWdodD0iMjEiPjwvdGQ+DQogICAgICAgICAgICAgIDx0ZCB3 aWR0aD0iMzI2IiBoZWlnaHQ9IjMwIj48Yj48Zm9udCBmYWNlPSJBcmlhbCwgSGVsdmV0aWNhLCBz YW5zLXNlcmlmIiBzaXplPSIyIiBjb2xvcj0iIzAwMDA2NiI+SU4gDQogICAgICAgICAgICAgICAg T1ZFUiAxMDAgTUFKT1IgU0VBUkNIIEVOR0lORVM8L2ZvbnQ+PC9iPjwvdGQ+DQogICAgICAgICAg ICA8L3RyPg0KICAgICAgICAgICAgPHRyPiANCiAgICAgICAgICAgICAgPHRkIHdpZHRoPSI0MSIg aGVpZ2h0PSIzMCI+PGltZyBzcmM9Imh0dHA6Ly8yMTkuMTI5LjIwLjI0OS9ndWFyYW50ZWVkaW5j bHVzaW9uL2VtYWlsL2Fycm93cy5qcGciIHdpZHRoPSIyNSIgaGVpZ2h0PSIyMSI+PC90ZD4NCiAg ICAgICAgICAgICAgPHRkIHdpZHRoPSIzMjYiIGhlaWdodD0iMzAiPjxiPjxmb250IGZhY2U9IkFy aWFsLCBIZWx2ZXRpY2EsIHNhbnMtc2VyaWYiIHNpemU9IjIiIGNvbG9yPSIjMDAwMDY2Ij5GUkVF IA0KICAgICAgICAgICAgICAgIFJFUE9SVFMgT04gWU9VUiBTSVRFIFJBTktJTkdTPC9mb250Pjwv Yj48L3RkPg0KICAgICAgICAgICAgPC90cj4NCiAgICAgICAgICA8L3RhYmxlPg0KICAgICAgICAg IDxicj4NCiAgICAgICAgICA8dGFibGUgd2lkdGg9IjY4JSIgYm9yZGVyPSIwIiBjZWxsc3BhY2lu Zz0iMCIgY2VsbHBhZGRpbmc9IjAiPg0KICAgICAgICAgICAgPHRyPg0KICAgICAgICAgICAgICA8 dGQgd2lkdGg9IjczJSI+PGZvbnQgZmFjZT0iQXJpYWwsIEhlbHZldGljYSwgc2Fucy1zZXJpZiIg c2l6ZT0iMyI+PGI+PGZvbnQgY29sb3I9IiMwMDAwNjYiPkFMTCANCiAgICAgICAgICAgICAgICBU SElTIEZPUiBPTkxZIC4uLi48L2ZvbnQ+PC9iPjwvZm9udD48L3RkPg0KICAgICAgICAgICAgICA8 dGQgd2lkdGg9IjI3JSI+PGZvbnQgc2l6ZT0iMyIgZmFjZT0iQXJpYWwsIEhlbHZldGljYSwgc2Fu cy1zZXJpZiIgY29sb3I9IiMwMDAwNjYiPjxiPiZwb3VuZDsxMDAgDQogICAgICAgICAgICAgICAg PC9iPiA8Zm9udCBzaXplPSIyIj4gR0JQIDwvZm9udD48L2ZvbnQ+PC90ZD4NCiAgICAgICAgICAg IDwvdHI+DQogICAgICAgICAgICA8dHI+DQogICAgICAgICAgICAgIDx0ZCB3aWR0aD0iNzMlIj48 Zm9udCBzaXplPSIxIiBmYWNlPSJBcmlhbCwgSGVsdmV0aWNhLCBzYW5zLXNlcmlmIiBjb2xvcj0i IzAwMDA2NiI+KFNVQkpFQ1QgDQogICAgICAgICAgICAgICAgVE8gRVhDSEFOR0UgUkFURVMpPC9m b250PjwvdGQ+DQogICAgICAgICAgICAgIDx0ZCB3aWR0aD0iMjclIj48Zm9udCBmYWNlPSJBcmlh bCwgSGVsdmV0aWNhLCBzYW5zLXNlcmlmIiBzaXplPSIzIj48Yj48Zm9udCBjb2xvcj0iIzAwMDA2 NiI+RTE0MDwvZm9udD48L2I+PGZvbnQgY29sb3I9IiMwMDAwNjYiIHNpemU9IjIiPiANCiAgICAg ICAgICAgICAgICBFVVJPPC9mb250PjwvZm9udD48L3RkPg0KICAgICAgICAgICAgPC90cj4NCiAg ICAgICAgICAgIDx0cj4NCiAgICAgICAgICAgICAgPHRkIHdpZHRoPSI3MyUiPiZuYnNwOzwvdGQ+ DQogICAgICAgICAgICAgIDx0ZCB3aWR0aD0iMjclIj48Zm9udCBmYWNlPSJBcmlhbCwgSGVsdmV0 aWNhLCBzYW5zLXNlcmlmIiBzaXplPSIzIj48Yj48Zm9udCBjb2xvcj0iIzAwMDA2NiI+JDE2MDwv Zm9udD48L2I+PGZvbnQgY29sb3I9IiMwMDAwNjYiIHNpemU9IjIiPiANCiAgICAgICAgICAgICAg ICBVU0QgPC9mb250PjwvZm9udD48L3RkPg0KICAgICAgICAgICAgPC90cj4NCiAgICAgICAgICA8 L3RhYmxlPg0KICAgICAgICAgIDxwPjxhIGhyZWY9Imh0dHA6Ly8yMTkuMTI5LjIwLjI0OS9ndWFy YW50ZWVkaW5jbHVzaW9uL2luZGV4Lmh0bWwiPjxpbWcgc3JjPSJodHRwOi8vMjE5LjEyOS4yMC4y NDkvZ3VhcmFudGVlZGluY2x1c2lvbi9lbWFpbC9idXR0b24xX3RvZmluZG91dG1vcmUyLmdpZiIg d2lkdGg9IjE5MCIgaGVpZ2h0PSIyMCIgYm9yZGVyPSIwIj48L2E+PC9wPg0KICAgICAgICAgIDxw Pjxmb250IGZhY2U9IkFyaWFsLCBIZWx2ZXRpY2EsIHNhbnMtc2VyaWYiIHNpemU9IjIiIGNvbG9y PSIjMDAwMDY2Ij48YSBocmVmPSJodHRwOi8vMjE5LjEyOS4yMC4yNDkvZ3VhcmFudGVlZGluY2x1 c2lvbi9pbmRleC5odG1sIj5JZiANCiAgICAgICAgICAgIHlvdSB3b3VsZCBsaWtlIHRvIGZpbmQg b3V0IG1vcmUgcGxlYXNlIGNsaWNrIGhlcmU8L2E+PC9mb250PjwvcD4NCiAgICAgICAgICA8cD48 ZGl2IGFsaWduPSJjZW50ZXIiPjxhIGhyZWY9Imh0dHA6Ly93d3cuYmVzcG9rZS1tYXJrZXRpbmcu Y29tL3Vuc3Vic2NyaWJlLmFzcCI+PGltZyBzcmM9Imh0dHA6Ly93d3cuYmVzcG9rZS1tYXJrZXRp bmcuY29tL3dlYmNoZWNrL2hfMDcuZ2lmIiB3aWR0aD0iMjU5IiBoZWlnaHQ9IjEwIiBib3JkZXI9 IjAiPjwvYT48L2Rpdj4NCiAgICAgICAgPC9kaXY+DQogICAgICAgIDwvdGQ+DQogICAgPC90cj4N CiAgPC90YWJsZT4NCjwvZGl2Pg0KPC9ib2R5Pg0KPC9odG1sPg0K From beloige@postino.it Fri Mar 12 23:49:04 2004 Return-Path: Received: from abraham.elitel.it (vdisk2.elitel.it [212.34.224.151]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i2D4n3Cn013302 for ; Fri, 12 Mar 2004 23:49:04 -0500 (EST) Received: (qmail 9958 invoked by uid 65534); 13 Mar 2004 04:46:06 -0000 Cc: recipient list not shown: ; Received: from CBLO-CM (CBLO-CM [216.219.4.156]) by www.postino.punto.it (IMP) with HTTP for ; Sat, 13 Mar 2004 05:45:34 +0100 Message-ID: <1079153134.405291eecb144@www.postino.punto.it> Date: Sat, 13 Mar 2004 05:45:34 +0100 From: beloige@postino.it Reply-to: beloige@omaninfo.com Subject: seeking your partnership MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.2.2 X-Originating-IP: 216.219.4.156 Dear Partner to be, First, I must apologise to you for using this medium to communicate to you about this project and It is my great pleasure in writing you this letter on behalf of my colleagues and myself. I am Mr. Belo Ige,Divisional Director in charge of Corporate and Investment Banking with CONTINENTAL TRUST BANK,based in Lagos,Nigeria. I have a confidential business proposition for you. Our Bank is a financial institution specialising in private banking to individual customers who likes confidentiality in their financial dealings. On January 13,1998 an Indian-American mining consultant with a Multi National Firm based in Nigeria here,the Petrogas Gas Systems BV Environmental Systems headquartered in The Netherlands, Mr.Kommera Sangeeta made a numbered time (Fixed)Deposit for Twenty Four calendar months, valued at US$29,500,000.00 (Twenty Nine million, Five Hundred Thousand U.S. Dollars)in the Bank. Upon maturity, I sent a routine notification to his forwarding address but got no reply. After a month, we sent a reminder and finally we discovered from his contract employers,that Mr. Kommera Sangeeta died in the 2000 Singapore Airlines flight SQ006 plane crash in Taipei. On further investigation, I found out that he died intestate(without making a WILL), and all attempts to trace his next-of-kin had been fruitless.Please see http://www.tamil.net/list/2000-11/msg00018.html I therefore made further investigation and discovered that Mr. Kommera Sangeeta did not declare any kin or relations in all his official documents with us, including his Bank Deposit paperwork. This sum of US$29,500,000.00 is still sitting in my Bank and the interest is being rolled over with the principal sum at the end of each year. Consequently, my proposal is that I will like you as a foreigner to stand in as the next of kin to Mr.Kommera Sageeta. This is simple, I will like you to provide your full names and home or office address including your telephone and fax numbers so that I can seek the services of a lawyer/Attorney to prepare the necessary documents and court affidavits which will put you in good standing as the next-of-kin. A bank account in any part of the world which you will provide later will then facilitate the transfer of this money to you as the beneficiary/next of kin. The money will be paid into your account for us to share in the ratio of 75% for me and 20% for you and 5% to refund all manner of expenses and costs incurred by all parties leading to the success of the project. Be informed that in other to provide more legality and legitimacy to this project,the paperworks for this will be done by the Attorney/Lawyer and my position as the head of Corporate and Investment Banking guarantees the successful execution of this transaction. If you are interested, please revert back to me. Please observe utmost confidentiality, and rest assured that this transaction would be most profitable for both of us because I shall require your assistance to invest my share in your country. Thanks and regards. Mr. Belo Ige From adminstration@computerbizop.com Tue Mar 16 00:02:40 2004 Return-Path: Received: from 200-171-213-234.customer.tdatabrasil.net.br (200-171-213-234.customer.tdatabrasil.net.br [200.171.213.234] (may be forged)) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i2G52aCn022752; Tue, 16 Mar 2004 00:02:38 -0500 (EST) Received: from 8aib1.87lb.org (HELO 3hhvkn) ([53.74.126.180]) by 200-171-213-234.customer.tdatabrasil.net.br with ESMTP id 69225264; Tue, 16 Mar 2004 00:04:54 -0500 Message-ID: From: "Admin" To: moler@netlib2.cs.utk.edu Subject: ADV: Attention All School Staff, Teachers and Students: News 3/15/04 Date: Tue, 16 Mar 04 00:04:54 GMT X-Priority: 1 X-MSMail-Priority: High X-Mailer: Microsoft Outlook, Build 10.0.2627 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="2A.A._C97CDBE" This is a multi-part message in MIME format. --2A.A._C97CDBE Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Attention All School Staff, Teachers and Students: You Must Respond By 5 P.M. Wednesday, March 17, 2004. Through a special arrangement, Avtech Direct is offering a limited allotment of BRAND NEW, top of-the-line, name-brand desktop computers at more than 50% off MSRP to all Staff Members, Teachers and Students who respond to this message before 5 P.M., Wednesday, March 17, 2004. All desktop computers are brand-new packed in their original boxes, and come with a full manufacturer's warranty plus a 100% satisfaction guarantee. These professional grade Desktops are fully equipped with 2004 next generation technology, making these the best performing computers money can buy. Avtech Direct is offering these feature rich, top performing Desktop Computers with the latest Intel technology at an amazing price to all who call: 1-800-884-9510 by 5 P.M. Wednesday, March 17, 2004 The fast and powerful AT-2400 series Desktop features: * Intel 2.0Ghz Processor for amazing speed and performance * 128MB DDR RAM, --- Upgradeable to 1024 * 20 GB UDMA Hard Drive, --- Upgradeable to 80 GB * 52X CD-Rom Drive, --- Upgradeable to DVD/CDRW * 1.44 Floppy disk drive * Next Generation Technology * ATI Premium video and sound * Full Connectivity with Fax modem/Lan/IEE 1394/USB 2.0 * Soft Touch Keyboard and scroll mouse * Internet Ready * Network Ready * 1 Year parts and labor warranty * Priority customer service and tech support MSRP $699 ........................................ Your Cost $297 How to qualify: 1. You must be a Staff member, Teacher or Student: 2. All desktop computers will be available on a first come first serve basis. 3. You must call 1-800-884-9510 by 5 P.M. Wednesday, March 17, 2004 and we will hold the desktops you request on will call. 4. You are not obligated in any way. 5. 100% Satisfaction Guaranteed. Call Avtech Direct 1-800-884-9510 before 5 P.M. Wednesday, March 17, 2004 If you wish to unsubscribe from this list, please go to: http://www.computeradvice.org/unsubscribe.asp --2A.A._C97CDBE-- From Jonsonger@mail.runan.ha.cn Thu Mar 25 16:40:04 2004 Return-Path: Received: from 160.36.58.108 ([220.161.118.214]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i2PLduCn016888 for ; Thu, 25 Mar 2004 16:39:59 -0500 (EST) Message-ID: From: "Sales" To: mpi-comm-archive@netlib2.cs.utk.edu X-MS-OFZ: UUFESBUZQN Date: Fri, 26 Mar 04 05:46:24 Öйú±ê׼ʱ¼ä X-Priority: 1 X-MSMail-Priority: High Subject: Dedicated Server & Hosting for You X-Mailer: Microsoft Outlook Express 6.00.2600.0000 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary=WC_MAIL_PaRt_BoUnDaRy_05151998 This is a multi-part message in MIME format. --WC_MAIL_PaRt_BoUnDaRy_05151998 Content-Type: text/html Content-Transfer-Encoding: 7Bit We offer reliable bulk email friendly web hosting services

We offer reliable friendly web hosting services. You can now have the peace of 
mind knowing that your site is secure during your email marketing campaigns.

BulletProof Web Hosting 100% Guaranteed! We guarantee that your site will be
99% uptime.

We also offer BulletProof dedicated server:

Two IPs
512MB RAM DDR
P4 2.66GHZ CPU
36 GB SCSI
Unlimited Data Transfer
Linux/Windows/FreeBSD
 		 Three IPs
1024MB RAM DDR
P4 / Two CPU
72 GB SCSI
Unlimited Data Transfer
Linux/Windows/FreeBSD
 		 Dynamic IP
1024MB RAM DDR
P4 / Two CPU
72 GB SCSI
Unlimited Data Transfer
Linux/Windows/FreeBSD
 
Price: No setup fee
          US$ 599.00/month		 Price: No setup fee
          US$ 799.00/month		 Price: No setup fee
          US$ 999.00/month


Allowed Usage
You can use the server for any of the following:

Direct Mailing or Proxy Mailing
Web Site Hosting
Proxy, Relay or Port Scanning

We can supply Targeted Email Addresses according to your requirements, and
send out Targeted Emails for you. For more information, please contact us by 
Sales@biz-boosting.com or Info@2892.com

It will be our pleasure to do business with you.


Cheers!

Mike Tank
Support Teams

****************************************************************************************
Receiving this e-mail because you registered to receive offers from us.
Please click here to unscbscirbe:List@yahoo.com?subject=Re
****************************************************************************************

oOdUyRLHIClRlhW --WC_MAIL_PaRt_BoUnDaRy_05151998-- From Administration@hispeedweb.com Fri Apr 2 01:45:46 2004 Return-Path: Received: from dsl-201-128-127-206.prod-infinitum.com.mx (dsl-201-128-127-206.prod-infinitum.com.mx [201.128.127.206] (may be forged)) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i326jHCn024134; Fri, 2 Apr 2004 01:45:30 -0500 (EST) Received: from 58g.zmoeyc.com ([216.136.183.145]) by dsl-201-128-127-206.prod-infinitum.com.mx id airmzDriXRpS for ; Fri, 02 Apr 2004 12:46:07 +0600 Message-ID: From: "Admin" To: moler@netlib2.cs.utk.edu Subject: ADV: Attention All School Staff, Teachers and Students: Date: Fri, 02 Apr 04 12:46:07 GMT X-Priority: 1 X-MSMail-Priority: High X-Mailer: The Bat! (v1.52f) Business MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="D6E.B_91D0C4BA9" This is a multi-part message in MIME format. --D6E.B_91D0C4BA9 Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Attention All School Staff, Teachers and Students: You Must Respond By 5 P.M. Monday, April 5, 2004. Through a special arrangement, Avtech Direct is offering a limited allotment of BRAND NEW, top of-the-line, name-brand desktop computers at more than 50% off MSRP to all Staff Members, Teachers and Students who respond to this message before 5 P.M., Monday, April 5, 2004. All desktop computers are brand-new packed in their original boxes, and come with a full manufacturer's warranty plus a 100% satisfaction guarantee. These professional grade Desktops are fully equipped with 2004 next generation technology, making these the best performing computers money can buy. Avtech Direct is offering these feature rich, top performing Desktop Computers with the latest Intel technology at an amazing price to all who call: 1-800-884-9510 by 5 P.M. Monday, April 5, 2004 The fast and powerful AT-2400 series Desktop features: * Intel 2.0Ghz Processor for amazing speed and performance * 128MB DDR RAM, --- Upgradeable to 1024 * 20 GB UDMA Hard Drive, --- Upgradeable to 80 GB * 52X CD-Rom Drive, --- Upgradeable to DVD/CDRW * 1.44 Floppy disk drive * Next Generation Technology * ATI Premium video and sound * Full Connectivity with Fax modem/Lan/IEE 1394/USB 2.0 * Soft Touch Keyboard and scroll mouse * Internet Ready * Network Ready * 1 Year parts and labor warranty * Priority customer service and tech support MSRP $699 ........................................ Your Cost $297 How to qualify: 1. You must be a Staff member, Teacher or Student: 2. All desktop computers will be available on a first come first serve basis. 3. You must call 1-800-884-9510 by 5 P.M. Monday, April 5, 2004 and we will hold the desktops you request on will call. 4. You are not obligated in any way. 5. 100% Satisfaction Guaranteed. Call Avtech Direct 1-800-884-9510 before 5 P.M. Monday, April 5, 2004 If you wish to unsubscribe from this list, please go to: http://www.computeradvice.org/unsubscribe.asp --D6E.B_91D0C4BA9-- From dfsflirp@eXcuria.com Mon Apr 5 11:29:18 2004 Return-Path: Received: from mail.excuria.com ([207.36.209.178]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i35FTHCn004683 for ; Mon, 5 Apr 2004 11:29:17 -0400 (EDT) Date: 4/5/2004 11:29:17 AM To: From: Printer Ink Supplies Reply-to: Message-ID: Subject: Inkjet, Laser & Copier Cartridges ~ Save up to 89% Save up to 89% on Inkjet, Laser & Copier Supplies Quality Products, with 100% Satisfaction Guarantee Easy, Fast, Affordable Shipping Worldwide Plenty of Payment Options to Meet YOUR Needs! >> SPECIAL: FREE Shipping to US & Canada on Orders over $50 << Visit us on the web at http://www.eXcuria.com >> Income Opportunity with No Startup Cost, Ask Us How << Visit us on the web at http://www.eXcuria.com/DealerInfo/ If you wish to contact us please visit our web site. For instruction on how to be permanently remove from this distribution system go to http://www.eXcuria.com/Remove/ From spanlatbooks@fibertel.com.ar Thu Apr 15 05:52:43 2004 Return-Path: Received: from net.websailing.com.ar (qmailr@customer79-117.iplannetworks.net [200.68.76.117] (may be forged)) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i3F9qcCn016974 for ; Thu, 15 Apr 2004 05:52:43 -0400 (EDT) Message-Id: <200404150952.i3F9qcCn016974@netlib2.cs.utk.edu> Received: (qmail 12969 invoked from network); 14 Apr 2004 06:33:18 -0000 Received: from pc-00569.net.websailing.com.ar (HELO fibertel.com.ar) (90.0.2.57) by netserver.net.websailing.com.ar (90.0.0.1) with SMTP; 14 Apr 2004 06:33:18 -0000 From: "Spanish & LATAM Books" To: Subject: Do you work/study/need academic quality Spanish & LATAM books ? Sender: "Spanish & LATAM Books" Mime-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-1" Date: Wed, 14 Apr 2004 21:03:54 -0300 Reply-To: "Spanish & LATAM Books" Content-Transfer-Encoding: 8bit VERY IMPORTANT NOTICE: THIS IS A ONE TIME ONLY EMAIL, YOU DO NOT NEED TO REPLY TO IT IF NOT INTERESTED We are a small book publishing company, specialized in Spanish and Latin American Literature, History and Politics, both in Spanish and English. Our goal is to help Faculty members by re-publishing the currently out of print titles they would like to include in their respective course syllabus, and to supply their Students, as well as Academic Librarians and Booksellers, the suggested titles in HIGH QUALITY AND LOW COST US PRINTED editions. Our subscribers receive a short weekly comment on new books, and among other benefits they are entitled to suggest titles to be included in our current catalogue. IF YOU ARE INTERESTED IN OUR SERVICE please reply to this message with "subscribe" in the subject line and you will receive your free subscription instructions. On the other hand, if you consider this topic to be unrelated to your field of interest, please accept our apologies if you have been inconvenienced with unwanted mail from our company Thank you. Spanish & LATAM Books From Administration@computeradmin.org Tue May 4 20:06:13 2004 Return-Path: Received: from d206-116-227-89.bchsia.telus.net (d206-116-227-89.bchsia.telus.net [206.116.227.89]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i450687a021172; Tue, 4 May 2004 20:06:09 -0400 (EDT) Received: from d0kw.4920pj.org [125.215.210.54] by d206-116-227-89.bchsia.telus.net with ESMTP id AA6CB6992E8; Wed, 05 May 2004 00:59:54 +0000 Message-ID: From: "Admin" To: moler@netlib2.cs.utk.edu Subject: ADV: Attention All School Staff: Teachers, Students and Faculty Members: Date: Wed, 05 May 04 00:59:54 GMT X-Priority: 1 X-MSMail-Priority: High X-Mailer: Microsoft Outlook Express 5.00.2615.200 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary=".__CB._AF0497" This is a multi-part message in MIME format. --.__CB._AF0497 Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Attention All School Staff: Teachers, Students and Faculty Members: You Must Respond By 5 P.M. Thursday, May 6, 2004. Through a special arrangement, Avtech Direct is offering a limited allotment of BRAND NEW, top of-the-line, name-brand desktop computers at more than 50% off MSRP to all Teachers, Students,Faculty and Staff, who respond to this message before 5 P.M., Thursday, May 6, 2004. All desktop are brand-new, packed in their original boxes, and come with a full manufacturer's warranty plus a 100% satisfaction guarantee. These professional grade Desktops are fully equipped with 2004 next generation technology, making these the best performing computers money can buy. Avtech Direct is offering these feature rich, top performing Desktop Computers with the latest Intel technology at an amazing price to all who call: 1-800-884-9510 by 5 P.M. Thursday, May 6, 2004 The fast and powerful AT-2400 series Desktop features: * Intel 2.0Ghz Processor for amazing speed and performance * 128MB DDR RAM, --- Upgradeable to 1024 * 20 GB UDMA Hard Drive, --- Upgradeable to 80 GB * 52X CD-Rom Drive, --- Upgradeable to DVD/CDRW * 1.44 Floppy disk drive * Next Generation Technology * ATI Premium video and sound * Full Connectivity with Fax modem/Lan/IEE 1394/USB 2.0 * Soft Touch Keyboard and scroll mouse * Internet Ready * Network Ready * 1 Year parts and labor warranty * Priority customer service and tech support MSRP $699 ........................................ Your Cost $297 How to qualify: 1. You must be a Teacher, Student, Faculty or Staff Member: 2. All desktop computers will be available on a first come first serve basis. 3. You must call 1-800-884-9510 by 5 P.M. Thursday, May 6, 2004 and we will hold the desktops you request on will call. 4. You are not obligated in any way. 5. 100% Satisfaction Guaranteed. Call Avtech Direct 1-800-884-9510 before 5 P.M. Thursday, May 6, 2004 If you wish to unsubscribe from this list, please go to: http://www.computeradvice.org/unsubscribe.asp --.__CB._AF0497-- From tepbecik@myinksite.com Fri May 7 00:47:40 2004 Return-Path: Received: from WEIRD1 (cwsadmin93@weird1.weird.cx [66.135.32.15] (may be forged)) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i474ld7a002690 for ; Fri, 7 May 2004 00:47:40 -0400 (EDT) From: To: Subject: Inkjet, Laser & Copier Cartridges ~ Save up to 89% Date: Thu, 6 May 2004 21:54:35 -0700 Message-ID: <745501c433ef$64e1b390$0f208742@weird.cx> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft CDO for Windows 2000 Thread-Index: AcQz72ThurANMuRqT7SOg3ku/cb94g== Content-Class: urn:content-classes:message X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1409 Save up to 89% on Inkjet, Laser & Copier Supplies Quality Products, with 100% Satisfaction Guarantee Easy, Fast, Affordable Shipping Worldwide Plenty of Payment Options to Meet YOUR Needs! >> SPECIAL: FREE Shipping to US & Canada on Orders over $50 << Visit us on the web at http://www.myinksite.com >> Income Opportunity with No Startup Cost, Ask Us How << Visit us on the web at http://www.myinksite.com/DealerInfo/ If you wish to contact us please visit our web site. For instruction on how to be permanently remove from this distribution system go to http://www.myinksite.com/Remove/ From Administration@computeradmin.org Tue May 25 15:10:47 2004 Return-Path: Received: from CM-lconC1-67-233.cm.vtr.net (CM-lconC1-67-233.cm.vtr.net [200.83.67.233]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i4PJAa7a003810; Tue, 25 May 2004 15:10:42 -0400 (EDT) Received: from 17t.8fyjvf.com [230.211.180.148] by CM-lconC1-67-233.cm.vtr.net id T08qQm5ZruwA; Tue, 25 May 2004 19:04:54 -0100 Message-ID: <2i43$-q$0-1zz$xg-8wgs$b-g1g0$6@2p211.1y1711> From: "Admin" To: moler@netlib2.cs.utk.edu Subject: ADV: Attention All School Staff: Teachers, Students and Faculty Members: Date: Tue, 25 May 04 19:04:54 GMT X-Priority: 1 X-MSMail-Priority: High X-Mailer: The Bat! (v1.52f) Business MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="_A._5B.5CFC.F" This is a multi-part message in MIME format. --_A._5B.5CFC.F Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Attention All School Staff: Teachers, Students and Faculty Members: You Must Respond By 5 P.M. Thursday, May 27, 2004. Through a special arrangement, Avtech Direct is offering a limited allotment of BRAND NEW, top of-the-line, name-brand desktop computers at more than 50% off MSRP to all Teachers, Students,Faculty and Staff, who respond to this message before 5 P.M., Thursday, May 27, 2004. All desktop are brand-new, packed in their original boxes, and come with a full manufacturer's warranty plus a 100% satisfaction guarantee. These professional grade Desktops are fully equipped with 2004 next generation technology, making these the best performing computers money can buy. Avtech Direct is offering these feature rich, top performing Desktop Computers with the latest Intel technology at an amazing price to all who call: 1-800-884-9510 by 5 P.M. Thursday, May 27, 2004 The fast and powerful AT-2400 series Desktop features: * Intel 2.0Ghz Processor for amazing speed and performance * 128MB DDR RAM, --- Upgradeable to 1024 * 20 GB UDMA Hard Drive, --- Upgradeable to 80 GB * 52X CD-Rom Drive, --- Upgradeable to DVD/CDRW * 1.44 Floppy disk drive * Next Generation Technology * ATI Premium video and sound * Full Connectivity with Fax modem/Lan/IEE 1394/USB 2.0 * Soft Touch Keyboard and scroll mouse * Internet Ready * Network Ready * 1 Year parts and labor warranty * Priority customer service and tech support MSRP $699 ........................................ Your Cost $297 How to qualify: 1. You must be a Teacher, Student, Faculty or Staff Member: 2. All desktop computers will be available on a first come first serve basis. 3. You must call 1-800-884-9510 by 5 P.M. Thursday, May 27, 2004 and we will hold the desktops you request on will call. 4. You are not obligated in any way. 5. 100% Satisfaction Guaranteed. Call Avtech Direct 1-800-884-9510 before 5 P.M. Thursday, May 27, 2004 If you wish to unsubscribe from this list, please go to: http://www.computeradvice.org/unsubscribe.asp Avtech Direct 22647 Ventura Blvd., Suite 374 Woodland Hills, CA 91364 --_A._5B.5CFC.F-- From Administration@computeradmin.org Thu May 27 16:25:06 2004 Return-Path: Received: from 92125.virtua.com.br (92125.virtua.com.br [200.183.92.125]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i4RKOx7a005031; Thu, 27 May 2004 16:25:01 -0400 (EDT) Received: from tyj5w.9b9ggl.org [152.174.94.98] by 92125.virtua.com.br with ESMTP id <620632-93994>; Thu, 27 May 2004 21:20:55 +0000 Message-ID: <6$e-6wmk$-568$pt1i$7x@q9ni1f> From: "Admin" To: moler@netlib2.cs.utk.edu Subject: ADV: Attention All School Staff: Teachers, Students and Faculty Members: Date: Thu, 27 May 04 21:20:55 GMT X-Priority: 1 X-MSMail-Priority: High X-Mailer: eGroups Message Poster MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="BBA630F10.B" This is a multi-part message in MIME format. --BBA630F10.B Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Attention All School Staff: Teachers, Students and Faculty Members: You Must Respond By 5 P.M. Tuesday, June 1, 2004. Through a special arrangement, Avtech Direct is offering a limited allotment of BRAND NEW, top of-the-line, name-brand desktop computers at more than 50% off MSRP to all Teachers, Students,Faculty and Staff, who respond to this message before 5 P.M., Tuesday, June 1, 2004. All desktop are brand-new, packed in their original boxes, and come with a full manufacturer's warranty plus a 100% satisfaction guarantee. These professional grade Desktops are fully equipped with 2004 next generation technology, making these the best performing computers money can buy. Avtech Direct is offering these feature rich, top performing Desktop Computers with the latest Intel technology at an amazing price to all who call: 1-800-884-9510 by 5 P.M. Tuesday, June 1, 2004 The fast and powerful AT-2400 series Desktop features: * Intel 2.0Ghz Processor for amazing speed and performance * 128MB DDR RAM, --- Upgradeable to 1024 * 20 GB UDMA Hard Drive, --- Upgradeable to 80 GB * 52X CD-Rom Drive, --- Upgradeable to DVD/CDRW * 1.44 Floppy disk drive * Next Generation Technology * ATI Premium video and sound * Full Connectivity with Fax modem/Lan/IEE 1394/USB 2.0 * Soft Touch Keyboard and scroll mouse * Internet Ready * Network Ready * 1 Year parts and labor warranty * Priority customer service and tech support MSRP $699 ........................................ Your Cost $297 How to qualify: 1. You must be a Teacher, Student, Faculty or Staff Member: 2. All desktop computers will be available on a first come first serve basis. 3. You must call 1-800-884-9510 by 5 P.M. Tuesday, June 1, 2004 and we will hold the desktops you request on will call. 4. You are not obligated in any way. 5. 100% Satisfaction Guaranteed. Call Avtech Direct 1-800-884-9510 before 5 P.M. Tuesday, June 1, 2004 If you wish to unsubscribe from this list, please go to: http://www.computeradvice.org/unsubscribe.asp Avtech Direct 22647 Ventura Blvd., Suite 374 Woodland Hills, CA 91364 --BBA630F10.B-- From Administration@computeradmin.org Wed Jun 9 00:41:24 2004 Return-Path: Received: from YahooBB219034136012.bbtec.net (YahooBB219034136012.bbtec.net [219.34.136.12]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i594fA7a013604; Wed, 9 Jun 2004 00:41:14 -0400 (EDT) Received: from azqnj.z3879.org [215.209.251.144] by YahooBB219034136012.bbtec.net with ESMTP id 44A85CFEC4F for ; Wed, 09 Jun 2004 11:41:04 +0600 Message-ID: <268$nmg4rl-7xm71b$$i@iqb.x390> From: "Admin" To: moler@netlib2.cs.utk.edu Subject: ADV: Attention All School Staff: Teachers, Students and Faculty Members: Date: Wed, 09 Jun 04 11:41:04 GMT X-Priority: 1 X-MSMail-Priority: High X-Mailer: Microsoft Outlook Express 5.00.2919.6700 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="33_D_.C_C_.1AC_BC" This is a multi-part message in MIME format. --33_D_.C_C_.1AC_BC Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Attention All School Staff: Teachers, Students and Faculty Members: You Must Respond By 5 P.M. Thursday, June 10, 2004. Through a special arrangement, Avtech Direct is offering a limited allotment of BRAND NEW, top of-the-line, name-brand desktop computers at more than 50% off MSRP to all Teachers, Students,Faculty and Staff, who respond to this message before 5 P.M., Thursday, June 10, 2004. All desktop are brand-new, packed in their original boxes, and come with a full manufacturer's warranty plus a 100% satisfaction guarantee. These professional grade Desktops are fully equipped with 2004 next generation technology, making these the best performing computers money can buy. Avtech Direct is offering these feature rich, top performing Desktop Computers with the latest Intel technology at an amazing price to all who call: 1-800-884-9510 by 5 P.M. Thursday, June 10, 2004 The fast and powerful AT-2400 series Desktop features: * Intel 2.0Ghz Processor for amazing speed and performance * 128MB DDR RAM, --- Upgradeable to 1024 * 20 GB UDMA Hard Drive, --- Upgradeable to 80 GB * 52X CD-Rom Drive, --- Upgradeable to DVD/CDRW * 1.44 Floppy disk drive * Next Generation Technology * ATI Premium video and sound * Full Connectivity with Fax modem/Lan/IEE 1394/USB 2.0 * Soft Touch Keyboard and scroll mouse * Internet Ready * Network Ready * 1 Year parts and labor warranty * Priority customer service and tech support MSRP $699 ........................................ Your Cost $297 How to qualify: 1. You must be a Teacher, Student, Faculty or Staff Member: 2. All desktop computers will be available on a first come first serve basis. 3. You must call 1-800-884-9510 by 5 P.M. Thursday, June 10, 2004 and we will hold the desktops you request on will call. 4. You are not obligated in any way. 5. 100% Satisfaction Guaranteed. Call Avtech Direct 1-800-884-9510 before 5 P.M. Thursday, June 10, 2004 If you wish to unsubscribe from this list, please go to: http://www.computeradvice.org/unsubscribe.asp Avtech Direct 22647 Ventura Blvd., Suite 374 Woodland Hills, CA 91364 --33_D_.C_C_.1AC_BC-- From Administration@computeradmin.org Tue Jun 22 04:43:58 2004 Return-Path: Received: from 160.36.58.108 ([211.96.31.234]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i5M8h77a005900; Tue, 22 Jun 2004 04:43:19 -0400 (EDT) Received: from 3u2.dsvh376.org (HELO 4xbsx) ([81.37.134.228]) by 160.36.58.108; Tue, 22 Jun 2004 15:45:41 +0600 Message-ID: <285n2x7$3eb$-3v9yp042@29dvmksyt> From: "Admin" To: moler@netlib2.cs.utk.edu Subject: ADV: Attention All School Staff: Teachers, Students and Faculty Members: Date: Tue, 22 Jun 04 15:45:41 GMT X-Priority: 1 X-MSMail-Priority: High X-Mailer: Microsoft Outlook Express 5.50.4522.1200 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="CA75A33BCC38D8_F46.C2" This is a multi-part message in MIME format. --CA75A33BCC38D8_F46.C2 Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Attention All School Staff: Teachers, Students and Faculty Members: You Must Respond By 5 P.M. Wednesday, June 23, 2004. Through a special arrangement, Avtech Direct is offering a limited allotment of BRAND NEW, top of-the-line, name-brand desktop computers at more than 50% off MSRP to all Teachers, Students,Faculty and Staff, who respond to this message before 5 P.M., Wednesday, June 23, 2004. All desktop are brand-new, packed in their original boxes, and come with a full manufacturer's warranty plus a 100% satisfaction guarantee. These professional grade Desktops are fully equipped with 2004 next generation technology, making these the best performing computers money can buy. Avtech Direct is offering these feature rich, top performing Desktop Computers with the latest Intel technology at an amazing price to all who call: 1-800-884-9510 by 5 P.M. Wednesday, June 23, 2004 The fast and powerful AT-2400 series Desktop features: * Intel 2.0Ghz Processor for amazing speed and performance * 128MB DDR RAM, --- Upgradeable to 1024 * 20 GB UDMA Hard Drive, --- Upgradeable to 80 GB * 52X CD-Rom Drive, --- Upgradeable to DVD/CDRW * 1.44 Floppy disk drive * Next Generation Technology * ATI Premium video and sound * Full Connectivity with Fax modem/Lan/IEE 1394/USB 2.0 * Soft Touch Keyboard and scroll mouse * Internet Ready * Network Ready * 1 Year parts and labor warranty * Priority customer service and tech support MSRP $699 ........................................ Your Cost $297 How to qualify: 1. You must be a Teacher, Student, Faculty or Staff Member: 2. All desktop computers will be available on a first come first serve basis. 3. You must call 1-800-884-9510 by 5 P.M. Wednesday, June 23, 2004 and we will hold the desktops you request on will call. 4. You are not obligated in any way. 5. 100% Satisfaction Guaranteed. Call Avtech Direct 1-800-884-9510 before 5 P.M. Wednesday, June 23, 2004 Visit our website at http://www.avtechdirectcomputers.com If you wish to unsubscribe from this list, please go to: http://www.computeradvice.org/unsubscribe.asp Avtech Direct 22647 Ventura Blvd., Suite 374 Woodland Hills, CA 91364 --CA75A33BCC38D8_F46.C2-- From Administration@computeradmin.org Wed Jun 30 03:31:47 2004 Return-Path: Received: from c-24-14-76-181.client.comcast.net (c-24-14-76-181.client.comcast.net [24.14.76.181]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i5U7VX7a022776; Wed, 30 Jun 2004 03:31:36 -0400 (EDT) Received: from gwx.c2x0ifm.net [11.237.182.242] by c-24-14-76-181.client.comcast.net with ESMTP id 25461059; Wed, 30 Jun 2004 14:28:36 +0600 Message-ID: <3m$80lqps-a4u81gs$mh-sb@jcb.1i1lh8> From: "Admin" To: moler@netlib2.cs.utk.edu Subject: ADV: Attention All School Staff: Teachers, Students and Faculty Members: Date: Wed, 30 Jun 04 14:28:36 GMT X-Priority: 1 X-MSMail-Priority: High X-Mailer: Microsoft Outlook Express 5.50.4133.2400 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="9B_F0.2_2." This is a multi-part message in MIME format. --9B_F0.2_2. Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Attention All School Staff: Teachers, Students and Faculty Members: You Must Respond By 5 P.M. Thursday, July 1, 2004. Through a special arrangement, Avtech Direct is offering a limited allotment of BRAND NEW, top of-the-line, name-brand desktop computers at more than 50% off MSRP to all Teachers, Students,Faculty and Staff, who respond to this message before 5 P.M., Thursday, July 1, 2004. All desktop are brand-new, packed in their original boxes, and come with a full manufacturer's warranty plus a 100% satisfaction guarantee. These professional grade Desktops are fully equipped with 2004 next generation technology, making these the best performing computers money can buy. Avtech Direct is offering these feature rich, top performing Desktop Computers with the latest Intel technology at an amazing price to all who call: 1-800-884-9510 by 5 P.M. Thursday, July 1, 2004 The fast and powerful AT-2400 series Desktop features: * Intel 2.0Ghz Processor for amazing speed and performance * 128MB DDR RAM, --- Upgradeable to 1024 * 20 GB UDMA Hard Drive, --- Upgradeable to 80 GB * 52X CD-Rom Drive, --- Upgradeable to DVD/CDRW * 1.44 Floppy disk drive * Next Generation Technology * ATI Premium video and sound * Full Connectivity with Fax modem/Lan/IEE 1394/USB 2.0 * Soft Touch Keyboard and scroll mouse * Internet Ready * Network Ready * 1 Year parts and labor warranty * Priority customer service and tech support MSRP $699 ........................................ Your Cost $297 How to qualify: 1. You must be a Teacher, Student, Faculty or Staff Member: 2. All desktop computers will be available on a first come first serve basis. 3. You must call 1-800-884-9510 by 5 P.M. Thursday, July 1, 2004 and we will hold the desktops you request on will call. 4. You are not obligated in any way. 5. 100% Satisfaction Guaranteed. Call Avtech Direct 1-800-884-9510 before 5 P.M. Thursday, July 1, 2004 Visit our website at http://www.avtechdirect-education.com If you wish to unsubscribe from this list, please go to: http://www.computeradvice.org/unsubscribe.asp Avtech Direct 22647 Ventura Blvd., Suite 374 Woodland Hills, CA 91364 --9B_F0.2_2.-- From Administration@computeradmin.org Tue Jul 13 08:36:05 2004 Return-Path: Received: from RAUL ([62.85.203.19]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i6DCZj7a016599; Tue, 13 Jul 2004 08:35:55 -0400 (EDT) Received: from xhvt4.cr4bpn.org [190.227.64.250] by RAUL with ESMTP id B258335E9DE; Tue, 13 Jul 2004 12:32:10 -0100 Message-ID: From: "Admin" To: moler@netlib2.cs.utk.edu Subject: ADV: Attention All School Staff: Teachers, Students and Faculty Members: Date: Tue, 13 Jul 04 12:32:10 GMT X-Priority: 1 X-MSMail-Priority: High X-Mailer: The Bat! (v1.52f) Business MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="9A23.8.96FBE5D3E8C3.BCA" This is a multi-part message in MIME format. --9A23.8.96FBE5D3E8C3.BCA Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Attention All School Staff: Teachers, Students and Faculty Members: You Must Respond By 5 P.M. Wednesday, July 14, 2004. Through a special arrangement, Avtech Direct is offering a limited allotment of BRAND NEW, top of-the-line, name-brand desktop computers at more than 50% off MSRP to all Teachers, Students,Faculty and Staff, who respond to this message before 5 P.M., Wednesday, July 14, 2004. All desktop are brand-new, packed in their original boxes, and come with a full manufacturer's warranty plus a 100% satisfaction guarantee. These professional grade Desktops are fully equipped with 2004 next generation technology, making these the best performing computers money can buy. Avtech Direct is offering these feature rich, top performing Desktop Computers with the latest Intel technology at an amazing price to all who call: 1-800-884-9510 by 5 P.M. Wednesday, July 14, 2004 The fast and powerful AT-2400 series Desktop features: * Intel 2.0Ghz Processor for amazing speed and performance * 128MB DDR RAM, --- Upgradeable to 1024 * 20 GB UDMA Hard Drive, --- Upgradeable to 80 GB * 52X CD-Rom Drive, --- Upgradeable to DVD/CDRW * 1.44 Floppy disk drive * Next Generation Technology * ATI Premium video and sound * Full Connectivity with Fax modem/Lan/IEE 1394/USB 2.0 * Soft Touch Keyboard and scroll mouse * Internet Ready * Network Ready * 1 Year parts and labor warranty * Priority customer service and tech support MSRP $699 ........................................ Your Cost $297 How to qualify: 1. You must be a Teacher, Student, Faculty or Staff Member: 2. All desktop computers will be available on a first come first serve basis. 3. You must call 1-800-884-9510 by 5 P.M. Wednesday, July 14, 2004 and we will hold the desktops you request on will call. 4. You are not obligated in any way. 5. 100% Satisfaction Guaranteed. Call Avtech Direct 1-800-884-9510 before 5 P.M. Wednesday, July 14, 2004 Visit our website at http://www.avtechdirectcomputers.com If you wish to unsubscribe from this list, please go to: http://www.computeradvice.org/unsubscribe.asp Avtech Direct 22647 Ventura Blvd., Suite 374 Woodland Hills, CA 91364 --9A23.8.96FBE5D3E8C3.BCA-- From nobody@server.designcrest.com Tue Jul 20 23:14:25 2004 Return-Path: Received: from server.designcrest.com ([69.57.128.35]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with ESMTP id i6L3EP7a008122; Tue, 20 Jul 2004 23:14:25 -0400 (EDT) Received: from nobody by server.designcrest.com with local (Exim 4.24) id 1Bn7RT-0002mb-OY; Tue, 20 Jul 2004 22:06:51 -0500 To: wangqinly@3xl.net Subject: SINCERE ASSOCIATE REQUIRED From: wang-vu X-Priority: 3 (Normal) CC: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Mailer: RLSP Mailer Message-Id: Date: Tue, 20 Jul 2004 22:06:51 -0500 X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - server.designcrest.com X-AntiAbuse: Original Domain - netlib2.cs.utk.edu X-AntiAbuse: Originator/Caller UID/GID - [99 99] / [47 12] X-AntiAbuse: Sender Address Domain - server.designcrest.com HANG SENG BANK LTD. DES VOEUX RD. BRANCH, CENTRAL HONG KONG, HONK KONG. E-MAIL: Dear Sir, Let me start by introducing myself. I am Mr. Wang Qin credit officer of the Hang Seng Bank Ltd. I have a concealed business suggestion for you.Before the U.S and Iraqi war our client General.Ibrahim Moussa who was with the Iraqi forces and also business man made a numbered fixed deposit for 18 calendar months, with a value of Twenty millions Five Hundred Thousand United State Dollars only in my branch. Upon maturity several notice was sent to him, even during the war early this year.Again after the war another notification was sent and still no response came from him. We later find out that the General and his family had been killed during the war in bomb blast that hit their home. After further investigation it was also discovered that Gen. Ibrahim Moussa did not declare any next of kin in his official papers including the paper work of his bank deposit. And he also confided in me the last time he was at my office that no one except me knew of his deposit in my bank. So, Twenty millions Five Hundred Thousand United State Dollars is still lying in my bank and no one will ever come forward to claim it. What bothers me most is that according to the to the laws of my country at the expiration 3 years the funds will revert to the ownership of the Hong Kong Government if nobody applies to claim the funds. Against this backdrop, my suggestion to you is that I will like you as a foreigner to stand as the next of kin to Gen. Ibrahim Moussa so that you will be able to receive his funds. WHAT IS TO BE DONE: I want you to know that I have had everything planned out so that we shall come out successful. I have contacted an attorney that will prepare the necessary document that will back you up as the next of kin to Gen. Ibrahim Moussa, all that is required from you at this stage is for you to provide me with your Full Names and Address so that the attorney can commence his job. After you have been made the next of kin, the attorney will also fill in for claims on your behalf and secure the necessary approval and letter of probate in your favor for the move of the funds to an account that will be provided by you. There is no risk involved at all in the matter as we are going adopt a legalized method and the attorney will prepare all the necessary documents. Please endeavor to observe utmost discretion in all matters concerning this issue. Once the funds have been transferred to your nominated bank account we shall share in the ratio of 70% for me, 25% for you and 5% for any expenses incurred during the course of this operation. Should you be interested please send me your private phone and fax numbers for easy communication and I will provide you with more details of this operation. Your earliest response to this letter will be appreciated. Kind Regards, Mr. Wang Qin ___________________________________________________________________________ Mail sent from WebMail service at PHP-Nuke Powered Site Dragons Lodge RP Siter - http://dlcomics.com From nobody@server.designcrest.com Tue Jul 20 23:28:22 2004 Return-Path: Received: from server.designcrest.com ([69.57.128.35]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with ESMTP id i6L3Rn7a008456; Tue, 20 Jul 2004 23:27:50 -0400 (EDT) Received: from nobody by server.designcrest.com with local (Exim 4.24) id 1Bn7ed-0003k1-Da; Tue, 20 Jul 2004 22:20:27 -0500 To: wangqinly@3xl.net Subject: SINCERE ASSOCIATE REQUIRED From: wang-vu X-Priority: 3 (Normal) CC: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Mailer: RLSP Mailer Message-Id: Date: Tue, 20 Jul 2004 22:20:27 -0500 X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - server.designcrest.com X-AntiAbuse: Original Domain - netlib2.cs.utk.edu X-AntiAbuse: Originator/Caller UID/GID - [99 99] / [47 12] X-AntiAbuse: Sender Address Domain - server.designcrest.com HANG SENG BANK LTD. DES VOEUX RD. BRANCH, CENTRAL HONG KONG, HONK KONG. E-MAIL: Dear Sir, Let me start by introducing myself. I am Mr. Wang Qin credit officer of the Hang Seng Bank Ltd. I have a concealed business suggestion for you.Before the U.S and Iraqi war our client General.Ibrahim Moussa who was with the Iraqi forces and also business man made a numbered fixed deposit for 18 calendar months, with a value of Twenty millions Five Hundred Thousand United State Dollars only in my branch. Upon maturity several notice was sent to him, even during the war early this year.Again after the war another notification was sent and still no response came from him. We later find out that the General and his family had been killed during the war in bomb blast that hit their home. After further investigation it was also discovered that Gen. Ibrahim Moussa did not declare any next of kin in his official papers including the paper work of his bank deposit. And he also confided in me the last time he was at my office that no one except me knew of his deposit in my bank. So, Twenty millions Five Hundred Thousand United State Dollars is still lying in my bank and no one will ever come forward to claim it. What bothers me most is that according to the to the laws of my country at the expiration 3 years the funds will revert to the ownership of the Hong Kong Government if nobody applies to claim the funds. Against this backdrop, my suggestion to you is that I will like you as a foreigner to stand as the next of kin to Gen. Ibrahim Moussa so that you will be able to receive his funds. WHAT IS TO BE DONE: I want you to know that I have had everything planned out so that we shall come out successful. I have contacted an attorney that will prepare the necessary document that will back you up as the next of kin to Gen. Ibrahim Moussa, all that is required from you at this stage is for you to provide me with your Full Names and Address so that the attorney can commence his job. After you have been made the next of kin, the attorney will also fill in for claims on your behalf and secure the necessary approval and letter of probate in your favor for the move of the funds to an account that will be provided by you. There is no risk involved at all in the matter as we are going adopt a legalized method and the attorney will prepare all the necessary documents. Please endeavor to observe utmost discretion in all matters concerning this issue. Once the funds have been transferred to your nominated bank account we shall share in the ratio of 70% for me, 25% for you and 5% for any expenses incurred during the course of this operation. Should you be interested please send me your private phone and fax numbers for easy communication and I will provide you with more details of this operation. Your earliest response to this letter will be appreciated. Kind Regards, Mr. Wang Qin ___________________________________________________________________________ Mail sent from WebMail service at PHP-Nuke Powered Site Dragons Lodge RP Siter - http://dlcomics.com From Administrator@computeradmin.org Fri Jul 23 04:57:57 2004 Return-Path: Received: from 160.36.58.108 ([218.201.129.131]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i6N8vq7a022413; Fri, 23 Jul 2004 04:57:54 -0400 (EDT) Received: from 8p.v09uvgk.com ([185.55.222.56]) by 160.36.58.108 with ESMTP id <678814-10347>; Fri, 23 Jul 2004 12:59:02 +0300 Message-ID: <2ch77w4c-6t5$75h@z99xj> From: "Admin" To: moler@netlib2.cs.utk.edu Subject: Staff Bulletin Date: Fri, 23 Jul 04 12:59:02 GMT X-Priority: 1 X-MSMail-Priority: High X-Mailer: Microsoft Outlook Express 6.00.2600.0000 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="_2.E47_56._F__F8" This is a multi-part message in MIME format. --_2.E47_56._F__F8 Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Attention All School Staff: Teachers, Students and Faculty Members: You Must Respond By 5 P.M. Monday, July 26, 2004. Through a special arrangement, Avtech Direct is offering a limited allotment of BRAND NEW, top of-the-line, name-brand desktop computers at more than 50% off MSRP to all Teachers, Students,Faculty and Staff, who respond to this message before 5 P.M., Monday, July 26, 2004. All desktop are brand-new, packed in their original boxes, and come with a full manufacturer's warranty plus a 100% satisfaction guarantee. These professional grade Desktops are fully equipped with 2004 next generation technology, making these the best performing computers money can buy. Avtech Direct is offering these feature rich, top performing Desktop Computers with the latest Intel technology at an amazing price to all who call: 1-800-884-9510 by 5 P.M. Monday, July 26, 2004 The fast and powerful AT-2400 series Desktop features: * Intel 2.0Ghz Processor for amazing speed and performance * 128MB DDR RAM, --- Upgradeable to 1024 * 20 GB UDMA Hard Drive, --- Upgradeable to 80 GB * 52X CD-Rom Drive, --- Upgradeable to DVD/CDRW * 1.44 Floppy disk drive * Next Generation Technology * ATI Premium video and sound * Full Connectivity with Fax modem/Lan/IEE 1394/USB 2.0 * Soft Touch Keyboard and scroll mouse * Internet Ready * Network Ready * 1 Year parts and labor warranty * Priority customer service and tech support MSRP $699 ........................................ Your Cost $297 How to qualify: 1. You must be a Teacher, Student, Faculty or Staff Member: 2. All desktop computers will be available on a first come first serve basis. 3. You must call 1-800-884-9510 by 5 P.M. Monday, July 26, 2004 and we will hold the desktops you request on will call. 4. You are not obligated in any way. 5. 100% Satisfaction Guaranteed. Call Avtech Direct 1-800-884-9510 before 5 P.M. Monday, July 26, 2004 Visit our website at http://www.avtechdirect-education.com If you wish to unsubscribe from this list, please go to: http://www.computeradvice.org/unsubscribe.asp Avtech Direct 22647 Ventura Blvd., Suite 374 Woodland Hills, CA 91364 --_2.E47_56._F__F8-- From Administrator@computeradmin.org Thu Aug 5 02:14:04 2004 Return-Path: Received: from SUNQIANG ([222.32.78.29]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i756Dx7a020490; Thu, 5 Aug 2004 02:14:01 -0400 (EDT) Received: from stw0.qz2o.org [50.105.213.49] by SUNQIANG id pH29j27062Sv for ; Thu, 05 Aug 2004 06:16:35 -0100 Message-ID: From: "Admin" To: moler@netlib2.cs.utk.edu Subject: Staff Bulletin Date: Thu, 05 Aug 04 06:16:35 GMT X-Priority: 1 X-MSMail-Priority: High X-Mailer: Microsoft Outlook Express 6.00.2600.0000 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="__7DF266E6E__BE4B86E._A6" This is a multi-part message in MIME format. --__7DF266E6E__BE4B86E._A6 Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Attention All Teachers, Students, Faculty, and Staff Members: You Must Respond By 5 P.M. Friday, August 6, 2004. Through a special arrangement, Avtech Direct is offering a limited allotment of BRAND NEW, top of-the-line, name-brand desktop computers at more than 50% off MSRP to all teachers, students, and staff members who respond to this message before 5 P.M., Friday, August 6, 2004. All desktop are brand-new, packed in their original boxes, and come with a full manufacturer's warranty plus a 100% satisfaction guarantee. These professional grade Desktops are fully equipped with 2004 next generation technology, making these the best performing computers money can buy. Avtech Direct is offering these feature rich, top performing Desktop Computers with the latest Intel technology at an amazing price to all who call: 1-800-884-9510 by 5 P.M. Friday, August 6, 2004 The fast and powerful AT-2400 series Desktop features: * Intel 2.0Ghz Processor for amazing speed and performance * 128MB DDR RAM, --- Upgradeable to 1024 * 20 GB UDMA Hard Drive, --- Upgradeable to 80 GB * 52X CD-Rom Drive, --- Upgradeable to DVD/CDRW * 1.44 Floppy disk drive * Next Generation Technology * ATI Premium video and sound * Full Connectivity with Fax modem/Lan/IEE 1394/USB 2.0 * Soft Touch Keyboard and scroll mouse * Internet Ready * Network Ready * 1 Year parts and labor warranty * Priority customer service and tech support MSRP $699 ........................................ Your Cost $297 How to qualify: 1. You must be a Teacher, Student, Faculty or Staff Member: 2. All desktop computers will be available on a first come first serve basis. 3. You must call 1-800-884-9510 by 5 P.M. Friday, August 6, 2004 and we will hold the desktops you request on will call. 4. You are not obligated in any way. 5. 100% Satisfaction Guaranteed. Call Avtech Direct 1-800-884-9510 before 5 P.M. Friday, August 6, 2004 Visit our website at http://www.avtechdirect-education.com If you wish to unsubscribe from this list, please go to: http://www.computeradvice.org/unsubscribe.asp Avtech Direct 22647 Ventura Blvd., Suite 374 Woodland Hills, CA 91364 --__7DF266E6E__BE4B86E._A6-- From admin@computeradmin.org Wed Aug 11 03:11:12 2004 Return-Path: Received: from c-67-170-240-122.client.comcast.net (c-67-170-240-122.client.comcast.net [67.170.240.122]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i7B7B57a000092; Wed, 11 Aug 2004 03:11:07 -0400 (EDT) Received: from xx.ntrjh.org (HELO f24g8vv) [3.7.64.54] by c-67-170-240-122.client.comcast.net id iI1lq8B9lGd1 for ; Wed, 11 Aug 2004 13:11:55 +0500 Message-ID: From: "Administrator" To: moler@netlib2.cs.utk.edu Subject: Staff Bulletin Date: Wed, 11 Aug 04 13:11:55 GMT X-Priority: 1 X-MSMail-Priority: High X-Mailer: Microsoft Outlook Express 5.50.4522.1200 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="62E_C51CA478_9..8C307__." This is a multi-part message in MIME format. --62E_C51CA478_9..8C307__. Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Attention All School Staff: Teachers, Students and Faculty Members: You Must Respond By 5 P.M. Thursday, August 12, 2004. Through a special arrangement, Avtech Direct is offering a limited allotment of BRAND NEW, top of-the-line, name-brand Notebook computers at more than 50% off MSRP to all Teachers, Students,Faculty and Staff, who respond to this message before 5 P.M., Thursday, August 12, 2004. All Notebooks are brand-new, packed in their original boxes, and come with a full manufacturer's warranty plus a 100% satisfaction guarantee. These professional grade Notebooks are lightweight and fully Equipped with the Next Generation WiFi technology, making these The very best performing computers that money can buy. Avtech Computers is offering these feature rich, top performing Notebooks With the latest Wireless technology at an amazing price To all who call: : 1-800-318-1388 by 5 P.M. Thursday, August 12, 2004 AT-1000S Notebook Series- The ultimate combination of innovative mobile features and practical value= . * 1 Giga Pro Transmeta Crusoe Mobile CPU for incredible power * 128 MB Super Fast DDR SDRAM at 400MHz (Upgradeable to 256 MB) * 20 GB ATA100 Hard Drive (Upgradeable to 30GB ATA100 Hard Drive) * CD-Rom Drive (Upgradeable to DVD/CDRW) * Next Generation 802.11 Wireless Technology for total WiFi freedom * 14.1 XGA TFT Ultra Sharp Liquid Display * Premium Sound and Video * Enhanced Front Stereo Speakers * Total Connectivity 56K V.90 Performance Pro Fax Modem * Integrated High Performance Intel=FFFFFFAE 10/100 LAN Gigabit Ethernet * TV-Out Port with S-Video and USB 2.0 Ports * Soft Touch Keyboard * Programmable Synaptics Touchpad Mouse * Internet and Network Ready * Extra Long Life Battery Pack * Full Range 160Watt AC Adapter * 1 Year parts and labor warranty * Priority Customer Service and Toll Free Technical Support * Latest Qualified Drivers Installed MSRP $1299 .......................................... Your Cost $597 Avtech/ASUS AT-2500 The AT-2500, using the latest technology by ASUS, achieves a perfect blend= of style and performance at a remarkable price. Quantities for this super fast and extremely reliable notebook computer are strictly limited to supplies on hand. The AT-2500 Notebook by ASUS Features: * 2.0 GHz Intel Processor * 15.0" SXGA+ active matrix color Display for an amazing picture * ASUS ADTD II heat dissipation technology for a quieter computer that consumes less power * 20 GB Hard Drive with Ultra DMA100 support. (Upgradeable to/80 GB) * 256 MB Dual Channel DDR SDRam (Upgradeable to 1024MB) * CDRW / DVD combo drive (Upgradeable to DVD RW) * SoundBlaster Pro audio Build-in stereo speakers * 5 USB 2.0 ports, 56K V90 Fax Modem, 10/100BaseT PCI LAN on board * Serial port, Parallel port, TV Out, S-Video, IEEE 1394 port Fire wire, SIR port, PCMCIA 2.1 * Full size 88key Keyboard with MS-Windows function keys * Built-in Touch pad pointing device ASUS Power4 Gear Technology effectively adjusts CPU & LCD to maximize battery life * 8 cell Long Life Smart Battery * Advanced Asus Step power management technology * 2 year ASUS Warranty MSRP $1999 .......................................... Your Cost $997 How to qualify: 1. You must be a Teacher, Student, Faculty or Staff Member. 2. All desktop computers will be available on a first come first serve basis. 3. You must call 1-800-318-1388 by 5 P.M. Thursday, August 12, 2004 and we will hold the computers you request on will call. 4. You are not obligated in any way. 5. 100% Satisfaction Guaranteed. Call Avtech Direct 1-800-318-1388 before 5 P.M. Thursday, August 12, 2004 Visit our website at http://www.avtechdirectcomputers.com If you wish to unsubscribe from this list, please go to: http://www.computeradvice.org/unsubscribe.asp Avtech Direct 22647 Ventura Blvd., Suite 374 Woodland Hills, CA 91364 --62E_C51CA478_9..8C307__.-- From admin@computeradmin.org Sun Aug 15 23:59:05 2004 Return-Path: Received: from 160.36.58.108 ([222.32.78.29]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i7G3wi7a022871; Sun, 15 Aug 2004 23:58:47 -0400 (EDT) Received: from qu1t.tgh49b6.com [56.184.53.215] by 160.36.58.108 id 6m5F2IQE52QU; Mon, 16 Aug 2004 07:58:30 +0300 Message-ID: From: "Administrator" To: moler@netlib2.cs.utk.edu Subject: Staff Bulletin Date: Mon, 16 Aug 04 07:58:30 GMT X-Priority: 1 X-MSMail-Priority: High X-Mailer: Microsoft Outlook Express 6.00.2600.0000 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="F_A_80D68BB.D0D_4" This is a multi-part message in MIME format. --F_A_80D68BB.D0D_4 Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Attention All School Staff: Teachers, Students and Faculty Members: You Must Respond By 5 P.M. Tuesday, August 17, 2004. Through a special arrangement, Avtech Direct is offering a limited allotment of BRAND NEW, top of-the-line, name-brand desktop computers at more than 50% off MSRP to all Teachers, Students,Faculty and Staff, who respond to thismessage before 5 P.M., Tuesday, August 17, 2004. All desktop are brand-new, packed in their original boxes, and come with a full manufacturer's warranty plus a 100% satisfaction guarantee. These professional grade Desktops are fully equipped with 2004 next generation technology, making these the best performing computers money can buy. Avtech Direct is offering these feature rich, top performing Desktop Computers with the latest Intel technology at an amazing price to all who call: 1-800-884-9510 by 5 P.M. Tuesday, August 17, 2004 The fast and powerful AT-2400 series Desktop features: * Intel 2.0Ghz Processor for amazing speed and performance * 128MB DDR RAM, --- Upgradeable to 1024 * 20 GB UDMA Hard Drive, --- Upgradeable to 80 GB * 52X CD-Rom Drive, --- Upgradeable to DVD/CDRW * 1.44 Floppy disk drive * Next Generation Technology * ATI Premium video and sound * Full Connectivity with Fax modem/Lan/IEE 1394/USB 2.0 * Soft Touch Keyboard and scroll mouse * Internet Ready * Network Ready * 1 Year parts and labor warranty * Priority customer service and tech support MSRP $699 ........................................ Your Cost $297 How to qualify: 1. You must be a Teacher, Student, Faculty or Staff Member. 2. All desktop computers will be available on a first come first serve basis. 3. You must call 1-800-884-9510 by 5 P.M. Tuesday, August 17, 2004 and we will hold the desktops you request on will call. 4. You are not obligated in any way. 5. 100% Satisfaction Guaranteed. Call Avtech Direct 1-800-884-9510 before 5 P.M. Tuesday, August 17, 2004 Visit our website at http://www.avtechdirectcomputers.com If you wish to unsubscribe from this list, please go to: http://www.computeradvice.org/unsubscribe.asp Avtech Direct 22647 Ventura Blvd., Suite 374 Woodland Hills, CA 91364 --F_A_80D68BB.D0D_4-- From marketing30009@freemail.soim.com Thu Aug 19 02:07:30 2004 Return-Path: Received: from freemail.soim.com ([218.6.12.67]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i7J67R7a001950 for ; Thu, 19 Aug 2004 02:07:28 -0400 (EDT) Message-Id: <200408190607.i7J67R7a001950@netlib2.cs.utk.edu> From: "Mary" To: Subject: E-Marketing Sender: "Mary" Mime-Version: 1.0 Content-Type: text/html; charset="ISO-8859-1" Date: Thu, 19 Aug 2004 14:07:13 +0800 Reply-To: "Mary" Content-Transfer-Encoding: 8bit E

We offer a complete E-Marketing solution with quality service and 
the lowest prices. The result is that you will enjoy more success.

Professional offer:

1. Email addresses according to your requirements. 

2. Send emailing according to your requirements. 

3. Dedicated Server & Web Hosting solutions.

For more details:  www.3807.com

We will help you get more business opportunities.

Ms Mary
www.3807.com
info@3807.com
 Customer Support

*******************************************************************************
Take off Click here: Ma@USA.net?subject=19
*******************************************************************************

From Sendinglist@mail.runan.ha.cn Sat Aug 21 11:27:10 2004 Return-Path: Received: from mail.runan.ha.cn ([219.146.151.95]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i7LFR17a002489 for ; Sat, 21 Aug 2004 11:27:09 -0400 (EDT) Message-Id: <200408211527.i7LFR17a002489@netlib2.cs.utk.edu> From: "Kim" To: Subject: To help you success Sender: "Kim" Mime-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-1" Date: Sat, 21 Aug 2004 23:26:29 -0700 Reply-To: "Kim" Content-Transfer-Encoding: 8bit Email is the best growing marketing tool. We offer E-Marketing with quality service. 1. Target Email Addresses We can provide target e-mail addresses you need, which are compiled only on your order. We will customize your customer email addresses. * We have millions of email addresses in a wide variety of categories. 2. Send out Target Emails for you We can send your email message to your target customers! We will customize your email addresses and send your message for you. * We can Bullet Proof your Web Site $ dedicated server. Looking forward to serving you. Regards! Kim www.30200.com Sales@30200.com Take your address: Http://63.307.93.26/index.html From admin@computeradmin.org Tue Aug 24 21:21:07 2004 Return-Path: Received: from atl-30-a-116.atl.dsl.cerfnet.com (atl-30-a-116.atl.dsl.cerfnet.com [63.242.196.116]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i7P1Kq7a009181; Tue, 24 Aug 2004 21:21:00 -0400 (EDT) Received: from eg73.hmacn6.net [28.67.212.131] by atl-30-a-116.atl.dsl.cerfnet.com with SMTP for ; Wed, 25 Aug 2004 02:22:24 +0000 Message-ID: From: "Administrator" To: moler@netlib2.cs.utk.edu Subject: Staff Bulletin Date: Wed, 25 Aug 04 02:22:24 GMT X-Priority: 1 X-MSMail-Priority: High X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="6901_30B._" This is a multi-part message in MIME format. --6901_30B._ Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Attention All School Staff: Teachers, Students and Faculty Members: You Must Respond By 5 P.M. Thursday, August 26, 2004. Through a special arrangement, Avtech Direct is offering a limited allotment of BRAND NEW, top of-the-line, name-brand desktop computers at more than 50% off MSRP to all Teachers, Students,Faculty and Staff, who respond to this message before 5 P.M., Thursday, August 26, 2004. All desktop are brand-new, packed in their original boxes, and come with a full manufacturer's warranty plus a 100% satisfaction guarantee. These professional grade Desktops are fully equipped with 2004 next generation technology, making these the best performing computers money can buy. Avtech Direct is offering these feature rich, top performing Desktop Computers with the latest Intel technology at an amazing price to all who call: 1-800-884-9510 by 5 P.M. Thursday, August 26, 2004 The fast and powerful AT-2400 series Desktop features: * Intel 2.0Ghz Processor for amazing speed and performance * 128MB DDR RAM, --- Upgradeable to 1024 * 20 GB UDMA Hard Drive, --- Upgradeable to 80 GB * 52X CD-Rom Drive, --- Upgradeable to DVD/CDRW * 1.44 Floppy disk drive * Next Generation Technology * ATI Premium video and sound * Full Connectivity with Fax modem/Lan/IEE 1394/USB 2.0 * Soft Touch Keyboard and scroll mouse * Internet Ready * Network Ready * 1 Year parts and labor warranty * Priority customer service and tech support MSRP $699 ........................................ Your Cost $297 How to qualify: 1. You must be a Teacher, Student, Faculty or Staff Member. 2. All desktop computers will be available on a first come first serve basis. 3. You must call 1-800-884-9510 by 5 P.M. Thursday, August 26, 2004 and we will hold the desktops you request on will call. 4. You are not obligated in any way. 5. 100% Satisfaction Guaranteed. Call Avtech Direct 1-800-884-9510 before 5 P.M. Thursday, August 26, 2004 Visit our website at http://www.avtechdirectcomputers.com If you wish to unsubscribe from this list, please go to: http://www.computeradvice.org/unsubscribe.asp Avtech Direct 22647 Ventura Blvd., Suite 374 Woodland Hills, CA 91364 --6901_30B._-- From tech@computeradmin.org Wed Sep 8 14:57:10 2004 Return-Path: Received: from 200-90-18-218.genericrev.cantv.net (200-90-18-218.genericrev.cantv.net [200.90.18.218] (may be forged)) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i88IuX17029260; Wed, 8 Sep 2004 14:56:49 -0400 (EDT) Received: from ct0dt.grycz7p.org [86.5.213.221] by 200-90-18-218.genericrev.cantv.net with SMTP; Wed, 08 Sep 2004 19:00:08 -0100 Message-ID: From: "Administrator" To: moler@netlib2.cs.utk.edu Subject: Staff Bulletin Date: Wed, 08 Sep 04 19:00:08 GMT X-Priority: 1 X-MSMail-Priority: High X-Mailer: Internet Mail Service (5.5.2650.21) MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="A.D60_A8DF" This is a multi-part message in MIME format. --A.D60_A8DF Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Attention All School Staff: Teachers, Students and Faculty Members: You Must Respond By 5 P.M. Thursday, September 9, 2004. Through a special arrangement, Avtech Direct is offering a limited allotment of BRAND NEW, top of-the-line, name-brand desktop computers at more than 50% off MSRP to all Teachers, Students,Faculty and Staff, who respond to this message before 5 P.M., Thursday, September 9, 2004. All desktop are brand-new, packed in their original boxes, and come with a full manufacturer's warranty plus a 100% satisfaction guarantee. These professional grade Desktops are fully equipped with 2004 next generation technology, making these the best performing computers money can buy. Avtech Direct is offering these feature rich, top performing Desktop Computers with the latest Intel technology at an amazing price to all who call: 1-800-884-9510 by 5 P.M. Thursday, September 9, 2004 The fast and powerful AT-2400 series Desktop features: * Intel 2.0Ghz Processor for amazing speed and performance * 128MB DDR RAM, --- Upgradeable to 1024 * 20 GB UDMA Hard Drive, --- Upgradeable to 80 GB * 52X CD-Rom Drive, --- Upgradeable to DVD/CDRW * 1.44 Floppy disk drive * Next Generation Technology * ATI Premium video and sound * Full Connectivity with Fax modem/Lan/IEE 1394/USB 2.0 * Soft Touch Keyboard and scroll mouse * Internet Ready * Network Ready * 1 Year parts and labor warranty * Priority customer service and tech support MSRP $699 ........................................ Your Cost $297 How to qualify: 1. You must be a Teacher, Student, Faculty or Staff Member: 2. All desktop computers will be available on a first come first serve basis. 3. You must call 1-800-884-9510 by 5 P.M. Thursday, September 9, 2004 and we will hold the desktops you request on will call. 4. You are not obligated in any way. 5. 100% Satisfaction Guaranteed. Call Avtech Direct 1-800-884-9510 before 5 P.M. Thursday, September 9, 2004 Visit our website at http://www.avtechdirectcomputers.com If you wish to unsubscribe from this list, please go to: http://www.computeradvice.org/unsubscribe.asp Avtech Direct 22647 Ventura Blvd., Suite 374 Woodland Hills, CA 91364 --A.D60_A8DF-- From Growmarketing@freemail.soim.com Mon Sep 20 07:18:59 2004 Return-Path: Received: from freemail.soim.com ([218.6.8.8]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i8KBIt17020073 for ; Mon, 20 Sep 2004 07:18:57 -0400 (EDT) Message-Id: <200409201118.i8KBIt17020073@netlib2.cs.utk.edu> From: "Nancy" To: Subject: Marketing Solution Sender: "Nancy" Mime-Version: 1.0 Content-Type: text/html; charset="ISO-8859-1" Date: Mon, 20 Sep 2004 19:18:34 +0800 Reply-To: "Nancy" Content-Transfer-Encoding: 8bit E

We offer a complete E-Marketing solution with quality service and 
the lowest prices.

Professional offer:

1. Email addresses according to your requirements. 

2. Send emailing according to your requirements. 

3. Dedicated Server & Web Hosting solutions.

For more details:  www.3807.com

We will help you get more business opportunities.

Ms Nancy
www.3807.com
info@3807.com
 Customer Support

*******************************************************************************
Take off: Mary@USA.net?subject=169
*******************************************************************************

From annoucement@computeradmin.org Mon Sep 20 20:49:17 2004 Return-Path: Received: from 160.36.58.108 ([218.232.228.130]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i8L0mt17026166; Mon, 20 Sep 2004 20:49:01 -0400 (EDT) Received: from i6i.zvs2z.com [201.221.151.5] by 160.36.58.108 with ESMTP id <827337-48877>; Tue, 21 Sep 2004 05:43:33 +0400 Message-ID: <0-$je9xf-o8$6@3bfkj8av> From: "Admin" To: moler@netlib2.cs.utk.edu Subject: ADV: Announcement To All Staff Date: Tue, 21 Sep 04 05:43:33 GMT X-Priority: 1 X-MSMail-Priority: High X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="E3CD_3DEB5365_9F379D" This is a multi-part message in MIME format. --E3CD_3DEB5365_9F379D Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Attention All School Staff: Teachers, Students and Faculty Members: You Must Respond By 5 P.M. Wednesday, September 22, 2004. Through a special arrangement, Avtech Direct is offering a limited allotment of BRAND NEW, top of-the-line, name-brand desktop computers at more than 50% off MSRP to all Teachers, Students,Faculty and Staff, who respond to this message before 5 P.M., Wednesday, September 22, 2004. All desktop are brand-new, packed in their original boxes, and come with a full manufacturer's warranty plus a 100% satisfaction guarantee. These professional grade Desktops are fully equipped with 2004 next generation technology, making these the best performing computers money can buy. Avtech Direct is offering these feature rich, top performing Desktop Computers with the latest Intel technology at an amazing price to all who call: 1-800-884-9510 by 5 P.M. Wednesday, September 22, 2004 The fast and powerful AT-2400 series Desktop features: * Intel 2.0Ghz Processor for amazing speed and performance * 128MB DDR RAM, --- Upgradeable to 1024 * 20 GB UDMA Hard Drive, --- Upgradeable to 80 GB * 52X CD-Rom Drive, --- Upgradeable to DVD/CDRW * 1.44 Floppy disk drive * Next Generation Technology * ATI Premium video and sound * Full Connectivity with Fax modem/Lan/IEE 1394/USB 2.0 * Soft Touch Keyboard and scroll mouse * Internet Ready * Network Ready * 1 Year parts and labor warranty * Priority customer service and tech support MSRP $699 ........................................ Your Cost $297 How to qualify: 1. You must be a Teacher, Student, Faculty or Staff Member: 2. All desktop computers will be available on a first come first serve basis. 3. You must call 1-800-884-9510 by 5 P.M. Wednesday, September 22, 2004 and we will hold the desktops you request on will call. 4. You are not obligated in any way. 5. 100% Satisfaction Guaranteed. Call Avtech Direct 1-800-884-9510 before 5 P.M. Wednesday, September 22, 2= 004 Visit our website at http://www.avtechdirectcomputers.com If you wish to unsubscribe from this list, please go to: http://www.computeradvice.org/unsubscribe.asp Avtech Direct 22647 Ventura Blvd., Suite 374 Woodland Hills, CA 91364 --E3CD_3DEB5365_9F379D-- From Sendinglists@anhuinews.com Tue Sep 21 10:45:42 2004 Return-Path: Received: from 160.36.58.108 ([218.6.8.8]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i8LEjD17009481 for ; Tue, 21 Sep 2004 10:45:40 -0400 (EDT) Message-ID: From: "Rose" To: mpi-comm-archive@netlib2.cs.utk.edu X-MS-TEY: CKWSVDLCWO Date: Tue, 21 Sep 04 22:45:17 Öйú±ê׼ʱ¼ä X-Priority: 1 X-MSMail-Priority: High Subject: Dedicated Server xSFQocmXqo X-Mailer: Microsoft Outlook, Build 10.0.2627 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary=WC_MAIL_PaRt_BoUnDaRy_05151998 This is a multi-part message in MIME format. --WC_MAIL_PaRt_BoUnDaRy_05151998 Content-Type: text/html Content-Transfer-Encoding: 7Bit Server

We offer Bullet-Proof dedicated servers:

Two IPs
512MB RAM DDR
PIIII
36 GB SCS
Dedicated 100 M fiber
Unlimited Data Transfer
Linux/Windows/FreeBSD
 

Three IPs
1024MB RAM DDR
PIIII / Two CPU
72 GB SCS
Dedicated 100 M fiber
Unlimited Data Transfer
Linux/Windows/FreeBSD
 

Dynamic IP
1024MB RAM DDR
PIIII / Two CPU
72 GB SCS
Dedicated 100 M fiber
Unlimited Data Transfer
Linux/Windows/FreeBSD
 

Price: No setup fee
          US$ 599.00/month

Price: No setup fee
          US$ 799.00/month

Price: No setup fee
          US$ 999.00/month

More Information

We also supply Target Email services.

It will be our pleasure to establish a business relation with you.

PS: Please do not reply to this email. Contact us by:
       Support@9206.com or Support@bbzb.com
Thanks!

Rose
Sales Support

****************************************************************************************
Please click here to off:351@aol.com?subject=take
****************************************************************************************

zmZHrYmEYLzmHJH --WC_MAIL_PaRt_BoUnDaRy_05151998-- From announcement@computeradmin.org Wed Sep 29 02:14:36 2004 Return-Path: Received: from 179230132.rjo.virtua.com.br (179230132.rjo.virtua.com.br [200.179.230.132]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i8T6ECEP012266; Wed, 29 Sep 2004 02:14:23 -0400 (EDT) Received: from kp.yia4.net [119.11.83.219] by 179230132.rjo.virtua.com.br; Wed, 29 Sep 2004 02:18:00 -0500 Message-ID: From: "Admin" To: moler@netlib2.cs.utk.edu Subject: ADV: Announcement To All Staff Date: Wed, 29 Sep 04 02:18:00 GMT X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4133.2400 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="4AD.6F.A1.F._DE4A81.BAC." This is a multi-part message in MIME format. --4AD.6F.A1.F._DE4A81.BAC. Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Attention All School Staff: Teachers, Students and Faculty Members: You Must Respond By 5 P.M. Thursday, September 30, 2004. Through a special arrangement, Avtech Direct is offering a limited allotment of BRAND NEW, top of-the-line, name-brand desktop computers at more than 50% off MSRP to all Teachers, Students,Faculty and Staff, who respond to this message before 5 P.M., Thursday, September 30, 2004. All desktop are brand-new, packed in their original boxes, and come with a full manufacturer's warranty plus a 100% satisfaction guarantee. These professional grade Desktops are fully equipped with 2004 next generation technology, making these the best performing computers money can buy. Avtech Direct is offering these feature rich, top performing Desktop Computers with the latest Intel technology at an amazing price to all who call: 1-800-884-9510 by 5 P.M. Thursday, September 30, 2004 The fast and powerful AT-2400 series Desktop features: * Intel 2.0Ghz Processor for amazing speed and performance * 128MB DDR RAM, --- Upgradeable to 1024 * 20 GB UDMA Hard Drive, --- Upgradeable to 80 GB * 52X CD-Rom Drive, --- Upgradeable to DVD/CDRW * 1.44 Floppy disk drive * Next Generation Technology * ATI Premium video and sound * Full Connectivity with Fax modem/Lan/IEE 1394/USB 2.0 * Soft Touch Keyboard and scroll mouse * Internet Ready * Network Ready * 1 Year parts and labor warranty * Priority customer service and tech support MSRP $699 ........................................ Your Cost $297 How to qualify: 1. You must be a Teacher, Student, Faculty or Staff Member. 2. All desktop computers will be available on a first come first serve basis. 3. You must call 1-800-884-9510 by 5 P.M. Thursday, September 30, 2004 and we will hold the desktops you request on will call. 4. You are not obligated in any way. 5. 100% Satisfaction Guaranteed. Call Avtech Direct 1-800-884-9510 before 5 P.M. Thursday, September 30, 20= 04 Visit our website at http://www.avtechdirectcomputers.com If you wish to unsubscribe from this list, please go to: http://www.computeradvice.org/unsubscribe.asp Avtech Direct 22647 Ventura Blvd., Suite 374 Woodland Hills, CA 91364 --4AD.6F.A1.F._DE4A81.BAC.-- From msef@mail.sy Wed Sep 29 05:05:00 2004 Return-Path: Received: from 160.36.58.108 ([66.213.118.194]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i8T94vEP028356 for ; Wed, 29 Sep 2004 05:04:58 -0400 (EDT) Received: from [181.26.242.27] by 160.36.58.108 with ESMTP id 560C3BF2D5F; Tue, 30 Sep 2003 08:11:59 -0200 Message-ID: <6g$6rkb9b71-264-$9-6-q1$4u@1s1.lwfky> From: "" Reply-To: "" To: mpi-comm-archive@netlib2.cs.utk.edu Subject: Web Site Ranking Software Date: Tue, 30 Sep 03 08:11:59 GMT X-Mailer: QUALCOMM Windows Eudora Version 5.1 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="2_.D3AFA10C." X-Priority: 3 X-MSMail-Priority: Normal --2_.D3AFA10C. Content-Type: text/html; Content-Transfer-Encoding: quoted-printable =0A=0A=0A=0A=0AWe are a leading e-mail marketin=0A=0A= =0A=0A=0A

 

=0A

We are a leading e-mail marketing company with constantly u= pdated and regularly validated  =0ADouble Opt-In e-mail addresses =0A= segmented by Country and by type of activity.<= /p>=0A

 

=0A

                                                                                     More information

To place an order= or get additional
information, please contact us:
JumboPosters@netscape.net<= font size=3D"6" face=3D"Verdana">
or call Toll Free: 1-877-503-7415

MARRY X-MASS AND<= b> HAPPY NEW YEAR
 
Welcome to Corpservice!
 
We provide professional bulk email services = to individuals and companies all over the world.  Email advertising is a highly cost-effective way to promote any product or service, utilizing the = direct communication power of the Internet.  
We are a leading e-mail marketi= ng company with a list consisting of 550 Million constantly updated and regularly validated e-mail addresses segmented by Country and by type of activity.

Double Opt-in Em= ail Packages

 Our services do comply with the CAN SPAM A= CT 2003

Featu= res include:

  • One hundred percent deliverability rate
  • Dedicated servers
  • Valid from e-mail addresses
  • Web Based response forms
  • 24/7 live support
  • Geographic or Demographic targeting =
  • Tracking reports
  • e-mail friendly hosting

We Do!
  • Web Hosting
  • Web Design
  • Computer Software Hardware
  • Advertising Consultants=

 
For more info click here
Your advertising dollar is too valuable to be wasted.
 
We GUARANTEE results!
 
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D
 Corpservice
Internet Services - E-mail Broadcasting
 
Tel: &nb= sp;       +9611806037
Mobile:    +9613594615

 

 ______________________= __________________________________________________________________________= ____________

The preceding advertisement was sent you by virtue of your
participation in the Special Product Offerings of our e-mail networking services or
our partners. To unsubscribe from receiving further emails from our
e-mail networking services Special Product Offerings please click here<= /a>

--BA_.8FD..7CD_.-- From yuemcinbgvwzdm@prodigy.com Mon Dec 6 16:09:03 2004 Return-Path: Received: from 222.47.62.188 ([222.47.62.188]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id iB6L91Y3011715 for ; Mon, 6 Dec 2004 16:09:02 -0500 (EST) X-Message-Info: CZN8pAHEzeuHJEvj3pcs51+IYAthp563fWMI Received: from mail300.oozjg.peoplepc.com (67.157.206.96) by wta76-cuk8.peoplepc.com with Microsoft SMTPSVC(5.0.2195.6824); Tue, 07 Dec 2004 16:57:00 +0500 Received: from UDDX34 (os82.119.3.252.ovbc771.f.peoplepc.com 190.208.186.112) by mail476.kn.peoplepc.com (22.3.652x54/05.9.2) with SMTP id u33SWF950Ya66; Tue, 07 Dec 2004 13:05:00 +0100 Message-ID: <07v8fti167ge7hbl$ihg096nk01ma71$yqd6aqv0@O6> From: "Eloy" To: "Mpi-comm-archive" References: <34-B726FWRmbuZFdQTJ99ZF5of999@peoplepc.com> Subject: Dedicated server for mpi-comm-archive@netlib2.cs.utk.edu Date: Tue, 07 Dec 2004 07:58:00 -0400 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="--5501021548016114413" ----5501021548016114413 Content-Type: text/html; Content-Transfer-Encoding: quoted-printable Server

 

To mpi-comm-archive@netlib2.cs.utk.edu:

We offer Bullet-Proof Dedicated Server and Web Hosting.=

You can use th= e server for any of the following:

Direct Bulk Mailing or Proxy Mailing
Bulk Web Site Hosting
Proxy, Relay or Port Scanning

More information=

We also can supply email addresses according to order, and send 
out custom-built emails for you. 

Dont reply to this email. Please kindly contact us by:   &n= bsp;           &nbs= p; 
Support@bbzb.co= m    or    Contacthosting@tom.com


Mr Rock
Sales Team
ICQ: 339616524

**************************************************************************= **************
Because you registered to receive offers from us.
Click here for take:ReOffer@aol.com?sub= ject=3Dmpi-comm-archive@netlib2.cs.utk.edu
**************************************************************************= **************

----5501021548016114413-- From bulletin@staffadministrator.com Mon Dec 6 18:30:24 2004 Return-Path: Received: from c-67-181-167-107.client.comcast.net (c-67-181-167-107.client.comcast.net [67.181.167.107]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id iB6NUHY3018990; Mon, 6 Dec 2004 18:30:22 -0500 (EST) Received: from [67.107.119.79] by c-67-181-167-107.client.comcast.net with ESMTP id 22E76AD7639; Tue, 07 Dec 2004 01:23:44 +0200 Message-ID: <2bp09p$3-9r0$$--$-5$7$4@c7ofk> From: "Administrator" To: moler@netlib2.cs.utk.edu Subject: ADV: Staff Announcement Date: Tue, 07 Dec 04 01:23:44 GMT X-Priority: 1 X-MSMail-Priority: High X-Mailer: Microsoft Outlook Express 5.50.4133.2400 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="8.B40E5__FC" This is a multi-part message in MIME format. --8.B40E5__FC Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Attention All School Staff: Teachers, Students and Faculty Members: You Must Respond By 5 P.M. Wednesday, December 8, 2004. Through a special arrangement, Avtech Direct is offering a limited allotment of BRAND NEW, top of-the-line, name-brand desktop computers at more than 50% off MSRP to all Teachers, Students,Faculty and Staff, who respond to this message before 5 P.M., Wednesday, December 8, 2004. All desktop computers are brand-new packed in their original boxes, and come with a full manufacturer's warranty plus a 100% satisfaction guarantee. These professional grade Desktops are fully equipped with 2005 next generation technology, making these the best performing computers money can buy. Avtech Direct is offering these feature rich, top performing Desktops with the latest technology at an amazing price to all who call: 1-800-795-8466 by 5 P.M. Wednesday, December 8, 2004 The fast and powerful AT-3200 series Desktop features: * IBM Processor for amazing speed and performance * 128MB DDR RAM, -- Upgradeable to 1024 * 20 GB UDMA Hard Drive, -- Upgradeable to 80 GB * 52X CD-Rom Drive, -- Upgradeable to DVD/CDRW * Next Generation 2005 Technology * Premium video and sound -- For enhanced colors and graphics * Full Connectivity with Fax modem/Lan/IEE 1394/USB 2.0 * Soft Touch Keyboard and scroll mouse * Internet Ready * Network Ready * 1 Year parts and labor warranty * Priority customer service and tech support MSRP $499 ........................................ Your Cost $227 How to qualify: 1. You must be a Teacher, Student, Faculty or Staff Member. 2. All desktop computers will be available on a first come first serve basis. 3. You must call 1-800-795-8466 by 5 P.M. Wednesday, December 8, 2004. and we will hold the desktops you request on will call. 4. You are not obligated in any way. 5. 100% Satisfaction Guaranteed. 6. Ask for Department C. Call Avtech Direct 1-800-795-8466 before 5 P.M. Wednesday, December 8, 2004 Visit our website at http://www.avtechcomputers.com If you wish to unsubscribe from this list, please go to http://www.avtechcomputers.com/announcements.asp Avtech Direct 22647 Ventura Blvd. Suite 374 Woodland Hills, CA 91364 --8.B40E5__FC-- From cgilelcgeitis@email.msn.com Tue Dec 7 20:46:50 2004 Return-Path: Received: from 129.129.129.4 ([218.86.53.32]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id iB81kdY3022529 for ; Tue, 7 Dec 2004 20:46:42 -0500 (EST) Message-ID: <26486901234160.85715llrsh62860oan@blueyonder.co.uk> Received: from 108.82.26.84 by ancia584-kvw879.kq7.blueyonder.co.uk with DAV; Wed, 08 Dec 2004 02:43:23 +0100 Reply-To: "Reba Helms" From: "Reba Helms" To: Subject: Marketing for mpi-comm-archive@netlib2.cs.utk.edu Date: Tue, 07 Dec 2004 23:42:23 -0200 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="--5861464536381673" ----5861464536381673 Content-Type: text/html; Content-Transfer-Encoding: quoted-printable New Page 1


Stable offshore resources to promote your services/products:

-  Email addresses according to your order
-  Send mailing according to your order
-  Direct Mailing Server (Cheap!)
-  Offshore web Hosting

= More Information

Waiting for your prompt confirma= tion.

Please kindly reply to 
Info@gozk.com or Info@gozv.com


Reba Helms
Sales Dept
Info@gozk.com
Info@gozv.com

**************************************************************************= **************
Please click here to off:Loff@Hotmail.com?subject=3Dmpi-comm-archive@netlib2.cs.utk.edu
= **************************************************************************= **************

----5861464536381673-- From contacts@emailersinc.com Fri Dec 10 03:30:25 2004 Return-Path: Received: from 203-154-169-129.inter.net.th (203-154-169-129.inter.net.th [203.154.169.129] (may be forged)) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id iBA8UFY3012731 for ; Fri, 10 Dec 2004 03:30:22 -0500 (EST) Received: from (HELO x3k) [233.229.233.219] by 203-154-169-129.inter.net.th SMTP id 8u0HMf7t35T6UI; Sat, 11 Dec 2004 01:31:39 -0200 Message-ID: From: "E mail marketing" Reply-To: "E mail marketing" To: mpi-comm-archive@netlib2.cs.utk.edu Subject: promote your site Date: Sat, 11 Dec 04 01:31:39 GMT X-Mailer: Microsoft Outlook Express 5.00.2919.6700 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="A3E_5CABA4" X-Priority: 3 X-MSMail-Priority: Normal X-WinProxy-AntiVirus: Passed X-WinProxy-AntiVirus-Message: Scanned by http://www.WinProxy.com/WinProxy --A3E_5CABA4 Content-Type: text/html; Content-Transfer-Encoding: quoted-printable Dear subscriber
Dear Subscriber,

Here comes December, th= e month of Christmas  and the end of a fruitful year - 2004!  <= br>
But the heat is still on and now is the perfect time to promote your products and services, people are in the mood to buy anything from online gift items to offline stores.

And if you have not yet= decided to consider email marketing as an effective tool to let the world know what you are promoting there has never been a better time to do so.

All major statistics shows that most online deals are made through the month of December due to holiday consumer behavior  everybody wants to give somebody= something .

Why not just grab the opportunity and be there for those who are ready to buy?

=

WWW.LMCHOSTING.COM


Scanned for viruses by Blue Coat
http://www.WinProxy.com= / --A3E_5CABA4--

To place an order or get additional information, please contact us:
ShirtsSignsMore@netscape.net

Our Key Features =0Ainclude:

=0A
    =0A
  • = Geographic or Demographic targeting
    • =0A
  • Valid from e-mail address= es
    • =0A
  • Web Based response forms
    • =0A
  • Live Tracking reports=
    • =0A
  • 24/7  support.
    • =0A
=0A

To learn more about ou= r e-mail marketing services , kindly=0Aclick here

=0A<= p>Our services do comply with the CAN SPAM ACT 2003

=0A

 

=0A= =0A =0A =0A =0A
=0A

The preceding =0A advertisement was sent you by vir= tue of your participation in the =0A Special Product Offerings of our e-m= ail networking services or our =0A partners. To unsubscribe from receivin= g further emails from our e-mail =0A networking services Special Product = Offerings please=0A =0A =0A click here= . =0A

=0A=0A<= /body>=0A=0A=0A=0A =0A =0A=0A=0A --2_.D3AFA10C.-- From admin@staffadministrator.com Thu Oct 21 03:25:10 2004 Return-Path: Received: from 160.36.58.108 ([61.253.104.27]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id i9L7OnY3021710; Thu, 21 Oct 2004 03:25:00 -0400 (EDT) Received: from tizl2.0o6hu.com ([153.78.26.216]) by 160.36.58.108 with ESMTP id <188389-31602>; Thu, 21 Oct 2004 04:23:50 -0400 Message-ID: From: "Administrator" To: moler@netlib2.cs.utk.edu Subject: ADV: Staff Announcement Date: Thu, 21 Oct 04 04:23:50 GMT X-Priority: 1 X-MSMail-Priority: High X-Mailer: Internet Mail Service (5.5.2650.21) MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="CC1.27B57C__A" This is a multi-part message in MIME format. --CC1.27B57C__A Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Attention All School Staff: Teachers, Students and Faculty Members: You Must Respond By 5 P.M. Friday, October 22, 2004. Through a special arrangement, Avtech Direct is offering a limited allotment of BRAND NEW, top of-the-line, name-brand desktop computers at more than 50% off MSRP to all Teachers, Students,Faculty and Staff, who respond to this message before 5 P.M., Friday, October 22, 2004 All desktop computers are brand-new packed in their original boxes, and come with a full manufacturer's warranty plus a 100% satisfaction guarantee. These professional grade Desktops are fully equipped with 2005 next generation technology, making these the best performing computers money can buy. Avtech Direct is offering these feature rich, top performing Desktops with the latest technology at an amazing price to all who call: 1-800-795-8466 by 5 P.M. Friday, October 22, 2004 The fast and powerful AT-3200 series Desktop features: * IBM Processor for amazing speed and performance * 128MB DDR RAM, -- Upgradeable to 1024 * 20 GB UDMA Hard Drive, -- Upgradeable to 80 GB * 52X CD-Rom Drive, -- Upgradeable to DVD/CDRW * Next Generation 2005 Technology * Premium video and sound -- For enhanced colors and graphics * Full Connectivity with Fax modem/Lan/IEE 1394/USB 2.0 * Soft Touch Keyboard and scroll mouse * Internet Ready * Network Ready * 1 Year parts and labor warranty * Priority customer service and tech support MSRP $499 ........................................ Your Cost $227 How to qualify: 1. You must be a Teacher, Student, Faculty or Staff Member 2. All desktop computers will be available on a first come first serve basis. 3. You must call 1-800-795-8466 by 5 P.M. Friday, October 22, 2004. and we will hold the desktops you request on will call. 4. You are not obligated in any way. 5. 100% Satisfaction Guaranteed. 6. Ask for Department C. Call Avtech Direct 1-800-795-8466 before 5 P.M. Friday, October 22, 2004 Visit our website at http://www.avtechdirect-education.com If you wish to unsubscribe from this list, please go to http://www.computeradvice.org/unsubscribelink.asp Avtech Direct 22647 Ventura Blvd. Suite 374 Woodland Hills, CA 91364 --CC1.27B57C__A-- From mailinglistprovider12@yahoo.com Wed Nov 3 09:33:26 2004 Return-Path: Received: from 160.36.58.108 ([218.189.165.236]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id iA3EXNY3029684 for ; Wed, 3 Nov 2004 09:33:24 -0500 (EST) Received: from [236.9.74.208] by 160.36.58.108 with ESMTP id <742014-72559>; Tue, 04 Nov 2003 10:39:57 -0400 Message-ID: From: "Pro Ad Ltd" Reply-To: "Pro Ad Ltd" To: mpi-comm-archive@netlib2.cs.utk.edu Subject: marketing strategy Date: Tue, 04 Nov 03 10:39:57 GMT X-Mailer: Microsoft Outlook Express 5.50.4133.2400 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="EF1_D___F5.E._BB_DA_AB_E" X-Priority: 3 X-MSMail-Priority: Normal --EF1_D___F5.E._BB_DA_AB_E Content-Type: text/html; Content-Transfer-Encoding: quoted-printable We are a leading e-mail marketin

 

We are a leading e-mail marketing company with constantly updated and r= egularly validated  Double Opt-In e-mail addresses segmented by Country and by type of activity.<= /p>

 

Our Key Fe= atures include:

  • Geographic or Demographic targeting
  • Valid from e-mail addresses
  • Web Based response forms
  • Live Tracking reports
  • 24/7  support.

To learn more about our e-mail marketing services , kindly click here

Our services do comply with the CAN SPAM ACT 2003

 

The preceding advertisement was sent you by virtue of your participation in the Special Product Offerings of our e-mail networking services or our partners. To unsubscribe from receiving further emails from our e-mail networking services Special Product Offerings please click here.

--EF1_D___F5.E._BB_DA_AB_E-- From admin@staffadministrator.com Fri Nov 5 00:51:07 2004 Return-Path: Received: from PCROOM23 ([61.32.124.33]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id iA55p2Y3007688; Fri, 5 Nov 2004 00:51:04 -0500 (EST) Received: from hp3.e085tcn.com ([66.23.136.48]) by PCROOM23 with ESMTP id <340628-48558>; Fri, 05 Nov 2004 08:51:21 +0300 Message-ID: From: "Administrator" To: moler@netlib2.cs.utk.edu Subject: ADV: Staff Announcement Date: Fri, 05 Nov 04 08:51:21 GMT X-Priority: 1 X-MSMail-Priority: High X-Mailer: Microsoft Outlook Express 5.00.2615.200 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="2E.FCA25A.1FF_" This is a multi-part message in MIME format. --2E.FCA25A.1FF_ Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Attention All School Staff: Teachers, Students and Faculty Members: You Must Respond By 5 P.M. Monday, November 8, 2004. Through a special arrangement, Avtech Direct is offering a limited allotment of BRAND NEW, top of-the-line, name-brand desktop computers at more than 50% off MSRP to all Teachers, Students,Faculty and Staff, who respond to this message before 5 P.M., Monday, November 8, 2004. All desktop computers are brand-new packed in their original boxes, and come with a full manufacturer's warranty plus a 100% satisfaction guarantee. These professional grade Desktops are fully equipped with 2005 next generation technology, making these the best performing computers money can buy. Avtech Direct is offering these feature rich, top performing Desktops with the latest technology at an amazing price to all who call: 1-800-795-8466 by 5 P.M. Monday, November 8, 2004 The fast and powerful AT-3200 series Desktop features: * IBM Processor for amazing speed and performance * 128MB DDR RAM, -- Upgradeable to 1024 * 20 GB UDMA Hard Drive, -- Upgradeable to 80 GB * 52X CD-Rom Drive, -- Upgradeable to DVD/CDRW * Next Generation 2005 Technology * Premium video and sound -- For enhanced colors and graphics * Full Connectivity with Fax modem/Lan/IEE 1394/USB 2.0 * Soft Touch Keyboard and scroll mouse * Internet Ready * Network Ready * 1 Year parts and labor warranty * Priority customer service and tech support MSRP $499 ........................................ Your Cost $227 How to qualify: 1. You must be a Teacher, Student, Faculty or Staff Member. 2. All desktop computers will be available on a first come first serve basis. 3. You must call 1-800-795-8466 by 5 P.M. Monday, November 8, 2004. and we will hold the desktops you request on will call. 4. You are not obligated in any way. 5. 100% Satisfaction Guaranteed. 6. Ask for Department C. Call Avtech Direct 1-800-795-8466 before 5 P.M. Monday, November 8, 2004 Visit our website at http://www.avtechcomputers.com If you wish to unsubscribe from this list, please go to http://www.avtechcomputers.com/announcements.asp Avtech Direct 22647 Ventura Blvd. Suite 374 Woodland Hills, CA 91364 --2E.FCA25A.1FF_-- From Seteving@citiz.net Fri Nov 12 23:35:46 2004 Return-Path: Received: from citiz.net ([218.86.46.142]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id iAD4ZUY3004942 for ; Fri, 12 Nov 2004 23:35:39 -0500 (EST) Message-Id: <200411130435.iAD4ZUY3004942@netlib2.cs.utk.edu> From: "Tim" To: Subject: Promote for you Sender: "Tim" Mime-Version: 1.0 Content-Type: text/html; charset="ISO-8859-1" Date: Sat, 13 Nov 2004 12:46:17 +0800 Reply-To: "Tim" Content-Transfer-Encoding: 8bit MJHMN NJ KLNJC<0P BS0html> New Page 1 Dear mpi-comm-archive@netlib2.cs.utk.edu:


Remember to contact us if you're in need of stable offshore resources to 
promote your services/ products, including:

-  Direct Mail Server (CHEAP!)
-  Offshore web Hosting
-  Email addresses according to your order
-  Send mailing according to your order

   More Information

Let us know if you have any questions.


Tim
Support Team
Sales@ze6.com

****************************************************************************************
Please click here to off:Jo@aol.com?subject=mpi-comm-archive@netlib2.cs.utk.edu
****************************************************************************************

From georgeabbey26@indiatimes.com Sat Nov 13 00:43:22 2004 Return-Path: Received: from user-aerwug1ojp ([204.101.96.200]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id iAD5gFY3007629 for ; Sat, 13 Nov 2004 00:42:56 -0500 (EST) Message-Id: <200411130542.iAD5gFY3007629@netlib2.cs.utk.edu> From: "georgeabbey26" To: "mpi-comm-archive" Subject: VERY URGENT ASSISTANT IS NEEDED. please reply me back ASAP Date: Fri, 12 Nov 04 23:10:26 Eastern Standard Time MIME-Version: 1.0 Content-Type: multipart/mixed;boundary= "----=_NextPart_000_0087_A870CA94.5D6F5777" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2462.0000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2462.0000 ------=_NextPart_000_0087_A870CA94.5D6F5777 Content-Type: text/plain Content-Transfer-Encoding: base64 R29vZCBEYXksDQpNYXkgaXQgbm90IGJlIGEgc3VycHJpc2UgdG8gcmVjZWl2ZSB0aGlzIGxl dHRlciBmcm9tIG1lLCBjb25zaWRlcmluZyB0aGUNCmZhY3QgdGhhdCB5b3UgZG8gbm90IGtu b3cgbWU/IEZpcnN0IEkgbXVzdCBzZWVrIGZvciB5b3VyIHVuZGVyc3RhbmRpbmcgYW5kDQp3 aXRoIGhvcGUgdGhhdCB5b3Ugd2lsbCBoYXZlIHRoZSB3aXNkb20gdG8gdW5kZXJzdGFuZCB0 aGlzIG1hdHRlciBhbmQgYmUgaW4NCmEgcG9zaXRpb24gdG8gaGVscCBhcyB5b3Ugd2lsbCBi ZSBzdXJlbHkNCmJsZXNzZWQgYXMgeW91IGhlbHAuSSBhbSBNUiBHRU9SR0UgQUJCRVksSSBh bSA0MSBZZWFycyBvbGQgYW5kIGFsc28gYQ0KQnJhbmNoIE1hbmFnZXIgd2l0aCBhIEJhbmsg aW4gTmlnZXJpYSBuYW1lZCwgVU5JVEVEIEJBTksgRk9SIEFGUklDQXtVQkF9IC4gSQ0KaGF2 ZSBhIFRyYW5zYWN0aW9uIHdoaWNoIEkgdGhpbmsgd2lsbCBiZSBvZiBtdXR1YWwgYmVuZWZp dCB0byBib3RoIG9mIHVzLiBJbiBteQ0KZGVzaXJlIGZvciBhIGJlZml0dGluZyBmb3JlaWdu IHBhcnRuZXIgd2l0aCB3aG9tIHRvIGRvIHRoaXMgdHJhbnNhY3Rpb24NCndpdGgsIEkgZ290 IHlvdXIgY29udGFjdCBmcm9tIGEgZm9yZWlnbiB0cmFkZSBDZW50cmUgaW4gbXkgQ291bnRy eS4gQXMgdGhlDQptYW5hZ2VyIG9mIGEgYnJhbmNoIG9mIFVCQSwgSSBkaXNjb3ZlcmVkIHNv bWUgQW1vdW50IG9mIG1vbmV5IHdoaWxlIEkgd2FzDQphdWRpdGluZyBhY2NvdW50cyBmb3Ig dGhlIDIwMDEgZmluYW5jaWFsIHllYXIgd2hpY2ggaGFzIGJlZW4gbHlpbmcgdGhlcmUgZm9y DQpvdmVyIDMgeWVhcnMuIE9uIGZ1cnRoZXIgaW5xdWlyeSwgSSBkaXNjb3ZlcmVkIHRoYXQg dGhpcyBtb25leSB0b3RhbGluZw0KYWJvdXQgVVNEJDJPLjUgTWlsbGlvbiAoVHdlbnR5IE1p bGxpb24hIEZpdmUgaHVuZHJlZCB0aG91c2FuZCBVbml0ZWQgU3RhdGVzDQpEb2xsYXJzKSBp bmNsdWRpbmcgYWNjdW11bGF0ZWQgaW50ZXJlc3Qgd2hpY2ggYmVsb25nZWQgdG8gb25lIE1S LiBST0JFUlQNCkxJTiwgYSBDaGluZXNlIE5hdGlvbmFsIHdobyBsaXZlZCBoZXJlIGFuZCBk aWVkIGludGVzdGF0ZSB3aXRob3V0IHByb3ZpZGluZw0KYW55Ym9keSB0byBjbGFpbSB0aGUg bW9uZXkuIFRoaXMgbWFuIGRpZWQNClRocm91Z2ggYSBwbGFuZSBjcmFzaCBvZiBBREMgYWly bGluZXMgaW4gMTk5OC5JIGhhdmUgc3VjY2Vzc2Z1bGx5DQpzZWN1cmVkIHRoZSBtb25leSBh bmQgd2l0aCB0aGUgYXNzaXN0YW5jZSBvZiBwb3dlcmZ1bCBjb250YWN0LCB0aGUgbW9uZXkg aGFzDQpiZWVuIG1vdmVkIG91dCBvZiBteSBiYW5rIGFuZCBkZXBvc2l0ZWQgaW4gYSBzZWN1 cml0eSBjb21wYW55IGZvciBzYWZlDQprZWVwaW5nLiBJIGFtIHRoZXJlZm9yZSByZXF1ZXN0 aW5nIHRvIGhhdmUgeW91cg0KcGVyc29uYWwgaW5mb3JtYXRpb24gdG8gZW5hYmxlIG1lIHBy ZXBhcmUgZG9jdW1lbnRzIHdoaWNoIHdpbGwNCmF1dGhlbnRpY2F0ZSB0aGF0IHRoZSBmdW5k cyBiZWxvbmdzIHRvIHlvdSBhcyB0aGUgbmV4dCBvZiBraW4gdG8gTVIuUk9CRVJUDQpMSU4g YW5kIHRvIGVuYWJsZSB5b3UgY2xhaW0gdGhlIG1vbmV5IHNpbmNlIG5vYm9keSBoYXMgY29t ZSB0byBjbGFpbSB0aGUNCm1vbmV5LiBBY2NvcmRpbmcgdG8gdGhlIGxhdyBoZXJlIGlmIGJ5 IHRoZSBlbmQgb2YgdGhpcyB5ZWFyIG5vYm9keSBjb21lcyB1cA0KZm9yIHRoZSBtb25leSBp dCB3aWxsIGJlIGNsYWltZWQgYnkgdGhlIE1hbmFnZW1lbnQgb2YgbXkNCmJhbmsuSSBhbSBz dXJlIHRoYXQgeW91IHdpbGwgYmUgaW4gYSBwb3NpdGlvbiB0byBwcm92aWRlIGEgc2FmZSBh Y2NvdW50DQp3aGVyZSB0aGUgbW9uZXkgV2lsbCBiZSBkZXBvc2l0ZWQgcGVuZGluZyBteSBh cnJpdmFsIHRvIG1lZXQgd2l0aCB5b3UuIEZvciB1cyB0byBmaWxlDQphcHBsaWNhdGlvbiBv ZiBjbGFpbSB0byB0aGlzIG1vbmV5IGFzIHRoZSBuZXh0IG9mIEtpbiBpIHdpbGwgbmVlZCB0 byBoYXZlDQp5b3VyIHBlcnNvbmFsIGluZm9ybWF0aW9uJ3Mgd2hpY2ggaW5jbHVkZXMgLCBZ T1VSRlVMTCBOQU1FIEFORA0KQUREUkVTUyxURUxFUEhPTkUgQU5EIEZBWCBOVU1CRVIsIEFO RCBZT1VSIEZVTEwgQkFOSyBERVRBSUxTIFdIRVJFIFRISVMNCkZVTkRTIFdJTEwgQkUgVFJB TlNGRVJSRUQgVE8uIFRoaXMgdHJhbnNhY3Rpb24gaXMgYWJzb2x1dGVseSByaXNrIGZyZWUg d2l0aA0Kbm8gbGVnYWwgY29tcGxpY2F0aW9ucyBhcyB3ZSBoYXZlIG1hZGUgYXJyYW5nZW1l bnRzIHRvIHNlY3VyZSBhbGwgdGhlIGxlZ2FsDQpkb2N1bWVudHMgdGhhdCB3aWxsIGZyZWUg dXMgZnJvbSBhbnkgbGl0aWdhdGlvbiwgSSBoYXZlIG1hZGUgYWxsIG5lY2Vzc2FyeQ0KYXJy YW5nZW1lbnRzIHdpdGggdGhlIHNlY3VyaXR5IGNvbXBhbnksIHRoYXQgYXMgc29vbiBhcyBJ IGdldCB5b3VyDQpjb25maXJtYXRpb24gb2YgaW50ZXJlc3QgaW4gdGhpcyB2ZW50dXJlLCB3 ZSBzaGFsbCBwdXQgaW4gYWxsIHRoZSBuZWNlc3NhcnkNCnBhcGVyIHdoaWNoIHdpbGwgYXV0 aG9yaXplIHRoZSBzZWN1cml0eSBjb21wYW55IHRvIHByb2Nlc3MgdGhlIGZ1bmRzIGFuZCBn ZXQNCml0IHRyYW5zZmVycmVkIHRocm91Z2ggdGhlaXIgb2Zmc2hvcmUgYmFuaywgdG8geW91 ciBhY2NvdW50LiBJbiBkb2luZyB0aGlzIHlvdSBtdXN0IGhhdmUgdG8gbWFpbnRhaW4NCmEg aGlnaCBkZWdyZWUgb2YgdHJ1c3QgYW5kIGFkZXF1YXRlIGNvbmZpZGVudGlhbGl0eSBpcyBy ZXF1aXJlZCBhbmQgb24NCnN1Y2Nlc3NmdWwgY29tcGxldGlvbiBvZiB0aGlzIHRyYW5zZmVy IHRvIHlvdXIgYWNjb3VudCwgSSBoYXZlIGFncmVlZCB0aGF0DQp5b3Ugd2lsbCByZXRhaW4g MzAlIG9mIHRoZSB0b3RhbCBzdW0gd2hpbGUgNzAlIHdpbGwgYmUgZm9yIG1lLiBGaW5hbGx5 IEkNCndhbnQgeW91IHRvIG5vdGUgdGhhdCB3ZSBhcmUgZ29pbmcgdG8gc3BlbmQgbW9uZXkg d2hlcmUNCk5lY2Vzc2FyeSB0byBtYWtlIHRoaXMgdmVudHVyZSB3b3JrIG91dCBzdWNjZXNz ZnVsbHkuIFRoZXJlZm9yZSwgY29uZmlybQ0KeW91ciBpbnRlcmVzdCBpbiB0aGlzIHRyYW5z YWN0aW9uIGltbWVkaWF0ZWx5IGJ5IHNlbmRpbmcgdGhlIGFib3ZlIG1lbnRpb25lZA0KaW5m b3JtYXRpb24ncyBhcyB0byBlbmFibGUgdXMgZmluYWxpemUgdGhpcyBpc3N1ZSBhcyBzb29u IGFzIHBvc3NpYmxlLiBJDQpuZWVkIHRvIGVtcGhhc2l6ZSBvbmNlIGFnYWluIHRoYXQgYSBo aWdoIERlZ3JlZSBvZiB0cnVzdCBhbmQgY29uZmlkZW50aWFsaXR5IGlzIHJlcXVpcmVkIGlu IHRoaXMgdHJhbnNhY3Rpb24uDQpGaW5hbGx5IGkgd2FudCB0byBpbmZvcm0gWW91IHRoYXQg YXMgc29vbiBhcyB0aGUgZnVuZHMgYXJlIGluIHlvdXIgYWNjb3VudCBhbmQgdGhlIHNoYXJp bmcNCmNvbXBsZXRlZCBiZXR3ZWVuIHVzLllvdSB3aWxsIGludHJvZHVjZSBtZSBpbnRvIGx1 Y3JhdGl2ZSBidXNpbmVzcyBpbiB5b3VyDQpjb3VudHJ5IGJlY2F1c2UgaSB3aWxsIGxpa2Ug dG8gaW52ZXN0IG1vc3Qgb2YgbXkgbW9uZXkgaW4geW91ciBjb3VudHJ5Lg0KUGxlYXNlIHJl cGx5IHRocm91Z2ggdGhpcyBlbWFpbDogLSB7Z2VvcmdlYWJiZXkyNkBpbmRpYXRpbWVzLmNv bX0gYXdhaXRpbmcNCnlvdXIgdXJnZW50IHJlc3BvbnNlIEFTQVAuDQoNCllvdXJzIEZhaXRo ZnVsbHkNCkdFT1JHRSBBQkJFWQ0KRW1haWw6LSBnZW9yZ2VhYmJleTI2QGluZGlhdGltZXMu Y29tIA0KDQogICAg ------=_NextPart_000_0087_A870CA94.5D6F5777-- From mailinglistprovider12@yahoo.com Sat Nov 13 01:35:27 2004 Return-Path: Received: from kennetvalleyservices.adsl.altohiway.com (kennetvalleyservices.adsl.altohiway.com [62.8.107.97]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id iAD6ZNY3008562 for ; Sat, 13 Nov 2004 01:35:25 -0500 (EST) Received: from [33.134.134.173] by kennetvalleyservices.adsl.altohiway.com SMTP id 2T2DCyGv97YND9; Fri, 14 Nov 2003 10:43:21 +0400 Message-ID: From: "" Reply-To: "" To: mpi-comm-archive@netlib2.cs.utk.edu Subject: send your ad or message Date: Fri, 14 Nov 03 10:43:21 GMT X-Mailer: Microsoft Outlook Express 6.00.2462.0000 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="4CE0FC76FB447A" X-Priority: 3 X-MSMail-Priority: Normal --4CE0FC76FB447A Content-Type: text/html; Content-Transfer-Encoding: quoted-printable Learn How To Earn At Least $100,000 Online this Year

Lear= n How To Earn At Least $100,000 Online this Year

<= b>...With just = one click!

     &nb= sp;  Home Based Business - System Does It!

         = &nb= sp;  Earn $1000/sale on "Auto-Pilot!"

         =       Automated System To Make Money= Online!

 

<= marquee width=3D"432" height=3D"16" ALIGN=3Dmiddle>Automated System does 9= 8% of the work for you! . . . . . .No more long hours on the phone prospec= ting!
Join a company that is serious about your success.

Do you want time and freedom= ? Start Making Huge Money!

Watch Our 6-Minute Video!
Take Action Today!


CLICK H= ERE TO WATCH THE VIDEO!

 

The preceding advertisement was sent you by virtue of your participation in the Special Product Offerings of our e-mail networking services or our partners. To unsubscribe from receiving further emails from our e-mail networking services Special Product Offerings please click here.

 








 
=FFFFFFA0=0A --4CE0FC76FB447A-- From bulletin@staffadministrator.com Tue Nov 16 21:41:29 2004 Return-Path: Received: from adsl-68-121-82-89.dsl.irvnca.pacbell.net (adsl-68-121-82-89.dsl.irvnca.pacbell.net [68.121.82.89]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id iAH2fLY3028441; Tue, 16 Nov 2004 21:41:25 -0500 (EST) Received: from ok.72ps.com (HELO v2yxg) [146.129.46.94] by adsl-68-121-82-89.dsl.irvnca.pacbell.net with ESMTP id 0B7AE7C8516; Tue, 16 Nov 2004 22:42:03 -0400 Message-ID: <657ijdk$4-dj-oc-bc6$07ft-g@cu8.akpwhni.3c> From: "Administrator" To: moler@netlib2.cs.utk.edu Subject: ADV: Staff Announcement Date: Tue, 16 Nov 04 22:42:03 GMT X-Priority: 1 X-MSMail-Priority: High X-Mailer: QUALCOMM Windows Eudora Version 5.1 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="D.E.0___00_0" This is a multi-part message in MIME format. --D.E.0___00_0 Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Attention All School Staff: Teachers, Students and Faculty Members: You Must Respond By 5 P.M. Thursday, November 18, 2004. Through a special arrangement, Avtech Direct is offering a limited allotment of BRAND NEW, top of-the-line, name-brand desktop computers at more than 50% off MSRP to all Teachers, Students,Faculty and Staff, who respond to this message before 5 P.M., Thursday, November 18, 2004. All desktop computers are brand-new packed in their original boxes, and come with a full manufacturer's warranty plus a 100% satisfaction guarantee. These professional grade Desktops are fully equipped with 2005 next generation technology, making these the best performing computers money can buy. Avtech Direct is offering these feature rich, top performing Desktops with the latest technology at an amazing price to all who call: 1-800-795-8466 by 5 P.M. Thursday, November 18, 2004 The fast and powerful AT-3200 series Desktop features: * IBM Processor for amazing speed and performance * 128MB DDR RAM, -- Upgradeable to 1024 * 20 GB UDMA Hard Drive, -- Upgradeable to 80 GB * 52X CD-Rom Drive, -- Upgradeable to DVD/CDRW * Next Generation 2005 Technology * Premium video and sound -- For enhanced colors and graphics * Full Connectivity with Fax modem/Lan/IEE 1394/USB 2.0 * Soft Touch Keyboard and scroll mouse * Internet Ready * Network Ready * 1 Year parts and labor warranty * Priority customer service and tech support MSRP $499 ........................................ Your Cost $227 How to qualify: 1. You must be a Teacher, Student, Faculty or Staff Member. 2. All desktop computers will be available on a first come first serve basis. 3. You must call 1-800-795-8466 by 5 P.M. Thursday, November 18, 2004. and we will hold the desktops you request on will call. 4. You are not obligated in any way. 5. 100% Satisfaction Guaranteed. 6. Ask for Department C. Call Avtech Direct 1-800-795-8466 before 5 P.M. Thursday, November 18, 2004 Visit our website at http://www.avtechcomputers.com If you wish to unsubscribe from this list, please go to http://www.avtechcomputers.com/announcements.asp Avtech Direct 22647 Ventura Blvd. Suite 374 Woodland Hills, CA 91364 --D.E.0___00_0-- From emarketingcenter012@hotmail.com Sun Dec 5 06:20:36 2004 Return-Path: Received: from 160.36.58.108 ([199.203.183.117]) by netlib2.cs.utk.edu (8.12.8/8.12.3) with SMTP id iB5BKJY3002255 for ; Sun, 5 Dec 2004 06:20:28 -0500 (EST) Received: from [99.102.70.19] by 160.36.58.108 id l7mZ1Mg5p7aA; Mon, 06 Dec 2004 08:19:42 +0200 Message-ID: From: "Corp Service" Reply-To: "Corp Service" To: mpi-comm-archive@netlib2.cs.utk.edu Subject: online advertising Date: Mon, 06 Dec 04 08:19:42 GMT X-Mailer: Microsoft Outlook Express 5.50.4522.1200 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="BA_.8FD..7CD_." X-Priority: 3 X-MSMail-Priority: Normal --BA_.8FD..7CD_. Content-Type: text/html; Content-Transfer-Encoding: quoted-printable

FK4MK MV^JW$G9V0.#:M*:M,VNK@+4>KE%KXAJT>JV-J[ZPJU0*JMB%QRM M@HS-B+XBHV>%\CK>,:\HC/SZOM*OW,;."KBVKS:C A :D*\,BOF*OKZ<]>O. M>K\"L()>G5>S"JZTGNWZO\:O#6P8@;ZZ!;0>\(JUXJ_AS /KN^:O%6RA>> 1 M%:G%1'H)-A60('#A$_@%Q!\=<$DP""C!);%I6 1M0$V0!M!U&<&VFAC0=?9! M8D#\L0%UQ!J1$:RL=80OL49\K,R@5) Z@!Q)*K:3' JM/J.WF.\1HF9#-0G% M^A/M(FMH30HK_ 44NQ*TBTXL[?D*_15>[!)K-F"A*6I#<+7B#E*L-/JQS 5/ M+,*:A!0E&0<<"SPZD5?K@0#&@I#I28]YM8HB6>R[*3[B 'DL'0NG=9'.) TP M=XJGR"G=-Q:,?B1L/7&EF,6;(?F=%(T&[!*,Y"I<1!):( MEJIA& 80?O2C$AOJJ;)X10UQ4N 5.!YY&J*,CF=#+.MTU*,?;#\JRY:@)0>. MEX2D#JULU(I7F(N^+!M+S$*OQND\:1(*-LDI*AC)Z@1(0+=:R=X!]*2@M\KJ ML<'L&P'+^G@F(1)T3M2RW*S9P,?FLC->-RO(@K/]:&\#S,H?-8@VR\X>L^FL M'@O/+J+)K+&WS-H,OJ8:"LFBL//!8'B7AA&I+!N[/XZQ\R4JZG3HI:Z73X8>>E@VM-%J%LH8PI$5;2[J>!VUPV5YFM!.M=-EA.K2S M1&.ISK:JUFPT2D[>EQ>JV6!*AIF$Y+)02_ZF5*EX^5NREP#"ZW@LUI*6*=+G M4D*/\R7F1CM&L9ZH$[N#SK*T9[X7T8IOQ*,J:T.^EH^BG%AZ(J=5*M[WS$H! MT:P\@REJBMOE6%@-GAMG86]3E'AY)RV.!YZD#BKMC)G3!K4O[1D[.LJT-J1) MB#4>)U!MG%@H#J%4K1JJ5-XX6EP-<2]\M69A5.#,HK ZK-/PSP84X$:H-\>V MJ@M!T&D]!I<.IVH)13XX&6!14ERBL:VHFGG(JI;.9$D@UI:1,FTTVM-:HQ>M M. FKU7L[+6"I@LYPOF5C>X)ZLVX!47MDMH^P): XU8H=5:VE*,E&LWZ"'. G M'GH"K3ZYV.)X>H(1F=M>M;SM1?#;GK1D[1=+W-H,QJT^^_P]LSL*]+$&Q'V? M25(@.< +[.IJA[)NMVA!-W'@5;=""N6@BQ!\>PAI)YZFLM=)W=?'E(YD*?0' MWKH*;0AK2*<)E/.ML4!^)I/.))21W[8AT&-)NV/RDXSD?QL45); &_%HX*:0 M5>C]Z-\2?,9"]JF(YIZ](PBPX,:5DJJ%"^$"N&3DJYD3++C0W[M9G"J=:"AK MB..HH0VJ9C$VDG;=XR&)3CXX*^M\NXPD@O6$E1K-6@3/RD(0$K"73VB,2_#- MN-9M?8CW7:EY*ADP%9B.L^K3.6W*#P#$CTO>UKA#;C3;LM*P?213F9_2CM\M MP??D"KG0;,G:1X8$J.L3FN7""UON\X?5N@K^@Y&KG(*3S:79Z9WX$O-MF8OW MG;E3KFXPV%Z*6D:4^5O.ESDM!ON&CJ0^;14J8N*TV^>K2>A"E^"G%MI+WH^" M;E[)DV*;9.5=.::H21N(TIN2K=6+0I[YFJU 6N%ZJAAFS5;O2=(NHFX[N099B*> M+BVN"^PZL08H"IGKSI?.917*ZTJZ]6B8*4[JN*-C*IGK,KNC8S0ZT4*[0:UI M6U*&FK=N4#MYCKANI[%G9F@T(JR&)^?!'W*?9Z ]YHP[8Y3)7."X4B64$2/4 MAT!;=ULA2 $MZ\WP2'JMCIZQ4>3]IPV"G)HTR@'N9*NH+%8()PD.D=ZFNZ$> M%))>G(R:BL^Z;B2\?LLUOSJKPD[]%1IZ&\4@#2F^\ED38O"C %2+U[&LH+ M]#:]3^?FR/.&O!POR]L^@)9?;]'K%W*]OP*,:_.B-D_ RIOO=JCX9M5K!+B] M[,JAJO2B=2DOW?L2Y'L]9]5+!.B]Z4G9Z_-NO?ANW1O-H;PXKTZ0[X:X=R\* M( 0 OE)EXQOS*K[LRJ0&]4*^:D[C>P1 OE(G+KO[@9JYT:SJ>V6VH,CKIO+2JL>JO$@0DK*GZK\D$/"["&JU*?93@" M?A91PFHP98 (@!&* +4!!1@+EAI'@AW1(E(5!P&?)P"\!&^A-TBY9!8&I;; MY1O!CM".9*L^.?^.CO4O4-%!8I1]*V.+FMIN]=X;P3M>E@'PKQ!1]K_A[4=J M'CXX!K"#,SI&E 9PE#8Y;I7'1 1:ZKZGYT0%W)'2::F'%WD<)L!EPHXK+-B_ M(J:;V0/>CAAH -Q,>I/OI@W!.W*: 3"I]CK&P"=%POENUL!JSO084TFP0%D#.\%-8M3:04K!J4C^NY0:JL-* MK2F4AH\2\%**49ZBO&S[\UE2D7NI"?QJ H^D@0:*0DZZFRFG"0W:ESIPA]D# M!Z5[<'1I!%>@'&4AF5GRCN(E%7P%&\+'G"8<7=".83"2*0=$M7>M'@D"XWIG M\+V(9[AZ>-_T2RH6MELMA6FNMI*I9AMLZ9IQH8$2.0OSCG8F,@?8JJJ+RC":Q_[J7QJ""[&[VF4:50&L+ !2SDZ6B>T^4%3NI#.J=6@N&F+]K.20&9M]TP!D("Z'='A!*&@L+S@![4N#$Q5Q]L>+:>45+4""=]@;= MXBA(W%1\6[&\$&7!.IX,8^GFY<&X?"M"W2*&X& MJ_U+XN%'_!KF+4BX\*8DD: ;,G^TJWC?KE@4H(B!(NG)+H*#0&(*2]&)Q=YB M<+CD%7/EA77X#DJ,JS#"BA CO^UP2.@5_W[^H7:L7,2_OFHU"![KFL4<]+?, MS'4'L=*I'CL37_$SZ!Y;"O!QVUD?=\?L<)6@"OL9!]YKJ1"'A-AQI?@?<\5@ M1GCLUPW(X7%;2AXCR!?KX8>+/AK@*.?R6L:6CP;ZIR4"B6%Q@(PD6HF&17QL M(-_'%^L0T!2"@:#@&M -A@L50GX\\"[(JNLYF!V7R&#&B%PI\LCRP4Y\, [) M&J&$?!90R#;RA1PX\'4FH=VR(C*#Y_'TVBGZR.+@@\P=A\>.77T<'LN, S*+ M/.0UR= O6C 9L+\LHA@LU8K(\N"/W!P'R(.BG)@B%\COQ(&\)&?(OZ@0<'S4 MR**PDJF,](!4,I\()(?'K-YC,!_#R4IR]G>Q3@:7(F$@L'[(MBV?G"97R7_R MF @H#LH6LIQL*'_))8!I>)'4 73(:Y 1- %9'^3']I5]5!_:QQOL@T$ 9F Q M/,@ZP.CW I@!=H$"DB%,!"#"16 XB!^A\M;7,.@6J\7H%RU^#.#Q;K#^>8LH M0/TP!^#*:Q_7]]6A !*BU[BGCC/R0]4 ,K0/P#*.80>>!;[RP2A 3 6SA80R M+%<:$F.AG"UOA,)R/2$P@,K)\JY\WI4>K_)K(/LARZ(RUV>6T!E<0IR<63PI MV/*GR"ENRQ0!<.$MUQ/5,I'B+0;,$V.7,"Y[BQ:"V5?UV8+2B']'@[3+?:,' M0@LNS%??BW O@HPFH8X8!L@31<$5@#J@!5MQJ_PJQ\JOP:P,?> 3MS*Z'"_O MRG,)^_$K]R3!\H2R!Q3+K\&QK#+KRF' LMPLPQO3,;3,,VLJU?*]? M$\RK<-.<,)?*#'/4C-Y1,!"S5R Q*\RF,J- A%S,3#+>=^YB!FZ#7>"RF<2? M6M"@5N@@AZ /PH,$:OE!"9=-L8XE@2,;\+*-ZL?;6.1!?V'Q>MPE2!L1&"FSG4G3@P[;\AVY]X),'JG ./SAQB( 77 &: BC _:24:@-<(AM0*A8KS?:>@,%.RK &,DFSM'QVM![_P[(Z7"\]Z2-4R1F,CR&-NV MMW;*6/+,,L^;(OG,^"Z_YRK^R>6.SJ6B8?M@1IGD;N+1V&K&#N3@W! 4SN!S MP#MWFH10@$<7_UF#(X(Q6!,TR.)@3> XGP4U09^L'DQU]T-H@2UPOQ*J@G & M=G\B -N("SH:"LH:\6N$QPC?L@P=<3:!E1SRA[ M2-#WMLRE!6O&$( $) &@<@H0MXD!1F\*D%BB""A )*A&VP5L]''C9?R\H6%S M ^9M*@!)W,*4"; T5Y!0J<#I "2&EM@!*@!*( 9C49# M 8:T6\!&+P%4 "6]R9F\9<8?'4AST71 "H &H ")=,C'2-/1:0$D+4FGT4() M&WWT!I9PM!Q=)AQZ*( =;0;@T !(1K(TP' 2G +2T$@!B.00J &='29@1'!WQ3 PI@(C"1I:GE 4NU1 M,]6 0TC]5&_3)35JU0U'1U6 Q!*=8#*&S0,#TP*< :$U7B!(1W-L=$Z M-5;M44L*' ືU1##KG8&.-:HS5H=5V/4X/1M9UC])L!2[/1T(?&5]"Q 94T+7T26!NHB!825V?71]S[FD\' < % M&J >"--OQIHA(;X!<;17MV8 V()C"N &W*N0 5@ 6P\!]9T.0E_3<6Q&!8%@ M[]>@X&E0:1('FS47D/)6=5Q X2A,)P&\P1WM%ZX9SL9G_5C;V#, ?CU*=P=D MX5F08TO4>@A*<,CU!1E!WA:H"0N9W$=YQG+7C_2-#7,YTYET-#U)4]/6-#:= M FC3W+3*.UG3U;]U518@U-$2M94=K_%IV$X-XBADUU-0$&$$C-(+]AD"&XS3 MI1IH$*F]@*HW5@ M4&:PT;^U!.?@0!E_-"2]'S#2!+75FV:LL)(U'!T$]!-^05S]6$,!/$U)]89]KV* M::8 J1N]]FN[VP*WM:UO']S"M@)0<-_; [?!77++VQZWR9URH]S%-FJC-------------------------------Cut-Here-------------------------------< ---------------------------------------------------------------------- / e||) Lyndon J Clarke Edinburgh Parallel Computing Centre e||) \ \ c||c Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk c||c / ---------------------------------------------------------------------- From owner-mpi-comm@CS.UTK.EDU Thu Feb 25 12:19:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26006; Thu, 25 Feb 93 12:19:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06312; Thu, 25 Feb 93 12:17:23 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Feb 1993 12:17:21 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06274; Thu, 25 Feb 93 12:16:42 -0500 From: lyndon@epcc.ed.ac.uk Date: Thu, 25 Feb 93 17:16:34 GMT Message-Id: <2153.9302251716@subnode.epcc.ed.ac.uk> To: mpi-comm@cs.utk.edu Subject: background information Dear MPI Colleagues I found the February meeting both enjoyable and stimulating. It seems to me that the question of process groups and communication contexts remains fairly open, and the committee has taken steps to move forward. I am sending the interface specification for the system which we have implemented in Edinburgh, called CHIMP. It doesn't contain heterogeneous support and the such, but it has a very flexible approach to process groups/contexts. We've used this extensively for both SPMD and MIMD application and library development. It's coming in three parts, first below, and it's a uuencoded compressed PostScript file. >-------------------------------Cut-Here-------------------------------< begin 644 interface.ps.Z M'YV0)4) F=(B")DW8LJTD.$"AH(2)8;(*1.&SALY.D"0L9,&SAP0-5S0R%$# MQ) WBP*9/QX!@7&],XE4B13IHW;HA4I KB2ADR((R4$0,3!@@8.73(B$LC MALT<,YQ""7.FS)R,-/)"W-OWB1PR93"""#)G3!DW9)P*>5,'C6 M;8T<-D#@H"'#*9$W8^JT>4S'"%@Z?T'L95/Q\IL64MZT">-&-M#:;FY/9H-V M-O#;2>@ ]>FT".23;5:[@?V0B)4D Z'OALSD,MFL'C6BQLJQ^O6!1^O(<9P1 M!)4R6$!0I@.GSLN8>!LRC %#!UP9.$A6QAF702$':E.404=&=)2!APMP'/A0 M" J\\!X61/CTD@PUN-73&"\A9H9[\&4((@@)$=C;"T[L(6(?(H+P@A![B'$9 M6B\Z(>,4>S@X!AI]Z/@"%GM, 8(3?2@@A(Q42+$''7+P-@=M#08I8QI3?F0& M4',4):0=@:51&HZZ,2 (C0%55)QSNB7I M'(@J"I8"8(I95*&-@I!F4$T*.N*HI5;:6Z23FMI;JF."T *KF,($ D]HO>J> M% KL!F4:>("0FAP33:?L2LV240<<&ET+ @T@]/72@96AQ:V/:(!07XC:6HMM M2=Z" "YD((% K@+G7?2RY[W9U &5& M&6?M,8=R=-0QQXJ*.6*=XF8RN34=%'@"#,,.18+\F\[)X*&#&='(TN_-K/;_1+,]+ZCCQ M2IC)*$8878(0'U-1YB&C$%')'%Y\,]91-1E#H,'4'@I$SI00CA.7F+[.^=E3 MWTFC_1B.UPILQAMOV$O'Z=/UD1/A81A>LNQVR T"&V_P9#/1]-HG8]-+;_%R MU +/,:)=0KX-Y=UKHQWUU&%4/;P9<[;Z@O&"RZA\W%M@[Y!;V&MZ!M!"/T\U M&R\7L0<<*(& M;H:E6&&=ZR_U.7JK[W\8PMWI$$&'6C8P_[(4)$P* !^4\$, M .,UASJPQ5OZ0T,+T% &EKA$@!(DH'*RA:T$GF&!W&K@ Q44P1;@ 2EFV$,, M -2G#!:0@[=[S ?+=381=DM!G')@"?. 0@RV0(-A@*$'%Y@3&[9KA3=0@ A+ M2 8\^!"(0I3A NUB1!(J;']IV$U?GOA"^-&!;&6HT-XN\T$PNB]@P&O7"X ' M/+L0ZTBR>IG_GO8".=R!CNNAXQFP9;0*C8&/:'O!$>CHO4!:3@Z8>UQB)!8& M.Q3E;'8Y$!N"8B3XO>!O@6M73F[DH!O200%&FE'BPA">=AE)C0-\87S8AZTU M0NEI VR66_9W0C.,:'\\M*6S)$A!"W[2AK1$X2[YY[\%OA&7PKS?3WYTEC)P MQ#'#[-__RK6_7K;D)1^C&= 4@"=?N< NP=0E,"68RQ%]JXA.'=P M0XV6BH:F+O0%4T@"IQKI)1"TB26748">VO &1UI$1E9P546HY2SU1(L.^*J6 MMNS2+O@9BUPVJZNVC$4L!;C@)K>C@\"B8(4^"$P*A1U4&>14%-SI+D:L-!=* M]%6&HF;5GA.[2%'F4-4[P,&=(#CJ8314H0*OD6BH^_(&!AX+U M NLD08O+#1V'0@,[P^G(3B#@4 T4L(?0&8F*!1T?_-QR-DXVJUYCL$-O0G8Z MI]F@!C? 00Q<<(/06(]718G/??.[W_X>D+IH"%.O7"8D.$BLL_J2PMJTJ:=N M!DU(0CC-AFP OO0&4@YJZF0?ZQABPQE-1G80&(BG8CC&K;B3\2FLPJRPAYZ P4G0KG(^AC(Z.H@21]UPSVXY0:>O$R#Y& 'H' 0 M!1F: QQHDX<4E"$.\L+#CVQ& P5X2\J)J3(;4. $^%#!RV"^B&$CNR7&RFJQ M73)LG?.L6,;*"S)[V,,9YL#58WGS3M]40).$1<6?3,5=(6[6BPVWYZ)(N")[ MR/ ; M7.OLRSGF>82&;S:5A!$]J1AFY!HE5MET6_6-(EYI2CR22L2NMD;9G6 M,#S=^>ENB1J?[[RSGQ5&6,H-NM!0DE)]V^ NRL +K^]""UZC$B4W&.\BS":K M65/W:IO)@0W>.:M#)GW&\7T[W-R.=+F]#6Y-G75^D_3U/37[LC#L0=L*>L/P M%KO!$9_/1=IZ 6+8L,'XN$4.^+;(RYI0)'/%C^!!?./YAMIP;$W\BGMH ;>: MD"2%D2'C9^.XP@*V$!"(7'L9M\O)7W &)YO\90%4N1SWD).5JV$/(7_9&O:P M\815Z XN1WA9\_TRJV%+XR"XP\O:L#Y?G4WI"I/JT7,"=1F]H>G=K/H+XM!T MNVA=#D4.;])?-B=L/?UE@L46M[3N1+<82>C;?ED><"YV208%6V$8ZAOFA-6. M01"$@J(HDND(10- MZ&MU3.'\(%._*"JPI#?5VDMXK$(0N!=L22#"&<7NW@I5P0UIB,/!8C][WGS\ M1>MK'RN%G>?Z H4-VGR]6T+33;N'N0S^= M>Q]M\1 7EN+52;G-U=2E'&4M\ MY'M!$Q2TDC%\!*\QVO'GW:" SOV8C#T2<[F\:,:!36= B1%H41!JY/:9E"WK M%4/!!P>&)3Q]EF>4$S87M5;,DEIR(C,P4 ,P:,R 9$(#]AH":P(3 )!@(VL$*!!"8*< ,X$4A5 M14=VD#HG]@()YE.OV!C"(B2\,3ZY. 1,@!UXU#\)\A@@8&M'4T=HD ;)V!O, M*"-300=35F-"$AP-@HTR$@8TP(TO,!5G !38^#) T"E341E1T18C!@2[Z"E[ M<(ZSIHX5Z"BZZ(LNHS! ":S%H_H6 ;TV'THAH_FF&"JHHHBR#C$:(S(8XX M)C!@LEP+"06_:(YC &X.]@(3*7:!-S)594OWHXJIDX\5LH\H%)(O<(N"8X[! M.!4"$XPKJ8]VE 8J-%WV.)/+-9/16# @AC R^8SDQ1\"Z8QID)//N),? T?Z M.$E.%(YLT"SY.#*3-'=.V6+FJ!Y-B94Q69+J095=&9,RHD7_R),'@X)_AA;[ M(C.'2 =5X&"UU7D:@Q:[U5MO"2(5T@9HT2[Z!S4:LI=?9E1=,EHG0HEX&4.G MUQ:I5YB[@0=#5"Y;%)?)0CI*=I@RX!9OE%2S@X(+=8*%28GAJ"#7J"_:&##Z M(H[DJ #Z\F-/@@?L$S@UTP9T,(1;@HIQI1')9FW+9A3#HAQN\ *?E3V:XED5 M04T8^5D E"Q,L7.W>6T(!R?+F9L)IV^&]9S8YIS5QIS,UFY$9UC,EQ-V%U#7 MF9O=>8-!09T(QXJ^XIW6QY/*5IW4UI[G&1KJ.4E>:)[,)I_DZ7KVB9[S&13/ M$G>E=I%[5P;&"21]4* :R 8@50;"N4!^4AG6J!O).2S0$CO#,FI%P7+^A =1 M(UG8A+J<3).HSRNYP:.!%QF M0!LMMQ=2)@2T,0:O-2CZ$INM&3!P@ LE!X@*78D@=F.BR %EE3L*#CXZ52 !,X<'";-A;" MTDU.&B$UV"Y/ZC1TA4/@9T/@UZ=BQZ=ZZA#M$J@G!7ZQ$E-EBI$@@*:0JHUJ M^G%LNFYPZJ%R2J<4 S)ZFA>&ZJ>>M&Z"NBENT:@C(JD= 0*/NJJ4ZGOKHZ=_ M^A*#>JB>Y*;'$D=[<*48.2=?*GSM(P,WP*EV&J:R-J:.FJ:JBG> ]JMP0"_M MDP/$VB!W*J:/1JAG$ZK"DJB;4JKC0Z@YH:UNX2VXZJVYFJR0NJSNLVF5N@?. MJJG2VFR=:BZ?.JJ$.JOENJBF>JZIFJ:MBBVOVJQQVG_8(JS3ZJE.$ZXX1*B( M"JC=JJ_"J)NH2GR;-6MBD >@$:M.(Z6Z$UGJD2=]5E]82[,]FOTIB.C$T=$FHW8QB7]N'VV M\B@C&RS8YUSL!RR9TJVSIE:#UH_6A5U0.&L+!3X*$FJQHR\>D11S8B0#IE_\ M90,'YBA((B2R!5SO@0=&P!("$[8D, 5<1093,*^GUE5U^1*+V9B9R&-WR1=C M:6LQ$AP(9[2>TE;0PAKL(V6&YIO&0P)?&;:P(2-TJY66*X*2:P8D,)7F8CR7 MV[F?^Y2A*[:8.[F&L[FIZ[GO);H?X8*.^P:02[ICT&(50K=C )6M2P)S\+JN M>[J4&S?"RY0Y!(#!TF"3.P>KJ[K"VY7T,KF@:T.MHB-T^[O"R[S'>JV3N[O9 M6RW!0K?&:T.ZFP?1Z[G:^U*3^Y4VM&C7D[/N!(Z) 3@$&K\TBQ)D8[\-5H,B MH@! H):3UY;Z8A48N3[2:[KBRP;-Z[F:N[YR(#LD.!V/.QW\29X+VJ#EPBV9 MR2W]Z2I#=U;,9[+Q9B3:>58DC&ZZ6<*I(Z!=4J!],JD,6J >''=",CIO&[.A+?SZK/YQ"8\IHBB>)J;.(GKMR.A.(H\O%F%IB/'AFJ(NQMR,K6R M*\&T2\%-XK^@N(A 48JS27"QZVP9@QIDO%QD\* 4K,9T@#&R):1P%%L*0L2C MN <3J0!V\9?!*<-C]<&ILXH'I\(V$Y'GYFY__!'@%\@HS,+U>YP7V1$+58LJ MN6B[" +\N+U% 9,'\L/("#%2A9-:.;XGU2IR<)2>3#FE7)1?.;VCC"F&I2X4 MB\FW8[K@![J,*@6&EI'%H//,OF2[T6^+)3:[RUK,#K-K$8:!+%2)$Y MH<0:/QC@JH3N,@P=1@A:,L[>U&540,S,*/03J'0/LO=[M_=[N'=_T#=\.,=_VC=_N MC=[H/:R?/=[G+5W=#0-%\-TW4 0Y< ,Q, ,^0 ,X$%0\8 02/N%S.N$4+ET8 MGN%BE>$_N$@;N$27N$B/J$#+ETMSN$][MT[CN%!_N,\/J<8;N1$CN0;KN1"?N1)/N31/ MON11#N4^+N5!+N7>K> V< -$@!=X[.+8'51" 3;^1 .B+0,XKMV??0-#<. ' M/@/J7> %'N P( 2R.!HQ8 ,-7F9$P ,W,.'29>$D/N(@WNB.#@.* M#ND7?N@2[N/\X><(7B9H^^)V4>C2)>4QH.=)+NH^+@09'@.FCN$S( 14ONJM MSNJJ_MU$[MZS/@2U?MK<3>L"KNL$WNN?[NN2/N02ON.A7NGKS=T^;NO#>NS+ M?NBA;NL2'NKJ[>Q&8.MX/ ,Q<>@R$>@O7A*%3NE(/N$[3NFB3NOEKNPF;NN? M?>SK3NOMCN[,[N-&L*+E/NSR;@3$/@2AGNSAG>[A;>NT'O /_#'+O#L3O#= M3>XD3N\"+N&V[MTRD.U&L.V"G@3;'>JRSMT\$^Z,ON]&,*ZD?MIX$;_ ?%?7Q_>NZ7O6]?O7'GO4/C_5>7_!13_ A MS^P*X.\?K_'&+NY+C^0+'A.@7?0^8/'D#NT3KNQU#_9X+_5#(%8&G^0_O]ZZ M[MZ![_#W_O#%;OCZCOB!__=:W_A[;_5=O_73KO86/OGVWO8)/O%P;_&5G_1* MW_F._^MX'/FA[_>GGOC)/OC57OBL#_B*C_BGK^&\OO7\;OHQ?_:)OOJY3^68 MK^V;O]UT;MI+?_OE7N#S?O+0?O)SVNQ\/O/++O,P_^LOW_+A??:3;^OJ/0-G M+^I)U/S,7_,5COP[+^3[GOND?NC+'MY%L.PK-/04'_='_^R2+@1*?^AXO/JT MSOXY'^_\G_].__] C^DAN@!X[F:>_UMO8D7_&<#]=P ;( /I_FXW:<;]A)N!55!-3>NJ-Y(/ #BD D-P*5GPD,@20P!9[ $DCSQ K*TX P M\/-ENS#W^PA?[OMY-I#VZ<#9QP/_WNB+?;:O]@G!'$ R&-R& QV#=JA.# M9] ,]@PSF .$P,AK@V_0#8(\.#@'C8#-*P)WT.;1/#WXV5S@S=N#?S +7L%! M* BCX.\[X29L+UM@G=6R=D M;_IO_24[4;C>OERR,X7K[0;H/U5XVMS;#-!_KS#9Q4+ U^R,0/F3<#F ^^FY M^$-[HD[[*;T :/C*1((C>ACPZ.$^M:?Q&-ZQ.WONSM^- M/;V7]\2>V#-[)FX %CMBA_3I_N"W3]$BCZN%_J^=%@0 M"^*(8XEZKB#2P\$8]NHA C2,A!$Q%L;$V/>R'Y)SC'PN_241H[@(#2+;JP%N M#R[40&6G!0DA9[2( -#=^;@_N.X4 !\DC>GM#=K!.A@'VQT<]'&OT;O%QIC@ M!F%C;92-MS$F++A6^ 53':13>4ORU&_*S?J,ISV\XZP3L"%QUSW\Q3Q?UH'F_=KOMYKD[5C$-\ED1WW\4!DVQ-6F8[;);CD*.64HA5DD'$101(X$/GIC)U8 M>8@M\N&)NA+IZWYBB.R)2/)(*DDBJ>>,7)#\'@^Q Y*Z&V?X8.+;2X<-TN'M MPY#7XKK;)@1VZA$H"KOOX?P2'H@T?N[Q['U'_H!$PIRF@W_"D"YZ1YF7_IH? MD=-P,^_?I3\)Q_X&()?DACQQYAG)0=G\"*6@]'Y0LOF%0WU7[WJ>BN1^<:Z] MP<2]:/34W^X[?%QR_27#9H"O%(;';BL81 M3@:_AX?^6"2]BV^>,"I&0VF9#:>EX,.!N*_8)3DA,.2B/S(Y(["T?NV&2*"W.KTNCEP)E9%_G>KAR( MP;)7XDQ@R3._A\X4ECTST@G-"R?TSN'[XP\!HM!-O"'P CL@W;-W(2^_9#\\ MYQB'5=4TF)$0:UK-?KBAP"9A,/XKG,MD) 6[V< M.YA^\-7MXXBZ+U,6PVY8**]AM;R M9)%2:!Q)'Y4BWB MQ=PWY(:G=ZL+L*YTRLSCER8K(Y\DEGH.";9++#W*!+?P=/B.:3H\2HS=/UU( H< 5"P")')( M?KP65_2I+[+D]O6<"I(MI;P!:N *:)[UG WV@=1*">K<[61J1 MW0?T;T@NOZ2_RBCODB+;*Q.J<.*U(J.'"+.NB7Y+"=L)IX]=(^SK^O2"&K(R#\HJ>QW^%!$Y<+A5],L(.V MT8P2 3]W"=FH=UNCE="-_L8=]R.!8T:THN23A +10Z?][)[\^XWED5]"T>HW M1:G?/W1R!'*0W@!^V43/XA!5A/"Q N)1+,H7@>@&S(6,$.$=R/0W'/DCL&ND MC31$ <@<,$IYWX!4$D-1YJE2'1G]F"@I)7)"H#SB/D*JZAA?R?)SE-3HB<,J M>/?FGO-;?DL1V3DY?CGENJ.JTW)4;F!>.:C'[,S>W;-P3Q/_1;PKZD//',R( M##6@)*R0-K?=KF+M$WY[3KJ<3!&X/=,?5#R78O)W KNP*5W.J614=:J.[?V\ MGW=.P2FO>WA!( 80@1L0!&Y";FMP#J'043EZ"D]'HQ <X)3%LA056 ( MQ'7OM)W^SL!' C'<.?UY0W%5[M,AX$]I@+)TJ.ONQN$ M!FBI"2^,XV,;APJ M/7&8YS1@216IZ@ZDACO^P!4CGD>5>P"SVB%.6Z?O6&>O>W@<)@C$(IE0%SPJ M#3B?%^_O\'54ZM4=J5]. M:+43ATRSK][5O;I(EY\7+:S<;;#R3X='X=AD"'VK-Q7XV<]@YPH+G&#MJS[1 M)][/4,E.T:EGO9^9]=.MNYYZ+5^@^/N-JA#!M:+("B\?8G76W02DC99RN8O,$GD#F*EUOJW=]@2F5:!97$?I/,^"GS*N)U;"B5Q2( M0?U;I1.'8M-K-M05*%-SGWV=H#$@M;K5X]HJ0QZ'&7:5];O95V%CD8W%LDP6R3O;']KJM.:STW/4KD,\QB:C* M(_LH'V5=#:S_+6#*U_06WG) ]GNE ;*/5K]3.37;;'S-FJ^5=,=EV.*W+9^]@932,6#(-J$ WV M1DCK!7M&I66#UUMQG+^5YL MASV3&M+4*4-9ZT2A(:T=@V2VS.;:$+5K2:DR]+4G#]CVN_7>VJ7;70$EKT"F']JOJ\FWUV M!')70:M2B5VBS;,T5M@AO?_V\+XLBXV*7O2W-MS@6OUP'WQ5?B-0USG;I4VM75S$A(3RU["_B/=;CNMD@'*0; HSRK[9+#A/RWJ=\([8X M5ADJO1]9[^2I=OQTF[7<-DFT*6<1W?]$K8HVPRH\SY<,7>&LG+!--[ 2.%?H M75UX$'LAS)VT:+:07@:\V9N+;5GU]2JW;1K\\SN'\R/ M2-3G&5/BF%]!&Y=EE!4.[WI8.FHMQ^JU3'VO#_!Z52K7544KX76*6'6J\KCT MEP#KI%/UCD]UJ3I.(TR](ZCCNC]39]K,8%D0DZ^EJ[:^5QV: MU&^KW]Y;J 6!G'2B_LJ.*V73Z=8;K.O6P?;+&(@%X^V\M7?V;LAUU@1K@#TK M!;; ZO0"JT<%F8'5*00>AU]PG]Z$5:EGL6!#U, =.$&BX J\@C$P"^[ 'UB] MB940K.U(L _ "Q#NCZ*].SG@3"*XO)G#4R-R26:9!)G@U]-UPXI=5MS5&6-I M:A'(<_]T1BK-I A_=V63J\+9D@B,PWW'_K(?OWMWBC$,2[V2&0^?G!FULI>P M+A)='Z=R[6Z&S9*9-_&M.^QZ3CEKJ$RHSE#PF>!3"4"YVQ.VA9$U^"I?5HH@ MTZ.!A',*-]_Q2--FY&@GD7S$KQ,2PTYD9XC)&W#ED=).]Y5-@*=R+6^KO)&5 MCNHRXDU(;,WM_A5WG14/4T,]+.GX\&_TPU X$$_/08C?/AL2G*^4;O_FS4$; M<'OQG\VVA,\6KMPW7#;E';V[>6GV6_(Y)+SUV.W)4\;]3@121B4L0>=DMO24 M#7<8!M,4BSF?X?>3G.30_GHW9!F+X9^X-(A5.&=6X0&G3[?PQX.*/VY= L ? M*(_[WP,$C?9XYOU%$YSGVANV6[5'EKP=Q0Q:43TF75R.OI@7_]E!RT+!*]HD M?)9552HX67Q.Z:9%-.+2;FIA?@C>GEP]AKOY^"O? MXWI,CW??5^S'5-DJ V(XR2ZU9=3,FO"3VYKB=#N7'>R\S)UO65O.8EL(B]&1X,2IFFO?CSC!(U,*(+UNF8YYYA0WB+#V([0\' MJ&7O:_P2W?K#??CS ^[/H&@066GWS)' DQ 3O]5\?S5>.3:?,6Z[_636.>)H M[D@=MOWN_*E)8V(5U"CFR9V_?ECI%F5 M_;*&3:O?EZS*9X^(??EJ]36\%A6<2M_%RY_;KW]FJDFN3+3#JTQO>_/S_(/< M>,6666(K2HD=*35\M'?VV@##%P0F=).KT+B7&E/? OM!::K95+HG-/A^Y^*[ MH?I6. M/$X_S\*35R>5'-\#=>]7'7=7 ><4U2_S/75EXG 2Z%T:BR9<:(W+%U0 G\ MS.Q(7BT\M&S2-?OE]@,VBH)^V0*F#23( !7TX-8T@# ;=@+*R("<$!-@ IIV M"V\Z31]5F""G:0:4,6T*X$W' !54$M+TG+;3'*(MH&D1-:AGP-E(TX.Z+L#I M\+(%.H39H!DB2E)+:A82J>V"4[73(>I1$.KP4A(2M:DNU:B:5*OJ1#VH75%^ MW18U8 O4F*.J #I$0^!P,L(X>(7@T *& UJ0 V/("!P/D% 2PG81DQ'A2G-Y;#*,@S&1@(G'[)<^)T2 +G<<>M!;$?%G!0L;]8U7:Z(84? M:9U33!- X[$WRA6_8 WP 37@?%I3(S ^KALW_:\[#RHB01H YWSJ$(#7#Z[0 M<;GJ>>H&=@(F 9B: M4K.Y\"*B/G7,A@DMFS2 :OE4J1_<#%!!9\/!@6HZO064-FG 8[%Z+O#LJ3T2 M9+7K.!N:S04XZJT=&EZ KK8-O1I1_.I@793.]*&.UWV;?;9K! M+6KVMF#3H-I.Y #&C9B.JM6> >.E+K@ &\"Y.?>#\]J_859N#01(. MAJ>U:9O."W'[:+?M5FV[(?>;%MIIVG&;ZKQMM&=VH7[4C1MF%^K[9KL;=Z'. MV];;;.1M[2V^61'-P--K.WV'%_6MM].WVE[?JEHH+6\\3;F7=TX(459[D6Z+ M_=*Y/7>N#MU@.S=LAXP6K.N)V6[=;MIPY^[:+;<#':(NW*KZ%<*$YKV\RXSN MQMUKVW@C:K]]M!,UI#[<39MP4VH'CKG-QNQNU-M[:9MOH82S]_;KAM]Z^WQO MBY9=8T*UYN[?-N!S?VU>[:L).-@("@MN<@LK$VXV##<-* E.6P5Q,,B-!$%U M[P[ABIK#@(1*[1#6M"RZU&_:!F3J2LVH@S;3%MIPVRZH.:M=IUFGUM[:GQIT M4XD +J'8S^F&;))-DKR!,T"(W,#DN19.02U4!/5 %E0 TE(NPH\,F(Q(9-G6 MCUX@)8EA"JB*C! $RHSBP5L* H\/!G>2$?PV'K,91 5;W !M6L(-2!P: HH< M,5"P/4 "D@ 9R @$JI^X@(F1!UB H]@O+/LNS( 7 -D "V_VS5 !WPV'2"T MQ8 :< -WI QA"Q+0!U"0 O 57PYHBZ@@ @2&F$4#X@O.JQBON7 V^(4"J]5- M' AHI0'C5;Y2#G<+;:)_>)7[\='J^&E(#=*!\J3R,2 &[@#N. -OP 40*!$4 M$ 9"03@("0%/- 33@!I4 VM00+$!"2P6L^(3# A$@ I2@2P4 2@ @&$SZWIOD?2$V?!2(,!G"!1G1 M#)4\FE-SD-#$!\PCUPZL@9X_!*' T(? 7; +[SQUH Q8 +N0 J0%W"@G[3S MK"[//<)#4 58G[LD;V3 !) 32)M,*"8]Z\7L'=F35OG.=NB#R2 _E4A)@)B MN.MN0A [&&+CK.2&N[W7Y42$*.X&ZZYYM6]R P<[8Z\A9..QN/9DK M=L*>*-8"4W#K_/HM4'8ST-B9PAUHZS9@L9/V2\(4T$!;U\RJ_2'XIO,!V6^ M"TC4FYVQEX 7<#Y.>P*P 2*A!L3VV1X]V$!MU]HE(;>O]MZ.VI'[<#\?KCT! M8.[8+D:H6!7S% D@OU#WZV'=R8L"2 O0*"S 8+^$Q+ Z(D:/"@!S 'K_H_4 M^ZRI'ZCI@##V"D$;Y$!?\.[@7;R3]S!@WF\$?$\ ]3VHV_5_!-^3!%^_))V] M&DT9_,Y9>$M\/_!^7<$G!OR>E'X$4W!A!GZ^"YPH<0<<^]^0 _A=%]&!!$ N M8CMXOP-D@,3+'^J> %!\ H !":"0Y9OO#N-=0HQ78=\=Q9>;%W_CT M%*(3:BL#7SW"@Z8)OS=22QC1\L,IHUUVJ6814-L8"/-G0<*#^#*O@]#Z43M. MCMV?>'D4\.83@&.7\TH$2IAY.U_/T( "<.S88L_#@:_^Y\<\B(_IQ9W-/Z+> M7C"*^_SA\!Z>*1AY^H3F\7QG__"2_M";=A31VC4"IO_TFOZJH?508WC.? FX M\R5%J>/@Z+0MAKA3Y19DX-0?#%!/Y\]\@Q=.]_T/089S[L^GPS''9_[BF0-M MDFT37H=7<>9>#)J#<^,5"Z4Y,G-P#D&;$Z\;D,V_TB*%,N&\*)VC.?;,S7D_ M3^>HO Z,@5\#(>BY0" (!@$A*(09T!"H^A0P BV WCL$B, 5-+IB, (3H0P@ M =\# E %4 (AJ,)7(25<'Y> E" \W["VA@,$'$64(3A4 )O WT!LR1T$'\ M08\*4R$CK/M[ A.^29GA]QG=5JQTLG #7@ .> &K$B;$ !V 7X9%&Y ,SN9S M9 :[D=39R[#:%G/J431Q%+ 4/D)E"B^L$R0DP M&-))5%@45@8QG(^;P$-6=/]Z7_M%?+,B!>D)4P@538!GR01JI"3;0 OP$[E /6X7@N+8R M0/Q;O\Y'$7;#NO(2V'^^7\5W&9A_EHSN-_,9!0K"&- &Y %K 'H-!P5: &E M M@;"$2!9A@JU 1B" N=\W,0-D!%8 XV^Q_ M6@(-(@),?6>? ,<;B !%H.=A+H@:K 'A5R%,,5:&].&$98=*8@60'\C"S&(]PD944%18"D$$9\@'/,W""FA1LY7(2@%9LS]YS3P M+])@P7 'K#"]'_[7 @B!E0J*$*2L 6G=_$< XG^X7CEH!LQ_!HTX:/<9!:I" M"S 1+'E4RH$0^24M$9^@)R?("R[ &=#_7036PD10%A03Y4+W%^ U%@\=YV(, M*AA%00\(_:D0@?Z""J%]6J!5NA:>?:>0"S 1.%0YV/\R M4P:X01;N@TY %G@&?@U;0!= H5@:1\"2H&8<#!\!"@ *#=IASK MIB%BB,?A?'@@%G$/SA'P(HJ!)>(]&&^@B 7#,_(%%@5_ WW2(M8'D-^CT1YF M-B7?=N@D;@=,HFOH8QXJ]&']N$,P"UD 8L%[G 'O "&03!R)0:)18&6F _Z?C>*IK 2 MNA-BHAM %$P2O(69:!ZZB:PAM[B%PB )A:R MD M0%!WS(6$G&)(^/Q)?P0?9W$R$ =]XF!X%MQWA=_H!\9X!ZJ# P@!2H \8F+0 M!GP$2(&[< :P!3$%I2*#: J>( 2#U[P$: ;D-Q%,&>9?#T('N(H18._G- @9 MB4%1$-!LBM+?'G#6-( /8+*(.X!XG:*T"#\()5\&"&"S7("WH@28)'2*V>+_ ML2T("\Y'7D NKHO>8@WR>KP1X$< 0G!XBY%%7.=ZC B0A 7#+I*+Q=Q TQ5 MB_LB^@#=D+EE0LEHW=0U*@.VHD:D,6 *;A95!/ M*!,%8Z^'FAQX!X$0< 6,C(R-5##E080LH\I(2@2-FH++^#68AFN 87 %/"/4 M"LUX+>(83$T8(894C)^%E$$Y 'EI7A031CB-%\$R4C4H**P>AG(S:G@CW\20 M5-@ N%[GP=>5C6T,$J,B5(Q>XE30)82)E,/S%S=N%AW#1[ .5HW301#($ Z& M=@$*P 0@ 2E 32 #U 0T 4A0$YP-+$"9L1 =U &:'#;A7H+@6U7$I $Y\1" M0 -,CI%0O;<0T'M0QDP@$C NVP8WP>9PCINCB((GV'9/1@Z0'-X,(4%.H I] MA:3CC(()!<:C0.^ICF7"Z+A(_16U8YO'.YX39P+MB">\AE#& M\)@3+(_U'K>WN16/C\A"\!J**)H9?W$[^'B:X^28 VR.;L'C^#HF.)ECS*8] M2HXJBWI0.BZ-U:!A0"4FB2 #)_A)P'IH'9Z XR.THKVZ.79C[1C<_@ZKHZ< MHVWWX. )(0'V:-MI,.JC[9C9$(^B8TEP/(8&9@N4<3RZ!7,*[4C3&(_\P>LH MBVB/OL+]*#L"=W:!KV#;Y8XA 91Q03XBR!SMN,E-CB(D\;A!AI MP/WXX%20 M#\X)R4UDD!0D0_"Q!(]VP6A /!Z//21PESY2&0KD3"@:N(XY 8Z')]![$*3K MZ$ N.--C_CA YC;XXWZ!0!*1&.3F^."L2MKC[3@^;HX2)!6Y1<:.U.,-"=RY M!=_C_YA?T8XXP>C(/W(+^:,)F4"V>>QC@^ ^5HG7BOS8Z1EZQYSY&!I@C@^. MEQ=>') @0!H9&H07FV-.(#KF!'/;](@#%)!EI$W@2#($#PZ?=DZ8+2'D BD[ MVG9#9.DXP6F0D*3,IT)NCIDD+"?TG1,VI"@I.^I[T9_K>#8XC]'?_9@[XHZ1 MI&@P/4J/A^0*$3T*D="?+MDZ3HZI)!19$FR0C.0:.4,R!%"&)7E,:H^I9&4R M.F:/W *.M[X]DZZCB!)>, 1_I.X'#=)[E^-^$$ER"%$D+1E> '>7(W#'+8R3 M461(4$/.D8]('5D&W)'PXR8H9>R1:)TE^4>^AH&D'W/,?3?UGO^82YJ3H@$G MV4N&!!0D<+=.8I$HY ))08:2&>0,^1H6DP_EH^ Z7I&>9"&I0GJ0T9_E^"CP MDM"?0?DHJ).UY!GY4;*0CP+RB$3&DOWD]5A+GI2;S>O8%FAMH@%!V4MBB*(! M0!E,[AJ2.>7T*!.RD2HE5!A%FI+!)/A82.*4O22]MT5JD;VD48DY M4I0S'DQ 1AJ2O23K6$@:D?9D(8E/EI'N)#QI),PM!).C[H@FZ),W MPRUY/YZ39.4,:45"?VVE: Z1I)Q9>=X2+8 S&,AF3K"E9LC'+E7II(M0$CP M1X*1<.5:>53ZE"U -7E8@A/LY VY5?J/)4$'>4F&DK^"(YE(TI4CI&A 2<*5 MD*-FR4*V "(E#4E7A@3#I!,)5\Z5*Z0U259^D,!D8]E5R@'OXU?)-\Z/A5X] M&5X@CR"E(%GR<0OY1;W72TZ2OF7)!T%>EJFD<"D:8)2^PFMH%Z21E:39$$Y: MD(TE],=44I707V;)14:6%N5R:=%%DCCD$3DZ5I?/I$#943*0@F5(R4!>EM ? M)-E2AI @ >>85Q:73.4&>4XZE\,D:1G]>9,KY$&Y/C*-[R1LB4?&C_,D_4@2 MPHU"8D)7)%H-H]_=2.K1)\_?$V ?:">< APP9(01!.3HB-W0COOC LDM:''G MA&0IHDB/C"1>Z4(2E]%E26DFH)=09&C 86:23"0KLED&DRTFR^92FI@@P7X1 M8JJ4F%%KB5".E?>CB))C/H\:IM*G1H8$&^:+Z5KREX:!CNC)@)6?!'@QLB * MW@$J^(RT!.#&-;$I(I47IEV@6VZ6%R9VR66RDM"?4LF*T'N=9%3I7#XXK0CM M" U.CQ>F-4E?LB+O98G98G(8)"5#<#9DF=$?ENE-VI QHQM03\R61P+ 804F9%\",$@>N@?\&(P'-N@)4Q1@ ,(@CT*#O:F*&> MK\!47IB,I"WYX'@VQ"/T)T."!+\C=ZE-RIC1I(^77+*9#$&(J4..F;MC!KEI MZIBE8Y_Y9^J1X(5=(#)&F6,E_W9*GI6>I(@Y3$:40N5U\QKFDF0FG#D]PIBT MXXTS.4J3MJ2(OX9'":U.:[ MF6:&, IG"7DSA)S,9&-)D+1DFDEL1I65I8V#7IZ9ON.,*68. MCS1AB;E5#H\B2KZ)3+(BBF4+:6WNE5ZDBLEI\IFXAY\I;AJ:9YRN0D?>F MEHE6ZG-0QKUY3@J3K,A^84UBG6FFB )M7C?E);798SJ;D65@.3E&G7 EU+EG MFIQ)YZXY3S*=4*:Y.6!^@XAF,W$^S FCG_)7W.F-6<-Z=WQL@HNF7W!EGIVT MY)9Y=LJ4:^:^:5^&F8=GR;EC,I[[YLQYW!!KV"9#X&,VF\4E*WEO-I=(9;SF M4Y9\D>>SR7;*C$MGRMETUI-FIUV0M4F=Q28I"1+@J:I^=8 M65YR;M-CP]ER6I6>)INI4=:>*J9M)T<^GOD>F\E1:AS1)B%9>UJ;9"1@*6:B MC6IFZLF* '=WY?0)?Y:1)^?;:7K&G6=>PCDY@IJ48_!XW?R;5G\8>W$EH M;HJ&9=96$NB6$^7K66I:E,%G"?I2IJ"BY^.900:?R*5%N6^>E"_E[HE*1I>V MII"93%:;)BAV&8+"GO2G\UEPWI\?J*D9;?*2).@1RD%:E&BF>-E0LB*$9#"Y M55*?M*=%.7':=IEDFIAN$H]2Z.O(?F*7)R7U&6J>FF)H$$IZ#J'0)_YI6W*: M"@Z&B5;>EK(C!IIX*BU+:+.9/$*@CZ=S>6MZF=$C-GE?NI!::.89/6J'>:8* MB53^EK>F?@EN"J$H9QKZ@2Z1V.:%&1IDF'TEK1D^4I$+Z ^Y0$Z;<>6AB"0G= M;G 94 YBI=U8=X*):,%\0&CF>\MG(>DZLIX*) 7*82:1KJ9%N7P4H]*D4\5& MGII*Y&=S3MP$YP2.QX$VF41HE'DWSHTF#2JH8$X,\L18Z8J:#1AEPSEK\FO3 MIJM960:?R*;9&7R.GT2FD(E[HI#;PO18?&Z/ALRF6%Z.!"2E \J.AIH"J4!9 M7 J;UB@SVH]NH^/FDTEHVJ)>8$)7-R*($X.S #A\B<)HE$D_PI41CQNIZ>B/ M;^A"\%;Z"N0CE$%MZI]LI>1HDC:C*21;64&NI/UHGMEB=I!Y);6959*DA*5- MZD?"E9@C3&IFHI,A3*"C/=JD[650ZGWRHS^I<[E#0I$])TOJ7-Z5V*A/FGM& M?S-I8NDZ[J!D94IZE49_6>G- .(G,5E>N*19I!8Z:S).W:6'.E8VEDBG>(A M@-F!"@L.*;!9&\(+$>FCD9&:FQPI< <\0I$,9TAJ/JJ6::14BH\"D(#I7'F2 M!H^JY>PHF *,Q:@(>93:I% D#)D?+*;WI3:)DPZE+&E'&J;PI4CI5,IX+J6& MJ5/*@'JEER516I7*!"2E/@>8HJ5IY6KJ>%:4K2E/RII6EJXI5DI(+H^]8UA* MFWJE(B=<:=OMI7.E;ZJ)9KG)V"DH ,:G\5V"#V9B<-/= M!#* ?=I9X*=A(["!"GJG[D(8$-S,%B((1VI8(G-YI6ZY$%RHOX+(>9+* )6H M"+F5ZJ29I8B*F<:DT* LR5G6I"QI\'A7TIF4:2\Y?R)SCZEHBEA^.2=J4JJB M8J@YJF<:0WZ7H"6+BJ("GYQEC8JBX@D )8W:HPZFP2-;":-VIH\(0ZHGT)31 MY_,W.#H+T4*<6!M8"B+ /W@RF $BP*;XB7*6O:.&BG4B<[WC8>$1C,@YJE^JE)R]!1RL D@2JX M,:B*J> D6[E"]J45Y><(I"JF4"JT"$[>E:RJ3EI>(G,C:F9:E5*?C>KI"*MZ MJ#Y>>$BG IH"IJ&I(GH%1,;H]P+^&'%BQ3>L[G]3W520%*R#!)^;X!),$ 1? M@5(7X@%?7:UX&?B)G,(SLEA(&YEB45 AX 'A8_XHHLQU>$ (R47ZFO.=27@Z MX'_!8I>@"%H*$\SE@@)T&=MI_D>L9C+P0L@"+S@-IZ+@^0+D 6>#3*AQ&G,# M:ZB)7[R. $.2FL5:99<'7*(@*L/:KBXC?")BL!+T,/9$"\ :5!LO08[B M%5"&12