Unique challenges are present when one tries to build distributed real-time applications using standard o-the-shelf systems which are in common use but are not necessarily designed specifically for real-time systems. In particular, to realize end-to-end predictability when, say, a process on one node sends data to a process on another node, several issues must be addressed: (1) mapping application real-time requirements into requirements imposed on the system schedulable entities (tasks), (2) ensuring predictable execution of the tasks in the face of possible priority inversions, limited OS level real-time scheduling support, and limited number of priorities, and (3) integrating real-time and non-real-time tasks in the same platform. In this paper, we propose solutions to these challenges. In particular, we present user-level scheduling schemes for communicating tasks. These solutions are practical and are based on simple primitives that can be found in most of today's commonly used operating systems. To validate our design and to examine the feasibility of user-level scheduling in actual systems, we have implemented our solutions in MidART running on PCs with Windows NT operating system over UDP/IP and Fast Ethernet LANs. This paper contributes to further our understanding of how to build real-time systems using commercially available o-the-shelf components.