PCIe PV Routing

The PCIeAPI bidirectional ports of user modules have to be bound to a PCIe Router (Switch).

Requirements:

• Generic devices should be able to be connected to PCIe devices, e.g. via the PCIeRouter. The functionality may be reduced, e.g. only memory transactions (address based) are supported.
• PCIe devices should be able to be connected to each other via a PCIeRouter.

PCIeTransactions derives from GenericTransactions. But using the GenericRouter would mean to perform (slow) dynamic casts for each routed transaction. For this reason the PCIe router PCIeRouter either has to derive from GenericRouter and has to be able to route GenericTransactions or the router only routes PCIe transactions and a generic device creates original PCIe transactions. We chose the latter:

To achieve compatibility the include inside a generic module which uses generic ports and transactions has to be changed into a PCIe compatibilty include. This include replaces the original API with an API which provides the same methods but creates PCIe memory transactions instead of GenericTransactions. For details see 'How to connect Generic Devices to PCIe Routers?' and 'How to connect PCIe Devices to Generic Routers?'.

PCIeRouter

The PCIeRouter has two ports: An Upstream Port (upstream_port) to connect other routers or the Root Complex and a Downstream Multi Port (downstream_port) to connect multiple devices and / or routers to. PCIe modules which use the PCIeAPI have one bidirectional port for PCIe type transactions.

Figure 1 shows a combined class (blue) and object (red) diagram. A PCIe router of class PCIeRouter can be seen. It has connected some peripherals to its Downstream Port: Generic peripherals and PCIe peripherals.

The Upstream Port is not allowed to be left unbound: At the top of each PCIe topology there has to be a Root Complex.

For the Downstream Ports we want to allow unbound ports. So the Downstream Port is internally bound to a dummy port (at position '0') to avoid warnings for unbound ports. The address map ignores this dummy as it knows its position (0) inside the connected_in_ports array.

The router has no protocol_port because at PV we have no timing so no protocol is needed. - TODO: If we will need a kind of protocol for routing there will be another (new) port to keep the protocol_port for phase handling in more detailed simulation levels with timing (e.g. BA).

Figure 1

Address Map

The address map (PCIeAddressMap) is used by the router to get the correct outgoing port (either the Upstream Port or one of the ports in the Downstream Multi Port).

The map handles the three routing types (address/ID/implicit). Dependent on the routing type different methods are provided by the address map (decode_address_based, decode_ID_based, decode_implicit).

Functions:

• Generates three address maps: two address based maps (I/O and Memory) and an ID based map
  • The ID based map is generated during end_of_elaboration (fixed map)
  • The address based maps have to be generated at runtime when the devices get their address by PCIe configuration TLP (transaction).
  • The propagation of addresses and IDs to other routers is done internally with method calls. A Router calls in the Upstream direction (and not Downstream). As a result a router always knows which addresses/IDs can be reached on which Downstream Port. All unknown addresses/IDs are routed upstream.
• The map returns the target port out of an address (call decode_address_based), an ID (call decode_ID_based) or implicit (call decode_implicit).
• : The returned data type is RouterPort which specifies which general port has to be used (outgoing_port) (Downstream, Upstream, no one) and which of the Downstream Ports (connected_port) has to be used. In the case of a broadcast to all Downstream Ports the outgoing_port is PCIeDownstreamBroadcast.
• Routing lookup:
• : For address based and ID based routing: Look for the address range or ID in the
• : address maps of the Downstream Ports and return the target port (RouterPort) if existing.
• : If address/ID does not match, return Upstream Port as target.

    For implicit routing: The returned target port (ÿ22ÿ) may be set to PCIeDownstreamBroadcast to allow broadcasts.

• When changes in the address map have to be performed, the add_address_range_to_port method has to be called by the owning router. The local address map will be changed and the method add_address_range_to_bus_no in the upstream switch is called. Then the upstream switch adds the new address range in its own address map to the concerning downstream ports (where the change was made). Afterwards the add method of its upstream switch has to be called etc... There is no check for overlapping address ranges in different routers. This check cannot be done inside one single router because it only knows the addresses in downstream direction, not in upstream direction.
• Setting of bus numbers: The bus numbers have to be unique in the system (PCIe bus). Each switch manages Bus Numbers for one internal logical bus for the logical PCI-to-PCI Bridge connection (see figure 1a) and one Number for each Downstream Port (see section ../PcieAddressing). The numbers are assigned during end_of_elaboration. Each assignement increases a static member variable. This way each switch gets unique Bus Numbers. Also the maximum number of buses in the system can be controlled.
• Building the address maps: Two methods inside the address map class create the two maps needed for routing. The method generate_ID_map() generates the map needed for ID based routing, generate_address_map() generates the address map. Both methods are automatically called during end_of_elaboration, the ID map before the address map.
  • The ID map generating method (generate_ID_map()) generates m_idMap with Switches (buses) located at Downstream Ports. It calls generate_ID_map on all Downstream located Switches to be able to call collect_served_bus_numbers on each of them. This method calls generate_ID_map on the Upstream located Switch to make sure that all Switches generate their ID maps once one of these methods was called. This method is called by the router or by other address maps during end_of_elaboration.
  • The address map generating method (generate_address_map()) has to be called by the router during the end_of_elaboration only after the ID map was generated! It is only needed for propagation of manually set addresses. Normally the address map is generated dynamically during simulation with special TLP transactions.
    This address map generation may be used by the user for debugging purpose: When an address range was set manually with add_address_XX_range_to_device before end_of_elaboration the forwarding to other routers is done here. For each stored address entry the add_address_range of the Upstream located Switch is called.
• Multiple address ranges: The router is able to handle multiple address ranges since the methods add_XX_address_range... can be called multiple times with the same target port/bus. The multiple address ranges are stored inside the device's register space.
• Manually set address ranges: A user is able to set address ranges manually. E.g. this is needed to connect generic devices which do not understand address setting transactions sent by the Root Complex or for debugging purpose. Manual address setting can be done with different methods:
  • add_IO/Memory_address_range(port) adds the (only) address range which is specified in the parameters of a specified tlm_port. This method automatically checks which bus and device number the port has.
  • add_IO/Memory_address_range(port, base, high) adds the specified address range to the device number the port represents. The address parameters of the port are ignored in this call.
  • add_IO/Memory_address_range_to_device adds an address range - specified with base and high address - to the specified device. Here the user has to know which bus and device number is assigned to the device whose address range should be set.

PCIe Switch (class PCIeRouter)

The GenericRouter was designed for buses where all participants are connected to each other over one multi port. PCIe switches have another structure: they have one Upstream Port and possibly some more Downstream Ports.

• The Upstream Port is a separate port (not a multi port) in the PCIeRouter class. It has no dummy binding because an unbound Upstream Port is a failure (see above).
• The Downstream Port consists of a multi ports with a dummy to avoid error messages.
• A PCIe Switch has to model internal logical PCI-to-PCI Bridges: one Bridge per Port (Upstream and each Downstream Port).

Router internals:

A PCIe Router consists of several logic PCI-to-PCI Bridges: One Bridge for the Upstream Port and one Bridge for each Downstream Port. Each of these Bridges has to have an own Configuration Space (header type 1). The internal logic PCI Bus, which is located in between the Upstream and the Downstream Bridges, is not allowed to have End Points (header type 0) connected. Only Downstream Bridges are allowed to appear (see. PCIe Spec. p.39). (see PCI-to-PCI Bridge Architecture Specification, Rev. 1.2, pp. 15, 18; PCI Express Base Specification, Rev. 2.0, pp. 434, 39)

Figure 1a

Figure 1a shows the idea not to model dedicated PCI-to-PCI Bridges due to simulation speed reasons but to model only the configuration registers for each Bridge. The logic handling actions concerning the Bridges are done together with other actions in the router. The configuration register objects are created during the end_of_elaboration callback when the amount of connected downstream ports is known.

• Interrupts, INTx, MSI:
  • Message Signaled Interrupts (MSI) need no extra API expect a Capability Register (see Configuration Registers)
  • Legacy support interrupts are provided by with INTx Messages (Assert/Deassert) which can be generated with the transaction access methods. The PCIe Router has to handle these Interrupt messages in a special way (see PCIe Spec. pp. 70, 336).
  • : The PCIe Router supports a basic mechanism handling INTx Assert and Deassert Messages: An (Downstream) incoming Interrupt Message is terminated at the receiver (router) (see MessageType). The Interrupt is forwarded after calculating the mapping using calculate_Interrupt_Message. Forwarding is done within the method send_Interrupt_Message. This approach does not match all requirements on pages 72f. but will care for forwarding the correct Message. (Not supported e.g.: Interrupt Disable bit of the Command register, Deassert Message in the case of DL_Down, Root Complex support)

Power Management Turn Off Message and Ack

The acknowledgment to the Power Management message PME_Turn_Off is a special routing case: The Turn_Off message is broadcasted to all devices and each device has to react with a PME_TO_Ack message. A Switch has to forward the Ack only after having received all Acks from its downstream devices. A PCIeRouter counts the incoming PME_TO_Ack messages (in the member variable m_received_PME_TO_Ack) and sends the PME_TO_Ack message to its upstream port after having received Acks from all Downstream Ports.

Topology

Figure 2 shows an example topology.

Figure 2

How to connect Generic Devices to PCIe Routers?

This section describes how a generic device can be connected to a PCIe topology using the extension mechanism.

How it was first: A generic device creates a generic transaction and sends it to the generic Downstream Port of the PCIe router. The router has to create a PCIe transaction and copy the quarks. The data quark is only a pointer so the PCIe data quarks points directly to the generic data quark. The new PCIe transaction is forwarded and routed through the PCIe system and later returns back to the router. The router maybe has to copy quarks which may have changed to the generic transaction and returns that (If we support reads and writes we don't have to copy much/any quarks back!).

This is simple for the user: He simply connects a generic device to the generic port(s) of the PCIeRouter.

For simulation performance this is the creation of one PCIeTransaction per GenericTransaction and copy of quarks (but not data).

How it is:

The main precondition for getting generic devices compatible to be connected to a PCIe topology is that the generic device has to use PCIe Transactions and ports.

In a generic device the included header is GP.h. This include has to typedef PCIe Transactions (and other classes) to generic ones. This is done either by changing the include to greenbus/transport/PCIe/PCIe.h or by using the make switch. Afterwards PCIe transactions are created instead of generic transactions while keeping the same generic API.

Generic transactions are address based so the compatibility API creates PCIe MemoryRead/Write transactions in parallel to generic transactions.

For this behavior, the convenience access methods in the PCIe Access classes which are used by the generic device have to be adapted. E.g. setMCmd(..) sets the PCIe quark MPCIeCmd instead of (more precisely in addition to) {MCmd</code>. See table 1. Now the generic devices can be connected to the PCIe port of the router. If the generic device has two ports (initiator_port and target_port) it has to be connected to the router twice. This is no issue for configuration because generic devices don't understand any ID routed TLPs anyway.

Technical View, Extension Mechanism

• A global define (USE_PCIE_GENERIC) has to be defined before the include of the generic.h (and other generic) file(s). This define has the effect that some typedefs are changed to use PCIe instead of generic. Afterwards generic modules which use generic ports (master, slave,...) will create PCIe transactions instead of generic ones without realizing that. All method calls, data types etc. keep the same.
• The define USE_PCIE_GENERIC has affect on the following files: TODO generic.h, generic.dynamic_casts.blocking.h, generic.dynamic_casts.h, generic.static_casts.h
• These typedefs are defined different: TODO GenericMasterBlockingPort, GenericRouterBlockingPort, GenericSlaveBlockingPort, GenericTransaction_P, GenericMasterPort, GenericRouterPort, GenericSlavePort, GenericRouterTargetPort, GenericRouterInitiatorPort, TRANSACTION, PHASE with effect on GenericSlavePort and many template specializations.
• Generic Read and Write Transactions have an equivalent in PCIe: Memory Read and Write Transactions. Due to this analogy it is possible to map generic Generic_MCMD_RD and Generic_MCMD_WR transactions to PCIe MemRead and MemWrite transactions. PCIe uses an own Command enumeration because the generic one is not extendable. The two analogical transactions have the same number.
• The supported features are: Read, Write Transactions, Error handling, Completion / BytesValid.
• Table 1 shows which changes to the generic methods are needed to allow generic devices to create PCIe transactions. The changes methods are overloaded in the PCIeTransaction which derives from GenericTransaction.

Method in GenericTransaction	Generic action	Overwritten in PCIeTransaction with changes	Description
void setMCmd(const MCmd& _mCmd)	Sets the quark mCmd	Additionally set mPCIeCmd to the same value	Works for the generic commands Generic_MCMD_RD and Generic_MCMD_WR
const MCmd& getMCmd() const	Gets the quark mCmd	Returns the quark mPCIeCmd, not mCmd	Works for the generic commands Generic_MCMD_RD and Generic_MCMD_WR and PCIe commands MemRead and MemWrite
void setMError(const Error& _mError) void setSError(const Error& _mError)	Sets the error quark mError	Additionally set a PCIe Completion error	Sets a PCIe Completion error when a generic device sets an error
const Error getMError() const const Error getSError() const	Gets the error quark mError	Returns also an error if completed not successfully	Returns an error to the generic device if a PCIe completion was unsuccessfully
void setMSBytesValid(const MSBytesValid& _msBytesValid)	Sets the quark msBytesValid which shows the progress of the answer	If the answer is complete (msBytesValid == mBurstLength) set a successful PCIe Completion (quark mComplStatus)	The PCIe Transaction is marked as completed when all bytes are transfered.
TODO: Way around: set msBytesValid in PCIe setMComplStatus(...)

Table 1: Extension to GenericTransaction

User View

To connect modules which use generic ports (initiator_port, target_port,...) to PCIe modules or routers the user has to do the following steps (see example examples/PCIe/generic/):

• Make sure to use the PCIe typedefs for generic classes in the generic modules (either include PCIe.h or use make switch).
• Instantiate the modules and routers as usual in the testbench.
• Manually set the address range of the generic module:

• Connect the generic device to a PCIe Router Downstream Port like it is done with PCIe modules, e.g.

• Generate ID map after all connections are done:

router.m_PCIeAddressMap.generate_ID_map(); // force creating ID map

• Manually register the generic address range at the router:

router.add_address_range(gen_s.target_port);

Alternative

An alternative to the direct connection is the PCIe2GenericPortWrapper which was developed for connecting a PCIe Device to a Generic Router. The wrapper maps the two ports of a master and slave Generic Device to one bidirectional PCIe port. Figure 3 shows the available alternatives connecting a Generic Device to a PCIe Router.

Figure 3

How to connect PCIe Devices to Generic Routers?

A PCIe Device can be connected to a generic topology (e.g. router) if the generic modules use PCIe classes instead of generic ones (same precondition as in the last section). The PCIe Device only may send Memory Read and Memory Write TLPs. At the PCIe Device received generic reads and writes are mapped to PCIe Memory Reads and Memory Writes.

If the PCIe device is only a master or a slave (only sends or receives TLPs) the device may be connected directly to the init_port or target_port of the Generic Router. The user has to set the base_addr and high_addr parameters of the API port manually.

If the PCIe device should act as master and slave a wrapper (PCIe2GenericPortWrapper) has to be used to map the bidirectional PCIe port to an init_port and a target_port. Now the target_port of the wrapper automatically gets the address range of the PCIe port (during before_end_of_elaboration).

An example using this wrapper (and comments for not using the wrapper) are in the example greenbus/examples/PCIe/generic/ (define USE_GENERIC_ROUTER_CONNECT). Figure 4 shows the available alternatives connecting a PCIe Device to a Generic Router.

Figure 4