Streaming Tdc Control Protocol for J-PARC Slow Extraction (StcpJsx)

Streaming Tdc Control Protocol (STCP) is an upper layer protocol to distribute the timing pulses and the command to the front-end electronics and to gather status information from FEEs. We developed the dedicated protocol for applications in the J-PARC hadron experiments where the DC beams is delivered by the slow extraction method. Thus, we call this protocol StcpJsx;

As the StcpJsx is still dedicated to the point-to-point data communication, the routing function is not supported. The figure represents the overview of the protocol. The data structure is asymmetric between downward and upward data path. In the downward data path, the 8-bit commands are sent together with 32-bit register values from the primary side as the payload of the MIKUMARI frame. In addition, the pulse transmission is supported. In the upward direction, 8-bit status flags are sent back from the secondary side. This is also realized by the MIKUMARI frame. As explained in the MIKUMARI section, there is the CBT character transfer cycle and the transmission priority among character types behind the command transmission, and then there is a range of latency from when the transfer request is issued until the data arrives. Thus, the commands are used for the semi-synchronous operation. The status flag update has also a range of latency.

STCP-OV — Figure 1: Overview of StcpJsx.

Pulse and Command definition

Pulse definition

Currently, 6 types of pulses are defined in the StcpJsx.

Pulse Type	Index	Comment
HardReset	0	Transfer the hardware reset pulse. The author expect that this signal is for emergency use. The front-end module should be reset by the reset signal generated by the local bus controller through the data link (SiTCP). It is not expected use this signal in the normal operation. Users may assert this signal when the data link is hang up.
HbCounterReset	1	Heartbeat counter reset. Use this signal to set the local heartbeat counter value 0. It works to synchronize the heartbeat frame. The author expects that the local heartbeat frame number should also be set to 0 by this signal.
GateStart	2	It provides the data acquisition gate ON timing. During gate ON, the data are streaming in the TDC and go to the PC through the data link.
GateEnd	3	It provides the data acquisition gate OFF timing. By de-asserting the gate signal, the data flow in the TDC is stopped.
VetoStart	4	It provides the VETO start timing. If the VETO signal is ON, the data flow in the TDC is stopped even if the gate is ON. This signal is used to stop the TDC data generation without turing off the data acquisition gate.
VetoEnd	5	It provides the VETO end timing.

Command definition

Currently, 4 type commands are defined.

Command	Index	Comment
RunStart	0	Change the run status from IDLE to RUN.
RunEnd	1	Change the run status from RUN to IDLE.
HbFrameNum	2	Global heartbeat frame number update. If the command is HbFrameNum, it indicates that the following register value is the global heartbeat frame number.
GateNum	3	Gate number update. If the command is GateNum, it indicates that the following register value is the the gate number.

Register definition

The command transmission in the downward direction, the command accompanies the 32-bit register value. What the received register indicates is determined by checking the received command. Currently, two types of register values exists.

Register	Width	Comment
Heartbeat frame number	16	Global heartbeat frame number.
Gate number	8	Global gate number

Status flag definition

In the upward direction, 8-bit status flags are returned from FEEs. Current definition of the 8-bit flag is as follows.

Flag	Index	Comment
ModBusy	0	Set to high when the streaming TDC is under the busy mode.
ModReady	1	Set to high when the module is ready to start data acquisition.
DLkinStatus	2	Set to high when the TCP connection is established.
SEUStatus	3	Set to high when the unrecoverable SEU error is detected.
TimeFrameSlip	4	Set to high if the local and global heartbeat frame numbers are miss matched at the timing of the delimiter word insertion timing.
NotInUse1	5	Not in use
NotInUse2	6	Not in use
NotInUse3	7	Not in use

Relation between streaming TDC and StcpJsx

The StcpJsx has a role to control the streaming TDC on the FEE. Here, the author describes the relation between behavior of the streaming TDC and the STCP. The data acquisition condition for the streaming TDC is defined as shown in the figure. The acquisition condition is defined by the run status, the gate signal, and the veto signal. The streaming can work and generate the TDC data when the run status is RUNNING, the gate signal is high, and the veto signal is low. The StcpJsx primary module set the run status to RUNNING, if the DAQ run start is requested by the PC and all FEEs return ModReady high. The streaming TDC does not work under the IDLE status even if the gate signal is low. After setting the run status RUNNING, the primary module sends the GateStart pulse to open the gate. Thus, the streaming TDC starts working. The data generation can be temporally stopped by the veto signal.

GATE-TC — Figure 2: Data acquisition condition for streaming TDC. Blue and red objects represent the command and the pulse, respectively.

The time chart for the heartbeat related things is represented in the figure. The HbCounterReset pulse resets the local heartbeat counter and the local heartbeat frame number. The local heartbeat frame number is internally incremented at the heartbeat timing in each FEE. The global (StcpJsx) frame number is updated after the heartbeat. These two number must be the same and at the heartbeat timing. The streaming TDC checks the sameness of two frame numbers, and if they are different, the FEE sets the global heartbeat frame number mismatch flag ON and sets the GHbfNumMismatch flag in the delimiter word of the streaming TDC.

The author expects that this signal is sent before opening the gate. As the gate number is inserted into the delimiter word, the gate number should be updated before sending the GateStart pulse.

HB-TC — Figure 3: Time chart for heartbeat information. Blue and red objects represent the command and the pulse, respectively.

Application in J-PARC hadron experiments

The author expects that the P3 signal from the MR is used as the HbCounterReset pulse. The P3 signal represents the flat-top start timing and is the important reference timing determined by the accelerator in the slow extraction method.

The gate signal corresponds to the spill gate in the J-PARC hadron experiments. The extraction starts several hundred ms after the P3 signal. The opening timing of the spill gate is user defined. It defines when the data acquisition starts during the slow extraction. The author expects that the GateStart pulse is generated as the delayed pulse of the P3 signal. The closing timing of the spill gate is also user defined.

StcpJsxPrimary

The StcpJsxPrimary is the command and pulse sender in the MIKUMARI system. It is implemented in the primary module, and if there is a distributer module, it is instantiated as the command and pulse sender to each FEE. The entity port structure is as follows.

entity StcpJsxPrimary is
  port
  (
    -- system --
    rst             : in std_logic;
    clkPar          : in std_logic;
    linkUpIn        : in std_logic;

    busyPulseSend   : out std_logic;
    busyCommandSend : out std_logic;

    -- Pulse input --
    stcpPulseIn     : in StcpJsxPulseType;
    pulseError      : out std_logic;

    -- Command input --
    stcpCommandIn   : in StcpJsxCommandType;
    commandError    : out std_logic;

    hbNumber        : in HbNumberType;
    gateNumber      : in GateNumberType;

    -- Slave flag output --
    stcpFlagOut     : out StcpJsxFlagType;


    -- MIKUMARI IF --
    dataOutTx       : out CbtUDataType;
    validOutTx      : out std_logic;
    frameLastOutTx  : out std_logic;
    txAck           : in std_logic;

    pulseOut        : out std_logic;
    pulseTypeOut    : out MikumariPulseType;
    busyPulseTx     : in std_logic;

    dataInRx        : in CbtUDataType;
    validInRx       : in std_logic;
    frameLastRx     : in std_logic

  );
end StcpJsxPrimary;

Port	In/Out	Comment
IO port
rst	In	Asynchronous reset. (active high)
clkPar	In	Parallel clock input.
linkUpIn	In	Input linkUp signal from the MIKUMARI link.
busyPulseSend	Out	The next pulse transmission request is ignored while this signal is high.
busyCommandSend	Out	The next command transmission request is ignored while this signal is high.
stcpPulseIn	In	Pulse transmission request. This is the std_logic_vector whose length is the number of pulse types. To send the pulse, set 1 to the bit corresponding to the target pulse type. Do not set more than 2-bits to 1. The MIKUMARI pulse transmission function can send only a pulse type at the same time. The request signal width must be one-shot.
pulseError	Out	This goes high if the user requests to send more than two types of pulses at the same time.
stcpCommandIn	In	Command transmission request. This is the std_logic_vector whose length is the number of commands. To send the command, set 1 to the bit corresponding to the target command type. Do not set more than 2-bits to 1. The values on hbNumber and gateNumber ports are latched when any of the bits in ndpCommandIn is set to 1.
commandError	Out	This goes high if the user requests to send more than two types of commands at the same time.
hbNumber	In	16-bit heartbeat frame number input. It is latched when any of the bits of ndpCommandIn is set to 1.
gateNumber	In	8-bit gate number input. It is latched when any of the bits of stcpCommandIn is set to 1.
stcpFlagOut	Out	StcpJsx flag output. This is the std_logic_vector whose length is the number of flags. The bits are flipped when the flag information is updated by the secondary side.
dataOutTx	Out	Connect to dataInTx in the MikumariLane.
validOutTx	Out	Connect to validInTx in the MikumariLane.
frameLastOutTx	Out	Connect to frameLastInTx in the MikumariLane.
txAck	In	Connect to txAck in the MikumariLane.
pulseOut	Out	Connect to pulseIn in the MikumariLane.
pulseTypeOut	Out	Connect to pulseTypeIn in the MikumariLane.
busyPulseTx	In	Connect to busyPulseTx in the MikumariLane.
dataInRx	In	Connect to dataOutRx in the MikumariLane.
validRxIn	In	Connect to validOutRx in the MikumariLane.
frameLastRx	In	Connect to frameLastOutRx in the MikumariLane.

The StcpJsxPrimary hides the txAck cycle of the MIKUMARI link. The pulse/command transmissions are managed by the busy signals trigger by the transmission request. The pulse and command transmissions are requested by setting 1 to the bit in stcpPulseIn and stcpCommandIn, respectively. For example, the HbCounterReset index is 1, and then the pulse transmission with the HbCounterReset is requested by setting the stcpPulseIn vector as 0b00_0010, like stcpPulseIn(kHbCounterReset) <= '1'. The request signal width must be one-shot. If the request is accepted, the busy signal goes high. When the busy signal is high, the next request is ignored. This is the same for the command, but the values on hbNumber and gateNumber are latched at the command request timing.

stcpFlatOut is the flag vector indicating the status of the secondary module. There is no valid signal for this status flags. If the secondary module updates the flag, it goes to the primary module and appears from this port.

StcpJsxSecondary

The StcpJsxSecondary is the command and pulse receiver in the MIKUMARI system. It is implemented in the secondary module, and if there is a distributer module, it is instantiated as the command and pulse receiver from the primary module. The entity port structure is as follows.

entity StcpJsxSecondary is
  port
  (
    -- system --
    rst             : in std_logic;
    clkPar          : in std_logic;
    linkUpIn        : in std_logic;

    -- Pulse output --
    stcpPulseOut    : out StcpJsxPulseType;
    pulseError      : out std_logic;

    -- Command output --
    stcpCommandOut  : out StcpJsxCommandType;
    commandError    : out std_logic;

    hbNumber        : out HbNumberType;
    gateNumber      : out GateNumberType;

    -- Slave flag input --
    stcpFlagIn      : in StcpJsxFlagType;

    -- MIKUMARI IF --
    dataOutTx       : out CbtUDataType;
    validOutTx      : out std_logic;
    frameLastOutTx  : out std_logic;
    txAck           : in std_logic;

    dataInRx        : in CbtUDataType;
    validInRx       : in std_logic;
    frameLastInRx   : in std_logic;
    checksumErr     : in std_logic;

    pulseIn         : in std_logic;         -- Reproduced one-shot pulse output.
    pulseTypeRx     : in MikumariPulseType  -- Short massange accompanying the pulse.

  );
end StcpJsxSecondary;

Port	In/Out	Comment
IO port
rst	In	Asynchronous reset. (active high)
clkPar	In	Parallel clock input.
linkUpIn	In	Input linkUp signal from the MIKUMARI link.
stcpPulseOut	Out	One-shot pulse output vector.
pulseError	Out	It indicates that the primary module tries to send more two pulse types at the same time.
stcpCommandOut	Out	Command output vector.
commandError	Out	It indicates that the primary module tries to send more two commands at the same time.
hbNumber	Out	16-bit heartbeat frame number output. The value is updated when the HbFrameNum command is received.
gateNumber	Out	8-bit gate number output. The value is updated when the GateNum command is received.
stcpFlagIn	In	StcpJsx flag input.
dataOutTx	Out	Connect to dataInTx in the MikumariLane.
validOutTx	Out	Connect to validInTx in the MikumariLane.
frameLastOutTx	Out	Connect to frameLastInTx in the MikumariLane.
txAck	In	Connect to txAck in the MikumariLane.
dataInRx	In	Connect to dataOutRx in the MikumariLane.
validRxIn	In	Connect to validOutRx in the MikumariLane.
frameLastRx	In	Connect to frameLastOutRx in the MikumariLane.
checksumErr	In	Connect to checksumErr in the MikumariLane.
pulseIn	In	Connect to pulseOut in the MikumariLane.
pulseTypeIn	In	Connect to pulseTypeOut in the MikumariLane.

The received pulse and command appear from stcpPulseOut and stcpCommandOut. For example, if the pulse with the HbCounterReset type is received, the 2nd bit of the stcpPulseOut goes high. This is the same for the command. In addition, hbNumber and gateNumber are also updated for HbFrameNum and GateNum commands, respectively. If errorPulse and errorCommand is high, it indicates that the primary module tries to send more than two types of pulses and commands, respectively.

The status flag vector is connected to the stcpFlagIn port. If status flag bit pattern is changed, the frame transmission is automatically generated by StcpJsxSecondary.

Fan-out

Since the CBT character transfer cycle is hidden by the STC protocol, the StcpJsx primary and secondary modules can be directly connected. Note that this is not confirmed yet and is just my expectation. Therefore, for fan-out, the StcpJsx secondary module can directly drive multiple StcpJsx primary modules in the intermediate (fan-oute) module.

Example design

StcpJsx_Primary/StcpJsx_Secondary

The path to the example designs projects. There are two repositories as follows. Each repository contains two Vivado projects, StcpJsx_Primary and StcpJsx_Secondary.

AMANEQ-official/StcpJsx-Example
spadi-alliance/StcpJsx-Example-Clone

The first one is for the developer. If you contributes the development of core functions, e.g., CBT and MIKUMARI link, please select this one. Vivado projects in this repository include source files other than project specific source from external git repository, AMANEQ-official/amaneq_devel.git, in order to use and modify the latest source codes. You need to fork and clone AMANEQ-official/amaneq_devel.git too. The second one is for users. This is copy of AMANEQ-official/StcpJsx-Example, but the Vivado projects have all the source code under their project directories.

CRV-PICTURE — Figure 4: AMANEQ with CRV card.

The StcpJsx_Primary/Secondary projects are example design for the point-to-point connection between the primary and the secondary AMANEQ modules. The mini-mezzanine CRV card is necessary to use these example designs. AMANEQ with the CRV card is shown in the picture. The reference clock is generated from an oscillator (100 MHz) on AMANEQ, and the secondary module is synchronized by the recovered clock.

Figure 5: VIO setting through hardware manager.

This example firmware can transfer StcpJsx pulses and commands from primary to secondary by inputting the signal to NIM-IN1 and 2, respectively. Since there are 6 types of pulses and 4 types of commands, we need to set registers to select which type is used before testing this firmware. The registers are set by VIO through JTAG, and then the PC, on which Vivado is installed, is necessary. Figure is the screen shot for register setting by VIO. enable_command and enable_pulse bits select the command and pulse types to be transferred. If you set 0x2 to enable_command, the command assigned to 2nd bit is sent to the secondary. Do NOT set more than 2-bits to high. You also can set the register values, gate_number and heartbeat number, using VIO. These values are sent to the secondary to the commands. The ILA is inserted to both primary and secondary. If you have two PCs, you can test them while checking the received signals on the secondary side.

The firmware is reset by pushing the reset switch (SW2) on AMANEQ. DIP SW setting is as follows (only for the secondary).

1st bit: 0: Use SiTCP default IP. 1: Use IP stored in EEPROM.
2nd bit: Select NIM-OUT2 signal. 0: Command output. 1: Recovered slow (parallel) clock.
3rd bit: NC
4th bit: NC

LED indicators.

LED1: MIKUMARI link is established.
LED2: PLLs in FPGA are locked.
LED3: TCP connection is established.
LED4: NC

Keys	Action
`?`	Open this help
`n`	Next page
`p`	Previous page
`s`	Search