EPiServer remote events troubleshooting

EPiserver uses WCF to send and receive events between servers configured in a multi-server setup (load-balanced or enterprise).

A site in a server will send events (publisher) to one or many sites in other servers (subscriber).

This is an example of an enterprise setup, with one publishing server and two public facing web servers.

episerver-enterprise-setup-2

To enable remote events (events between EPiServer sites), the following has to be enabled in your episerver.config file:

<episerver>
    <sites>
      <site>
        <siteSettings enableEvents="true" enableRemoteEvents="true">

Also, the web.config file has to be modified to enable sending and receiving events. The example shipped with the default installation configures your site to use UDP multicast.

<system.serviceModel>
        <client>
            <endpoint name="SiteName" address="soap.udp://239.255.255.19:5000/RemoteEventService" binding="customBinding" bindingConfiguration="RemoteEventsBinding" contract="EPiServer.Events.ServiceModel.IEventReplication" />
        </client>
        <bindings>
            <customBinding>
                <binding name="RemoteEventsBinding">
                    <binaryMessageEncoding />
                    <udpTransport multicast="True" />
                </binding>
            </customBinding>
        </bindings>
        <extensions>
            <bindingElementExtensions>
                <add name="udpTransport" type="Microsoft.ServiceModel.Samples.UdpTransportElement, EPiServer.Events" />
            </bindingElementExtensions>
        </extensions>
        <services>
            <service name="EPiServer.Events.Remote.EventReplication" behaviorConfiguration="DebugServiceBehaviour">
                <endpoint name="RemoteEventServiceEndPoint" contract="EPiServer.Events.ServiceModel.IEventReplication" binding="customBinding" bindingConfiguration="RemoteEventsBinding" address="soap.udp://239.255.255.19:5000/RemoteEventService" />
            </service>
        </services>
        <behaviors>
            <endpointBehaviors>
                <behavior name="EventServiceEndpointBehavior">
                    <webHttp />
                </behavior>
            </endpointBehaviors>
            <serviceBehaviors>
                <behavior name="DebugServiceBehaviour">
                    <serviceDebug includeExceptionDetailInFaults="true" />
                </behavior>
            </serviceBehaviors>
        </behaviors>
        <serviceHostingEnvironment aspNetCompatibilityEnabled="true" multipleSiteBindingsEnabled="true" />
    </system.serviceModel>

You can test this setup by:

Web 1: navigate to page A

Web 2: navigate to page A

Web 1: edit and publish page A

Web 2: refresh page A

Page A in “Web 2” should now have the content published in “Web 1”. This indicates that remote events are working and invalidation messages are being transmitted between the servers.

If it doesn’t reflect the new content, you have a few options to troubleshoot this problem.

  1. Use the EPiServer remote events listener program to see if the events are being received on each server.

http://world.episerver.com/PageFiles/84954/EPiServerRemoteEventsListener.zip

  1. Install the Epinova.Diagnostics plugin to monitor remote events

http://www.epinova.no/blog/Thomas-Leela/dates/2012/12/testing-remote-events-in-load-balanced-environments-with-ping-pong/

  1. Use iperf to diagnose UDP multicast messages.

Option 1 and 2 are well documented so let’s look at option 3.

Download Iperf from: http://iperf.sourceforge.net/

Stop the EPiServer websites.

Start an Iperf server listening to EPiServer’s multicast address on the target web servers:

iperf -s -u -B 239.255.255.19 –p 5000 -i 1

Generate UDP multicast traffic from the publishing web server:

iperf -c 239.255.255.19 -p 5000 -u -T 32 -t 3 -i 1

This is what you see in the publishing web server, Iperf generates traffic and sends it to the multicast IP address:

iperf -c 239.255.255.19 –p 5000 -u -T 32 -t 3 -i 1
------------------------------------------------------------
Client connecting to 239.255.255.19, UDP port 5000
Sending 1470 byte datagrams
Setting multicast TTL to 32
UDP buffer size: 8.00 KByte (default)
------------------------------------------------------------
[216] local 192.168.100.20 port 62546 connected with 239.255.255.19 port 5000
[ ID] Interval       Transfer     Bandwidth
[216]  0.0- 1.0 sec   126 KBytes  1.03 Mbits/sec
[216]  1.0- 2.0 sec   128 KBytes  1.05 Mbits/sec
[216]  2.0- 3.0 sec   129 KBytes  1.06 Mbits/sec
[216]  0.0- 3.0 sec   385 KBytes  1.04 Mbits/sec
[216] Sent 268 datagrams

On the target web servers this is what you should see:

iperf -s -u -B 239.255.255.19 –p 5000 -i 1
------------------------------------------------------------
Server listening on UDP port 5000
Binding to local address 192.168.100.1
Receiving 1470 byte datagrams
UDP buffer size: 8.00 KByte (default)
------------------------------------------------------------
[192] local 192.168.100.1 port 5000 connected with 192.168.100.20 port 62546
[ ID] Interval       Transfer     Bandwidth       Jitter   Lost/Total Datagrams
[192]  0.0- 1.0 sec   128 KBytes  1.05 Mbits/sec  2.265 ms 1464813651/   89 (1.6e+009%)
[192]  1.0- 2.0 sec   128 KBytes  1.05 Mbits/sec  0.032 ms    0/   89 (0%)
[192]  2.0- 3.0 sec   126 KBytes  1.03 Mbits/sec  0.281 ms    0/   88 (0%)
[192]  0.0- 3.0 sec   385 KBytes  1.05 Mbits/sec  1.223 ms    0/  268 (0%)

If you don’t see anything in the target server, then the UDP packets are not reaching it, meaning the event is not received.

You have a few options here:

  1. Check the Windows firewall rules (highly unlikely a software firewall is up in a production server but worth a punt).

  2. Check firewall rules between the servers. It is possible that in an enterprise setup some non-standards ports are not open between servers.

  3. Analyse your servers and determine which network cards are being used in each machine.

It is common for a server to have multiple network cards, where you can segregate traffic more easily. A typical example is to have a network card for public/external access and a second for internal/management purposes.

If during the server’s setup, the default network card is set differently on each server, UDP multicast traffic will not reach the target servers.

In this scenario, this is what you will see in the target server:

iperf -s -u -B 239.255.255.19 –p 5000 -i 1
------------------------------------------------------------
Server listening on UDP port 5000
Binding to local address 10.20.0.11
Receiving 1470 byte datagrams
UDP buffer size: 8.00 KByte (default)

The target server is using a network card connected to a different network to the one used by the publishing server.

Running “ipconfig /all” will show you the network adapter setting of your server.

Ethernet adapter Public:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection
   Physical Address. . . . . . . . . : xx-xx-xx-xx-xx-xx
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 192.168.100.20(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.254.0
   Default Gateway . . . . . . . . . : 192.168.1.1
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter Internal:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection #2
   Physical Address. . . . . . . . . : xx-xx-xx-xx-xx-xx
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 10.20.0.1(Preferred)

You can change the network bindings in the web.config file and tell it to use the “Public” network adapter:

<udpTransport multicast="True" nicToUseName="Public" />

After this, your remote events will propagate correctly between your servers and you will see new content after a page publication on a separate server.