ixl iWARP FreeBSD* driver for Intel(R) Ethernet Connection X722
================================================================
July 31, 2017

Contents
========

- Prerequisites
- Building and Installation
- Testing
- Configuration
- Interoperability
- Known Issues


================================================================================


Prerequisites
-------------

- FreeBSD version 10.3 or 11.0, or later.
- Kernel configuration: 
    Please add the following kernel configuration options:
	include GENERIC
	options OFED
	options OFED_DEBUG_INIT
	options COMPAT_LINUXKPI
	options SDP
	options IPOIB_CM
	options IXL_IW

	nodevice ixl
	nodevice ixlv
	Note: IXL_IW is required for FreeBSD-CURRENT branch.
- For the iw_ixl driver to work, an if_ixl driver with iwarp interface
  is required. The interface is available in if_ixl version 1.7.12 or later.
  It should be enabled prior to usage, as the setting is switched off by
  default. To enable iwarp compatibility, add
	hw.ixl.enable_iwarp=1
to
	/boot/loader.conf

The lan driver can be downloaded from
https://downloadcenter.intel.com/download/25160/Ethernet-Intel-Network-Adapter-D
river-for-PCIe-40-Gigabit-Ethernet-Network-Connection-under-FreeBSD
Or search on downloadcenter.intel.com using '40 Gigabit Ethernet Network
Connection under FreeBSD'. Newer OS releases contain the if_ixl driver in
the ixl driver version 1.7.12-k or later.

There are some known issues with the interface on if_ixl-1.7.12. Please
use version 1.7.13 or later.

- fastreg memory mode in krping needs a patch applied to krping.
  Refer to the 'Testing' and 'Known Issues' sections for details.
- fr test in krping in FreeBSD 10.3 needs a patch applied to krping.
  Refer to 'Testing' and 'Known Issues' section for details.


Building and Installation
-------------------------

1. Untar ixl-<version>.tar.gz and iw_ixl-<version>.tar.gz
	tar -xf ixl-<version>.tar.gz
	tar -xf iw_ixl-<version>.tar.gz
2. Install the if_ixl driver:
	cd ixl-<version>/src directory
	make
	make install
3. Install the iw_ixl driver:
	cd iw_ixl-<version>/src
	make clean
	make IXL_DIR=$PATH_TO_IXL/ixl-<version>/src
	make install
4. Install the man page for the iw_ixl driver by copying the iw_ixl.4.gz file
   to the directory where manual pages are held on your system, for instance:
	cp iw_ixl-<version>/doc/iw_ixl.4.gz /usr/share/man/man4/

For in-tree driver if_ixl-1.7.12-k or later, it is sufficient to follow
the instruction from point 3 but ensure the correct path to if_ixl source
folder is supplied, for instance:
	IXL_DIR=/usr/src/sys/dev/ixl/


Testing
-------
1. To load iw_ixl driver call:
	kldload iw_ixl
   If if_ixl is not already loaded, the system will load it on its own.
   Please remember to add
	hw.ixl.enable_iwarp=1
   to /boot/loader.conf file prior to if_ixl loading, to ensure the ixl
   driver has the iwarp interface enabled.
2. To validate the load of the driver check:
	sysctl -a | grep infiniband
   a number of sys.class.infiniband should appear provided at least one
   port of the X722 is up.
3. The source code for krping software is provided with the kernel in
/usr/src/sys/contrib/rdma/krping/. To compile the software, change directory
to /usr/src/sys/modules/rdma/krping/ and invoke:
	make clean
	make
	make install
   Note: In FreeBSD 10.3, some changes need to be applied to the krping.c file
   in order to enable fastreg memory mode and/or fr test.
   For details please refer to Known Issues section.
4. Start krping server on one machine:
	echo size=64,count=1,port=6601,addr=100.0.0.189,server > /dev/krping
5. Connect client from another machine:
	echo size=64,count=1,port=6601,addr=100.0.0.189,client > /dev/krping


================================================================================


Configuration
-------------

The following sysctl options are visible:
  - hw.iw_ixl.max_ceq
	determines the maximum number of msix vectors available to the driver
	for CEQ usage.
  - hw.iw_ixl.debug
	defines level of debug messages.
  - hw.iw_ixl.mpa_version
	shows the current MPA version used.

The max_ceq setting may be changed by adding:
	hw.iw_ixl.max_ceq=$value
to /boot/loader.conf file. The final number of CEQ is evaluated depending
on the available msix vectors, number of cpu cores, and hardware limits.

If max_ceq=0, the value is ignored.

The debug setting may be changed either by adding:
	hw.iw_ixl.debug=$value
to the /boot/loader.conf file or by calling
	sysctl hw.iw_ixl.debug=$value

The mpa_version may be changed by adding:
	hw.iw_ixl.mpa_version=$value
to the /boot/loader.conf file.


Interoperability
----------------

To interoperate with Chelsio iWARP devices:

1. Load the ixl driver with parameter mpa_version set to 1.
Add the line:
	hw.iw_ixl.mpa_version=1
to /boot/loader.conf

2. Load Chelsio T4/T5 RDMA driver (iw_cxgb4) with parameters
  c4iw_max_read_depth   set to 63 and dack_mode set to 0.


================================================================================


Known Issues
------------

- Loopback is not supported.
- MTU changes are not supported.
- IPv6 is not supported.
- MW memory mode is not supported.
- MR memory mode supports only single buffer.
- The function ib_cq_resize is not supported.
- The max number of registered cq, qp, pd or mr reported by the device may
  differ from the actual number of registrations achievable.
- A kernel crash may occur when trying to run krping without ensuring that the
  two machines are able to ping each other.
- A kernel crash may occur when trying to load the iw_ixl driver when
  hw.ixl.enable_iwarp=0 (fixed with if_ixl 1.7.13).
- A kernel crash may occur when loading the iw_ixl driver on a card that is
  supported by if_ixl driver, but does not have iWARP capability (fixed with
  if_ixl 1.7.13).
- On FreeBSD 10.3 you can run krping traffic with a Chelsio T5 card if
  mpa_version on the X722 card is set to '1'. To set the MPA version, add:
	hw.iw_ixl.mpa_version=1
  to /boot/loader.conf.
  Setting MPA version on both the X722 and the Chelsio T5 card to '2' may
  result in a kernel crash.
  On FreeBSD 11.0, krping traffic is possible with the default settings.
- Krping with fastreg memory mode will not work unless some changes are made
  to krping. To workaround the issue, modify the krping_rdma_rkey function
  such that, in the case of FASTREG memory mode, the ib_post_send function with
  &cd->invalidate_wr parameter is not called during the first run of the 
function
- In FreeBSD 10.3, krping fr test is incorrectly implemented. Please consult
  the krping.c file provided with the FreeBSD 11.0 kernel for the required
  fixes. Needed changes include, but are not limited to:
  - The fr test requires the server side (unlike what is written in
    the comments).
	function: krping_doit
	function: krping_fr_test_server
	function: krping_run_server
  - The fr test require MR memory mode.
	function: krping_doit
  - Wrong device is queried on the client side when checking fastreg support.
	function: fastreg_supported
  - Note: the fr itself is designed to work endlessly, however lack of
    wait_event_interruptible_timeout function support causes the test to do
    nothing after the first iteration.
	function: krping_fr_test

-------------------
Changes since 0.1.2
-------------------
- Fix for kernel crash on FreeBSD 11.0 when setting ixl3 down.
- Fix for krping test with fastreg and server_inv options enabled.

--------------------
Changes since 0.0.24
--------------------
- Added patch for 1.7.12 to fix a number of bugs (These changes are available
  in the ixl driver, version 1.7.13 and later):
    o kernel crash on loading iw_ixl on XL710 cards.
    o kernel crash on loading iw_ixl with enable_iwarp tunable set to 0.
- MSIX vector utilization has been reworked.
- CPU affinity of the CEQ has been implemented.
- Added sysctl controls.

--------------------
Changes since 0.0.18
--------------------
- No patch for OFED is necessary for FreeBSD 11.0.

--------------------
Changes since 0.0.11
--------------------
- Support for FreeBSD 11.0 connection is added.
- No patch for OFED is necessary for FreeBSD 10.3.


================================================================================


Support
-------
For general information, go to the Intel support website at:
www.intel.com/support/

or the Intel Wired Networking project hosted by Sourceforge at:
http://sourceforge.net/projects/e1000
If an issue is identified with the released source code on a supported kernel 
with a supported adapter, email the specific information related to the issue 
to e1000-rdma@lists.sourceforge.net


================================================================================


License
-------

This software is available to you under a choice of one of two
licenses. You may choose to be licensed under the terms of the GNU
General Public License (GPL) Version 2, available from the file
COPYING in the main directory of this source tree, or the
OpenFabrics.org BSD license below:

  Redistribution and use in source and binary forms, with or
  without modification, are permitted provided that the following
  conditions are met:

  - Redistributions of source code must retain the above
    copyright notice, this list of conditions and the following
    disclaimer.

  - Redistributions in binary form must reproduce the above
    copyright notice, this list of conditions and the following
    disclaimer in the documentation and/or other materials
    provided with the distribution.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.


================================================================================


Trademarks
----------
Intel and Itanium are trademarks or registered trademarks of Intel Corporation 
or its subsidiaries in the United States and/or other countries.

* Other names and brands may be claimed as the property of others.


