Contents

  1. Summary
  2. Kernel Version
  3. Output Of oops
  4. Example program which reproduces oops
  5. Output of ver_linux
  6. Processor Info
  7. Module Info
  8. Loaded driver and hardware information
  9. PCI Info
  10. SCSI Info
  11. RAID Info
  12. Boot Messages
  13. Kernel Config

  1. Summary:

    Kernel oops during heavy I/O on Software RAID device with LVM and XFS.

    I'm using raid 10 mirrored stripes.

    I set the machine up over the previous weekend to run the stress tests shown below, and the only message I found in the logs was the oops below.

  2. Kernel Version:
    Linux calculon 2.6.10 #1 SMP Thu Dec 30 13:27:35 CST 2004 i686 Intel(R) Xeon(TM) CPU 2.80GHz GenuineIntel GNU/Linux
    
  3. Output Of oops:
    Dec 30 20:25:34 calculon Unable to handle kernel NULL pointer dereference at virtual address 00000004
    Dec 30 20:25:34 calculon printing eip:
    Dec 30 20:25:34 calculon c02adb94
    Dec 30 20:25:34 calculon *pde = 00000000
    Dec 30 20:25:34 calculon Oops: 0002 [#1]
    Dec 30 20:25:34 calculon SMP 
    Dec 30 20:25:34 calculon Modules linked in:
    Dec 30 20:25:34 calculon CPU:    2
    Dec 30 20:25:34 calculon EIP:    0060:[]    Not tainted VLI
    Dec 30 20:25:34 calculon EFLAGS: 00010286   (2.6.10) 
    Dec 30 20:25:34 calculon EIP is at xfs_ail_insert+0x84/0xd0
    Dec 30 20:25:34 calculon eax: 00000000   ebx: ffffffff   ecx: fffffc19   edx: ffffffff
    Dec 30 20:25:34 calculon esi: f436730c   edi: f703cc18   ebp: cc3e930c   esp: f7d55dd8
    Dec 30 20:25:34 calculon ds: 007b   es: 007b   ss: 0068
    Dec 30 20:25:34 calculon Process xfslogd/2 (pid: 235, threadinfo=f7d54000 task=f7cfd020)
    Dec 30 20:25:34 calculon Stack: 0001079c 00000160 00000000 00000000 cc3e930c f703cc18 f703cc00 f436730c
    Dec 30 20:25:34 calculon c02ad8c9 f703cc18 cc3e930c 00000092 cc3e930c 0001079c 00000160 00000000
    Dec 30 20:25:34 calculon cc3e930c 0001079c 00000160 c02ad447 f703cc00 cc3e930c 0001079c 00000160
    Dec 30 20:25:34 calculon Call Trace: 
    Dec 30 20:25:34 calculon [] xfs_trans_update_ail+0x59/0xc0
    Dec 30 20:25:34 calculon [] xfs_trans_chunk_committed+0x167/0x200
    Dec 30 20:25:34 calculon [] xfs_trans_committed+0x4a/0x120
    Dec 30 20:25:34 calculon [] xlog_state_do_callback+0x223/0x2d0
    Dec 30 20:25:34 calculon [] xlog_state_done_syncing+0x6d/0xa0
    Dec 30 20:25:34 calculon [] xlog_iodone+0x55/0xe0
    Dec 30 20:25:34 calculon [] pagebuf_iodone_work+0x4d/0x50
    Dec 30 20:25:34 calculon [] worker_thread+0x1b9/0x260
    Dec 30 20:25:34 calculon [] pagebuf_iodone_work+0x0/0x50
    Dec 30 20:25:34 calculon [] default_wake_function+0x0/0x20
    Dec 30 20:25:34 calculon [] default_wake_function+0x0/0x20
    Dec 30 20:25:34 calculon [] worker_thread+0x0/0x260
    Dec 30 20:25:34 calculon [] kthread+0xba/0xc0
    Dec 30 20:25:34 calculon [] kthread+0x0/0xc0
    Dec 30 20:25:34 calculon [] kernel_thread_helper+0x5/0x10
    Dec 30 20:25:34 calculon Code: b9 e7 03 00 00 31 db 89 c8 89 da 85 d2 7e 08 8b 76 04 eb b3 8d 76 00 7c 05 83 f8 00 77 f1 89 75 04 8b 06 89 45 00 89 2e 8b 45 00 <89> 68 04 83 c4 10 5b 5e 5f 5d c3 90 8b 5c 24 08 31 c0 31 d2 8b
    
  4. Example program which reproduces oops:

    I execute nfs_fsstress.sh as well as the script below at the same time:

    #!/bin/bash
    
    last=/dev/zero;
    while true; do
        for i in `seq 1 1 130000`; do
            num=`printf %06d $i`;
            filename="testfile-$num";
            echo "writing $filename...";
            dd if=$last of=$filename bs=1M count=1 &> /dev/null;
            last=$filename;
        done;
        rm -f testfile-??????;
        last=/dev/zero;
    done
    
  5. Output of ver_linux:
    Linux calculon 2.6.10 #1 SMP Thu Dec 30 13:27:35 CST 2004 i686 Intel(R) Xeon(TM) CPU 2.80GHz GenuineIntel GNU/Linux
     
    Gnu C                  3.3.3
    Gnu make               3.80
    binutils               2.14.90.0.8
    util-linux             2.12
    mount                  2.12
    module-init-tools      3.0
    e2fsprogs              1.35
    jfsutils               1.1.4
    reiserfsprogs          3.6.18
    reiser4progs           line
    xfsprogs               2.6.13
    nfs-utils              1.0.6
    Linux C Library        2.3.3
    Dynamic linker (ldd)   2.3.3
    Procps                 3.1.15
    Net-tools              1.60
    Kbd                    1.12
    Sh-utils               5.2.1
    Modules Loaded         
    
  6. Processor Info:
    processor       : 0
    vendor_id       : GenuineIntel
    cpu family      : 15
    model           : 2
    model name      : Intel(R) Xeon(TM) CPU 2.80GHz
    stepping        : 7
    cpu MHz         : 2799.924
    cache size      : 512 KB
    physical id     : 0
    siblings        : 2
    fdiv_bug        : no
    hlt_bug         : no
    f00f_bug        : no
    coma_bug        : no
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 2
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid
    bogomips        : 5521.40
    
    processor       : 1
    vendor_id       : GenuineIntel
    cpu family      : 15
    model           : 2
    model name      : Intel(R) Xeon(TM) CPU 2.80GHz
    stepping        : 7
    cpu MHz         : 2799.924
    cache size      : 512 KB
    physical id     : 0
    siblings        : 2
    fdiv_bug        : no
    hlt_bug         : no
    f00f_bug        : no
    coma_bug        : no
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 2
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid
    bogomips        : 5586.94
    
    processor       : 2
    vendor_id       : GenuineIntel
    cpu family      : 15
    model           : 2
    model name      : Intel(R) Xeon(TM) CPU 2.80GHz
    stepping        : 7
    cpu MHz         : 2799.924
    cache size      : 512 KB
    physical id     : 3
    siblings        : 2
    fdiv_bug        : no
    hlt_bug         : no
    f00f_bug        : no
    coma_bug        : no
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 2
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid
    bogomips        : 5586.94
    
    processor       : 3
    vendor_id       : GenuineIntel
    cpu family      : 15
    model           : 2
    model name      : Intel(R) Xeon(TM) CPU 2.80GHz
    stepping        : 7
    cpu MHz         : 2799.924
    cache size      : 512 KB
    physical id     : 3
    siblings        : 2
    fdiv_bug        : no
    hlt_bug         : no
    f00f_bug        : no
    coma_bug        : no
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 2
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid
    bogomips        : 5586.94
    
  7. Module Info:
    No modules
    
  8. Loaded driver and hardware information:

    /proc/ioports

    0000-001f : dma1
    0020-0021 : pic1
    0040-0043 : timer0
    0050-0053 : timer1
    0060-006f : keyboard
    0070-0077 : rtc
    0080-008f : dma page reg
    00a0-00a1 : pic2
    00c0-00df : dma2
    00f0-00ff : fpu
    01f0-01f7 : ide0
    02f8-02ff : serial
    03c0-03df : vga+
    03f6-03f6 : ide0
    03f8-03ff : serial
    0400-047f : 0000:00:1f.0
      0400-0403 : PM1a_EVT_BLK
      0404-0405 : PM1a_CNT_BLK
      0408-040b : PM_TMR
      0410-0415 : ACPI CPU throttle
      0420-0420 : PM2_CNT_BLK
      0428-042b : GPE0_BLK
      042c-042f : GPE1_BLK
    0500-053f : 0000:00:1f.0
    0540-055f : 0000:00:1f.3
    0cf8-0cff : PCI conf1
    8800-88ff : 0000:01:02.0
    8c00-8c3f : 0000:01:01.0
      8c00-8c3f : e100
    9000-bfff : PCI Bus #02
      9000-afff : PCI Bus #03
      b000-bfff : PCI Bus #04
        b000-b0ff : 0000:04:02.0
          b000-b0ff : aic79xx
        b400-b4ff : 0000:04:02.0
          b400-b4ff : aic79xx
        b800-b8ff : 0000:04:02.1
          b800-b8ff : aic79xx
        bc00-bcff : 0000:04:02.1
          bc00-bcff : aic79xx
    c000-dfff : PCI Bus #05
      c000-cfff : PCI Bus #06
        c400-c4ff : 0000:06:03.0
        c800-c8ff : 0000:06:03.1
      d000-dfff : PCI Bus #07
        d880-d8bf : 0000:07:01.0
          d880-d8bf : e1000
        dc00-dc3f : 0000:07:01.1
          dc00-dc3f : e1000
    ec00-ec1f : 0000:00:1d.0
      ec00-ec1f : uhci_hcd
    ffa0-ffaf : 0000:00:1f.1
      ffa0-ffa7 : ide0
      ffa8-ffaf : ide1
    

    /proc/iomem

    00000000-0009fbff : System RAM
    0009fc00-0009ffff : reserved
    000a0000-000bffff : Video RAM area
    000c0000-000c7fff : Video ROM
    000c8000-000c97ff : Adapter ROM
    000c9800-000d27ff : Adapter ROM
    000f0000-000fffff : System ROM
    00100000-f7feffff : System RAM
      00100000-0048923b : Kernel code
      0048923c-005a7a7f : Kernel data
    f7ff0000-f7ffefff : ACPI Tables
    f7fff000-f7ffffff : ACPI Non-volatile Storage
    f8000000-f80003ff : 0000:00:1f.1
    fc000000-fcffffff : 0000:01:02.0
    fdea0000-fdebffff : 0000:01:01.0
      fdea0000-fdebffff : e100
    fdefe000-fdefefff : 0000:01:01.0
      fdefe000-fdefefff : e100
    fdeff000-fdefffff : 0000:01:02.0
    fdf00000-fe3fffff : PCI Bus #02
      fdf00000-fdffffff : PCI Bus #03
      fe000000-fe2fffff : PCI Bus #04
        fe2fc000-fe2fdfff : 0000:04:02.0
        fe2fe000-fe2fffff : 0000:04:02.1
      fe3fe000-fe3fefff : 0000:02:1c.0
      fe3ff000-fe3fffff : 0000:02:1e.0
    fe400000-feafffff : PCI Bus #05
      fe400000-fe8fffff : PCI Bus #06
        fe8c0000-fe8cffff : 0000:06:03.0
        fe8d0000-fe8dffff : 0000:06:03.0
        fe8e0000-fe8effff : 0000:06:03.1
        fe8f0000-fe8fffff : 0000:06:03.1
      fe900000-fe9fffff : PCI Bus #07
        fe9c0000-fe9dffff : 0000:07:01.0
          fe9c0000-fe9dffff : e1000
        fe9e0000-fe9fffff : 0000:07:01.1
          fe9e0000-fe9fffff : e1000
      feafe000-feafefff : 0000:05:1c.0
      feaff000-feafffff : 0000:05:1e.0
    fec00000-fecfffff : reserved
    fee00000-fee00fff : reserved
    ff400000-ff6fffff : PCI Bus #02
      ff400000-ff4fffff : PCI Bus #03
      ff500000-ff5fffff : PCI Bus #04
    ff700000-ff9fffff : PCI Bus #05
      ff700000-ff7fffff : PCI Bus #06
      ff800000-ff8fffff : PCI Bus #07
    ffb80000-ffffffff : reserved
    
  9. PCI Info:

    lspci -vvv

    0000:00:00.0 Host bridge: Intel Corp. E7501 Memory Controller Hub (rev 01)
    	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
    	Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- SERR- TAbort- SERR- Reset- FastB2B-
    
    0000:00:03.0 PCI bridge: Intel Corp. E7500/E7501 Hub Interface C PCI-to-PCI Bridge (rev 01) (prog-if 00 [Normal decode])
    	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
    	Status: Cap- 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- SERR- Reset- FastB2B-
    
    0000:00:1d.0 USB Controller: Intel Corp. 82801CA/CAM USB (Hub #1) (rev 02) (prog-if 00 [UHCI])
    	Subsystem: Intel Corp.: Unknown device 2480
    	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
    	Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- SERR- TAbort- SERR- Reset- FastB2B-
    
    0000:00:1f.0 ISA bridge: Intel Corp. 82801CA LPC Interface Controller (rev 02)
    	Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
    	Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- SERR- TAbort- SERR- 
    	Region 1: I/O ports at 
    	Region 2: I/O ports at 
    	Region 3: I/O ports at 
    	Region 4: I/O ports at ffa0 [size=16]
    	Region 5: Memory at f8000000 (32-bit, non-prefetchable) [size=1K]
    
    0000:00:1f.3 SMBus: Intel Corp. 82801CA/CAM SMBus Controller (rev 02)
    	Subsystem: Intel Corp.: Unknown device 2480
    	Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
    	Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- SERR- TAbort- SERR- TAbort- SERR- TAbort- SERR- TAbort- SERR- Reset- FastB2B-
    	Capabilities: [50] PCI-X bridge device.
    		Secondary Status: 64bit+, 133MHz+, SCD-, USC-, SCO-, SRD- Freq=2
    		Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, SCO-, SRD-
    		: Upstream: Capacity=0, Commitment Limit=0
    		: Downstream: Capacity=0, Commitment Limit=0
    
    0000:02:1e.0 PIC: Intel Corp. 82870P2 P64H2 I/OxAPIC (rev 04) (prog-if 20 [IO(X)-APIC])
    	Subsystem: Intel Corp. 82870P2 P64H2 I/OxAPIC
    	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
    	Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- TAbort- SERR- Reset- FastB2B-
    	Capabilities: [50] PCI-X bridge device.
    		Secondary Status: 64bit+, 133MHz+, SCD-, USC-, SCO-, SRD- Freq=3
    		Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, SCO-, SRD-
    		: Upstream: Capacity=0, Commitment Limit=0
    		: Downstream: Capacity=0, Commitment Limit=0
    
    0000:04:02.0 SCSI storage controller: Adaptec AIC-7902 U320 (rev 03)
    	Subsystem: Adaptec: Unknown device ffff
    	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
    	Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- SERR- TAbort- SERR- TAbort- SERR- TAbort- SERR- Reset- FastB2B-
    	Capabilities: [50] PCI-X bridge device.
    		Secondary Status: 64bit+, 133MHz+, SCD-, USC-, SCO-, SRD- Freq=3
    		Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, SCO-, SRD-
    		: Upstream: Capacity=0, Commitment Limit=0
    		: Downstream: Capacity=0, Commitment Limit=0
    
    0000:05:1e.0 PIC: Intel Corp. 82870P2 P64H2 I/OxAPIC (rev 04) (prog-if 20 [IO(X)-APIC])
    	Subsystem: Intel Corp. 82870P2 P64H2 I/OxAPIC
    	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
    	Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- TAbort- SERR- Reset- FastB2B-
    	Capabilities: [50] PCI-X bridge device.
    		Secondary Status: 64bit+, 133MHz+, SCD-, USC-, SCO-, SRD- Freq=3
    		Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, SCO-, SRD-
    		: Upstream: Capacity=0, Commitment Limit=0
    		: Downstream: Capacity=0, Commitment Limit=0
    
    0000:06:03.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07)
    	Subsystem: LSI Logic / Symbios Logic: Unknown device 1040
    	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
    	Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- SERR- TAbort- SERR- TAbort- SERR- TAbort- SERR- 
    	
  10. SCSI Info:

    /proc/scsi/scsi

    Attached devices:
    Host: scsi0 Channel: 00 Id: 00 Lun: 00
      Vendor: SEAGATE  Model: ST318406LC       Rev: 010A
      Type:   Direct-Access                    ANSI SCSI revision: 03
    Host: scsi0 Channel: 00 Id: 01 Lun: 00
      Vendor: SEAGATE  Model: ST318406LC       Rev: 010A
      Type:   Direct-Access                    ANSI SCSI revision: 03
    Host: scsi1 Channel: 00 Id: 01 Lun: 00
      Vendor: SEAGATE  Model: ST373307LC       Rev: 0007
      Type:   Direct-Access                    ANSI SCSI revision: 03
    Host: scsi1 Channel: 00 Id: 02 Lun: 00
      Vendor: SEAGATE  Model: ST373307LC       Rev: 0007
      Type:   Direct-Access                    ANSI SCSI revision: 03
    Host: scsi1 Channel: 00 Id: 04 Lun: 00
      Vendor: SEAGATE  Model: ST373307LC       Rev: 0007
      Type:   Direct-Access                    ANSI SCSI revision: 03
    Host: scsi1 Channel: 00 Id: 05 Lun: 00
      Vendor: JMR ELEC Model: FORTRA SERIES.   Rev: 1.21
      Type:   Processor                        ANSI SCSI revision: 02
    Host: scsi2 Channel: 00 Id: 01 Lun: 00
      Vendor: SEAGATE  Model: ST373307LC       Rev: 0006
      Type:   Direct-Access                    ANSI SCSI revision: 03
    Host: scsi2 Channel: 00 Id: 02 Lun: 00
      Vendor: SEAGATE  Model: ST373307LC       Rev: 0007
      Type:   Direct-Access                    ANSI SCSI revision: 03
    Host: scsi2 Channel: 00 Id: 05 Lun: 00
      Vendor: JMR ELEC Model: FORTRA SERIES.   Rev: 1.21
      Type:   Processor                        ANSI SCSI revision: 02
    
  11. RAID Info:

    /proc/mdstat

    Personalities : [linear] [raid0] [raid1] [raid5] [multipath] [raid6] 
    md4 : active raid1 md2[0] md3[1]
          143363776 blocks [2/2] [UU]
          
    md1 : active raid1 sdb3[1] sda3[0]
          12924224 blocks [2/2] [UU]
          
    md2 : active raid0 sdd1[1] sdc1[0]
          143363840 blocks 32k chunks
          
    md3 : active raid0 sdg1[0] sdf1[1]
          143363840 blocks 32k chunks
          
    md0 : active raid1 sdb1[1] sda1[0]
          995904 blocks [2/2] [UU]
          
    unused devices: 
    

    mdadm --detail /dev/md0

    /dev/md0:
            Version : 00.90.01
      Creation Time : Thu Nov  4 10:21:57 2004
         Raid Level : raid1
         Array Size : 995904 (972.56 MiB 1019.81 MB)
        Device Size : 995904 (972.56 MiB 1019.81 MB)
       Raid Devices : 2
      Total Devices : 2
    Preferred Minor : 0
        Persistence : Superblock is persistent
    
        Update Time : Mon Nov 22 10:07:30 2004
              State : clean
     Active Devices : 2
    Working Devices : 2
     Failed Devices : 0
      Spare Devices : 0
    
    
        Number   Major   Minor   RaidDevice State
           0       8        1        0      active sync   /dev/sda1
           1       8       17        1      active sync   /dev/sdb1
               UUID : b4fc9db3:8e3dd741:fa6e8488:94027aa9
             Events : 0.6632
    

    mdadm --detail /dev/md1

    /dev/md1:
            Version : 00.90.01
      Creation Time : Thu Nov 11 14:29:28 2004
         Raid Level : raid1
         Array Size : 12924224 (12.33 GiB 13.23 GB)
        Device Size : 12924224 (12.33 GiB 13.23 GB)
       Raid Devices : 2
      Total Devices : 2
    Preferred Minor : 1
        Persistence : Superblock is persistent
    
        Update Time : Mon Nov 22 10:32:55 2004
              State : clean
     Active Devices : 2
    Working Devices : 2
     Failed Devices : 0
      Spare Devices : 0
    
    
        Number   Major   Minor   RaidDevice State
           0       8        3        0      active sync   /dev/sda3
           1       8       19        1      active sync   /dev/sdb3
               UUID : 451d8a4b:00a532e9:420e9472:7ec250b7
             Events : 0.3176
    

    mdadm --detail /dev/md2

    /dev/md2:
            Version : 00.90.01
      Creation Time : Wed Nov 10 13:49:41 2004
         Raid Level : raid0
         Array Size : 143363840 (136.72 GiB 146.80 GB)
       Raid Devices : 2
      Total Devices : 2
    Preferred Minor : 2
        Persistence : Superblock is persistent
    
        Update Time : Fri Nov 19 10:56:54 2004
              State : clean
     Active Devices : 2
    Working Devices : 2
     Failed Devices : 0
      Spare Devices : 0
    
         Chunk Size : 32K
    
        Number   Major   Minor   RaidDevice State
           0       8       33        0      active sync   /dev/sdc1
           1       8       49        1      active sync   /dev/sdd1
               UUID : 490e2896:1454fd6a:ab6ca2ee:c604587d
             Events : 0.8
    

    mdadm --detail /dev/md3

    /dev/md3:
            Version : 00.90.01
      Creation Time : Thu Nov 11 09:05:51 2004
         Raid Level : raid0
         Array Size : 143363840 (136.72 GiB 146.80 GB)
       Raid Devices : 2
      Total Devices : 2
    Preferred Minor : 3
        Persistence : Superblock is persistent
    
        Update Time : Fri Nov 19 10:56:54 2004
              State : clean
     Active Devices : 2
    Working Devices : 2
     Failed Devices : 0
      Spare Devices : 0
    
         Chunk Size : 32K
    
        Number   Major   Minor   RaidDevice State
           0       8       97        0      active sync   /dev/sdg1
           1       8       81        1      active sync   /dev/sdf1
               UUID : cc83e519:5dc0385a:83a55403:7b87a920
             Events : 0.9
    

    mdadm --detail /dev/md4

    /dev/md4:
            Version : 00.90.01
      Creation Time : Fri Nov 19 11:00:14 2004
         Raid Level : raid1
         Array Size : 143363776 (136.72 GiB 146.80 GB)
        Device Size : 143363776 (136.72 GiB 146.80 GB)
       Raid Devices : 2
      Total Devices : 2
    Preferred Minor : 4
        Persistence : Superblock is persistent
    
        Update Time : Mon Nov 22 10:32:50 2004
              State : clean
     Active Devices : 2
    Working Devices : 2
     Failed Devices : 0
      Spare Devices : 0
    
    
        Number   Major   Minor   RaidDevice State
           0       9        2        0      active sync   /dev/md2
           1       9        3        1      active sync   /dev/md3
               UUID : 6aa6fd6c:95afa59a:157febbf:f8383bee
             Events : 0.178
    
  12. Boot Messages:
    Linux version 2.6.10 (root@calculon) (gcc version 3.3.3 20040412 (Gentoo Linux 3.3.3-r6, ssp-3.3.2-2, pie-8.7.6)) #1 SMP Wed Dec 29 11:39:38 CST 2004
    BIOS-provided physical RAM map:
     BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
     BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
     BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
     BIOS-e820: 0000000000100000 - 00000000f7ff0000 (usable)
     BIOS-e820: 00000000f7ff0000 - 00000000f7fff000 (ACPI data)
     BIOS-e820: 00000000f7fff000 - 00000000f8000000 (ACPI NVS)
     BIOS-e820: 00000000fec00000 - 00000000fed00000 (reserved)
     BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
     BIOS-e820: 00000000ffb80000 - 0000000100000000 (reserved)
     BIOS-e820: 0000000100000000 - 0000000108000000 (usable)
    Warning only 4GB will be used.
    Use a PAE enabled kernel.
    3200MB HIGHMEM available.
    896MB LOWMEM available.
    found SMP MP-table at 000ff780
    DMI 2.3 present.
    ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
    Processor #0 15:2 APIC version 20
    ACPI: LAPIC (acpi_id[0x02] lapic_id[0x06] enabled)
    Processor #6 15:2 APIC version 20
    ACPI: LAPIC (acpi_id[0x03] lapic_id[0x01] enabled)
    Processor #1 15:2 APIC version 20
    ACPI: LAPIC (acpi_id[0x04] lapic_id[0x07] enabled)
    Processor #7 15:2 APIC version 20
    ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
    IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
    ACPI: IOAPIC (id[0x09] address[0xfec80000] gsi_base[24])
    IOAPIC[1]: apic_id 9, version 32, address 0xfec80000, GSI 24-47
    ACPI: IOAPIC (id[0x0a] address[0xfec80400] gsi_base[48])
    IOAPIC[2]: apic_id 10, version 32, address 0xfec80400, GSI 48-71
    ACPI: IOAPIC (id[0x0b] address[0xfec81000] gsi_base[72])
    IOAPIC[3]: apic_id 11, version 32, address 0xfec81000, GSI 72-95
    ACPI: IOAPIC (id[0x0c] address[0xfec81400] gsi_base[96])
    IOAPIC[4]: apic_id 12, version 32, address 0xfec81400, GSI 96-119
    ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
    ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
    Enabling APIC mode:  Flat.  Using 5 I/O APICs
    Using ACPI (MADT) for SMP configuration information
    Built 1 zonelists
    Kernel command line: auto BOOT_IMAGE=2.6.10 ro root=900 gentoo=nodevfs console=ttyS0,38400
    Initializing CPU#0
    CPU 0 irqstacks, hard=c05ec000 soft=c05e8000
    PID hash table entries: 4096 (order: 12, 65536 bytes)
    Detected 2799.992 MHz processor.
    Using tsc for high-res timesource
    Console: colour VGA+ 80x25
    Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
    Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
    Memory: 4023024k/4194304k available (3620k kernel code, 39228k reserved, 1138k data, 240k init, 3145664k highmem)
    Checking if this processor honours the WP bit even in supervisor mode... Ok.
    Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
    CPU: Trace cache: 12K uops, L1 D cache: 8K
    CPU: L2 cache: 512K
    CPU: Physical Processor ID: 0
    Intel machine check architecture supported.
    Intel machine check reporting enabled on CPU#0.
    CPU0: Intel P4/Xeon Extended MCE MSRs (12) available
    CPU0: Thermal monitoring enabled
    Enabling fast FPU save and restore... done.
    Enabling unmasked SIMD FPU exception support... done.
    Checking 'hlt' instruction... OK.
    CPU0: Intel(R) Xeon(TM) CPU 2.80GHz stepping 07
    per-CPU timeslice cutoff: 1463.17 usecs.
    task migration cache decay timeout: 2 msecs.
    Booting processor 1/1 eip 3000
    CPU 1 irqstacks, hard=c05ed000 soft=c05e9000
    Initializing CPU#1
    CPU: Trace cache: 12K uops, L1 D cache: 8K
    CPU: L2 cache: 512K
    CPU: Physical Processor ID: 0
    Intel machine check architecture supported.
    Intel machine check reporting enabled on CPU#1.
    CPU1: Intel P4/Xeon Extended MCE MSRs (12) available
    CPU1: Thermal monitoring enabled
    CPU1: Intel(R) Xeon(TM) CPU 2.80GHz stepping 07
    Booting processor 2/6 eip 3000
    CPU 2 irqstacks, hard=c05ee000 soft=c05ea000
    Initializing CPU#2
    CPU: Trace cache: 12K uops, L1 D cache: 8K
    CPU: L2 cache: 512K
    CPU: Physical Processor ID: 3
    Intel machine check architecture supported.
    Intel machine check reporting enabled on CPU#2.
    CPU2: Intel P4/Xeon Extended MCE MSRs (12) available
    CPU2: Thermal monitoring enabled
    CPU2: Intel(R) Xeon(TM) CPU 2.80GHz stepping 07
    Booting processor 3/7 eip 3000
    CPU 3 irqstacks, hard=c05ef000 soft=c05eb000
    Initializing CPU#3
    CPU: Trace cache: 12K uops, L1 D cache: 8K
    CPU: L2 cache: 512K
    CPU: Physical Processor ID: 3
    Intel machine check architecture supported.
    Intel machine check reporting enabled on CPU#3.
    CPU3: Intel P4/Xeon Extended MCE MSRs (12) available
    CPU3: Thermal monitoring enabled
    CPU3: Intel(R) Xeon(TM) CPU 2.80GHz stepping 07
    Total of 4 processors activated (22282.24 BogoMIPS).
    ENABLING IO-APIC IRQs
    ..TIMER: vector=0x31 pin1=2 pin2=-1
    checking TSC synchronization across 4 CPUs: passed.
    Brought up 4 CPUs
    NET: Registered protocol family 16
    PCI: PCI BIOS revision 2.10 entry at 0xf0031, last bus=7
    PCI: Using configuration type 1
    mtrr: v2.0 (20020519)
    ACPI: Subsystem revision 20041105
    ACPI: Interpreter enabled
    ACPI: Using IOAPIC for interrupt routing
    ACPI: PCI Root Bridge [PCI0] (00:00)
    PCI: Probing PCI hardware (bus 00)
    PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.1
    PCI: Transparent bridge - 0000:00:1e.0
    ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 *9 10 11 12 14 15)
    ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 *10 11 12 14 15)
    ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 9 10 *11 12 14 15)
    ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 *5 6 7 9 10 11 12 14 15)
    ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled.
    ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled.
    ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled.
    ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled.
    Linux Plug and Play Support v0.97 (c) Adam Belay
    SCSI subsystem initialized
    usbcore: registered new driver usbfs
    usbcore: registered new driver hub
    PCI: Using ACPI for IRQ routing
    ** PCI interrupts are no longer routed automatically.  If this
    ** causes a device to stop working, it is probably because the
    ** driver failed to call pci_enable_device().  As a temporary
    ** workaround, the "pci=routeirq" argument restores the old
    ** behavior.  If this argument makes the device work again,
    ** please email the output of "lspci" to bjorn.helgaas@hp.com
    ** so I can fix the driver.
    Machine check exception polling timer started.
    audit: initializing netlink socket (disabled)
    audit(1104417287.909:0): initialized
    highmem bounce pool size: 64 pages
    Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
    SGI XFS with large block numbers, no debug enabled
    Initializing Cryptographic API
    ACPI: Power Button (FF) [PWRF]
    ACPI: Processor [CPU1] (supports 8 throttling states)
    Real Time Clock Driver v1.12
    serio: i8042 AUX port at 0x60,0x64 irq 12
    serio: i8042 KBD port at 0x60,0x64 irq 1
    Serial: 8250/16550 driver $Revision: 1.90 $ 8 ports, IRQ sharing disabled
    ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
    ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
    io scheduler noop registered
    io scheduler anticipatory registered
    io scheduler deadline registered
    io scheduler cfq registered
    elevator: using anticipatory as default io scheduler
    FDC 0 is a post-1991 82077
    loop: loaded (max 8 devices)
    Intel(R) PRO/1000 Network Driver - version 5.5.4-k2
    Copyright (c) 1999-2004 Intel Corporation.
    ACPI: PCI interrupt 0000:07:01.0[A] -> GSI 48 (level, low) -> IRQ 48
    e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
    ACPI: PCI interrupt 0000:07:01.1[B] -> GSI 49 (level, low) -> IRQ 49
    e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
    e100: Intel(R) PRO/100 Network Driver, 3.2.3-k2-NAPI
    e100: Copyright(c) 1999-2004 Intel Corporation
    ACPI: PCI interrupt 0000:01:01.0[A] -> GSI 19 (level, low) -> IRQ 19
    e100: eth2: e100_probe: addr 0xfdefe000, irq 19, MAC addr 00:E0:81:25:CF:17
    Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
    ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
    ICH3: IDE controller at PCI slot 0000:00:1f.1
    PCI: Enabling device 0000:00:1f.1 (0005 -> 0007)
    ACPI: PCI interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 18
    ICH3: chipset revision 2
    ICH3: not 100% native mode: will probe irqs later
        ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:DMA, hdb:pio
        ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:pio, hdd:pio
    hda: SR244W, ATAPI CD/DVD-ROM drive
    ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
    hda: ATAPI 24X CD-ROM drive, 128kB Cache, UDMA(33)
    Uniform CD-ROM driver Revision: 3.20
    ACPI: PCI interrupt 0000:04:02.0[A] -> GSI 98 (level, low) -> IRQ 98
    ACPI: PCI interrupt 0000:04:02.1[B] -> GSI 99 (level, low) -> IRQ 99
    scsi0 : Adaptec AIC79XX PCI-X SCSI HBA DRIVER, Rev 1.3.11
            
            aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 67-100Mhz, 512 SCBs
    
    (scsi0:A:0): 160.000MB/s transfers (80.000MHz DT, 16bit)
    (scsi0:A:1): 160.000MB/s transfers (80.000MHz DT, 16bit)
      Vendor: SEAGATE   Model: ST318406LC        Rev: 010A
      Type:   Direct-Access                      ANSI SCSI revision: 03
    scsi0:A:0:0: Tagged Queuing enabled.  Depth 32
      Vendor: SEAGATE   Model: ST318406LC        Rev: 010A
      Type:   Direct-Access                      ANSI SCSI revision: 03
    scsi0:A:1:0: Tagged Queuing enabled.  Depth 32
    scsi1 : Adaptec AIC79XX PCI-X SCSI HBA DRIVER, Rev 1.3.11
            
            aic7902: Ultra320 Wide Channel B, SCSI Id=7, PCI-X 67-100Mhz, 512 SCBs
    
    (scsi1:A:0): 160.000MB/s transfers (80.000MHz DT|IU|QAS, 16bit)
    (scsi1:A:1): 160.000MB/s transfers (80.000MHz DT|IU|QAS, 16bit)
    (scsi1:A:4): 160.000MB/s transfers (80.000MHz DT|IU|QAS, 16bit)
      Vendor: SEAGATE   Model: ST373307LC        Rev: 0007
      Type:   Direct-Access                      ANSI SCSI revision: 03
    scsi1:A:0:0: Tagged Queuing enabled.  Depth 32
      Vendor: SEAGATE   Model: ST373307LC        Rev: 0007
      Type:   Direct-Access                      ANSI SCSI revision: 03
    scsi1:A:1:0: Tagged Queuing enabled.  Depth 32
      Vendor: SEAGATE   Model: ST373307LC        Rev: 0007
      Type:   Direct-Access                      ANSI SCSI revision: 03
    scsi1:A:4:0: Tagged Queuing enabled.  Depth 32
      Vendor: JMR ELEC  Model: FORTRA SERIES.    Rev: 1.21
      Type:   Processor                          ANSI SCSI revision: 02
    SCSI device sda: 35843670 512-byte hdwr sectors (18352 MB)
    SCSI device sda: drive cache: write back
    SCSI device sda: 35843670 512-byte hdwr sectors (18352 MB)
    SCSI device sda: drive cache: write back
     sda: sda1 sda2 sda3
    Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
    SCSI device sdb: 35843670 512-byte hdwr sectors (18352 MB)
    SCSI device sdb: drive cache: write back
    SCSI device sdb: 35843670 512-byte hdwr sectors (18352 MB)
    SCSI device sdb: drive cache: write back
     sdb: sdb1 sdb2 sdb3
    Attached scsi disk sdb at scsi0, channel 0, id 1, lun 0
    SCSI device sdc: 143374744 512-byte hdwr sectors (73408 MB)
    SCSI device sdc: drive cache: write back
    SCSI device sdc: 143374744 512-byte hdwr sectors (73408 MB)
    SCSI device sdc: drive cache: write back
     sdc: sdc1
    Attached scsi disk sdc at scsi1, channel 0, id 0, lun 0
    SCSI device sdd: 143374744 512-byte hdwr sectors (73408 MB)
    SCSI device sdd: drive cache: write back
    SCSI device sdd: 143374744 512-byte hdwr sectors (73408 MB)
    SCSI device sdd: drive cache: write back
     sdd: sdd1
    Attached scsi disk sdd at scsi1, channel 0, id 1, lun 0
    SCSI device sde: 143374744 512-byte hdwr sectors (73408 MB)
    SCSI device sde: drive cache: write back
    SCSI device sde: 143374744 512-byte hdwr sectors (73408 MB)
    SCSI device sde: drive cache: write back
     sde: sde1
    Attached scsi disk sde at scsi1, channel 0, id 4, lun 0
    Attached scsi generic sg0 at scsi0, channel 0, id 0, lun 0,  type 0
    Attached scsi generic sg1 at scsi0, channel 0, id 1, lun 0,  type 0
    Attached scsi generic sg2 at scsi1, channel 0, id 0, lun 0,  type 0
    Attached scsi generic sg3 at scsi1, channel 0, id 1, lun 0,  type 0
    Attached scsi generic sg4 at scsi1, channel 0, id 4, lun 0,  type 0
    Attached scsi generic sg5 at scsi1, channel 0, id 5, lun 0,  type 3
    Fusion MPT base driver 3.01.18
    Copyright (c) 1999-2004 LSI Logic Corporation
    ACPI: PCI interrupt 0000:06:03.0[A] -> GSI 24 (level, low) -> IRQ 24
    mptbase: Initiating ioc0 bringup
    ioc0: 53C1030: Capabilities={Initiator}
    ACPI: PCI interrupt 0000:06:03.1[B] -> GSI 25 (level, low) -> IRQ 25
    mptbase: Initiating ioc1 bringup
    ioc1: 53C1030: Capabilities={Initiator}
    Fusion MPT SCSI Host driver 3.01.18
    scsi2 : ioc0: LSI53C1030, FwRev=01013d00h, Ports=1, MaxQ=255, IRQ=24
    mptscsih: ioc0: >> Attempting task abort! (sc=f7d73200)
    mptscsih: ioc0: >> Attempting target reset! (sc=f7d73200)
    mptscsih: ioc0: >> Attempting bus reset! (sc=f7d73200)
    mptbase: ioc0: IOCStatus(0x0048): SCSI Task Terminated
      Vendor: SEAGATE   Model: ST373307LC        Rev: 0007
      Type:   Direct-Access                      ANSI SCSI revision: 03
    SCSI device sdf: 143374744 512-byte hdwr sectors (73408 MB)
    SCSI device sdf: drive cache: write back
    SCSI device sdf: 143374744 512-byte hdwr sectors (73408 MB)
    SCSI device sdf: drive cache: write back
     sdf: sdf1
    Attached scsi disk sdf at scsi2, channel 0, id 0, lun 0
    Attached scsi generic sg6 at scsi2, channel 0, id 0, lun 0,  type 0
      Vendor: SEAGATE   Model: ST373307LC        Rev: 0006
      Type:   Direct-Access                      ANSI SCSI revision: 03
    SCSI device sdg: 143374744 512-byte hdwr sectors (73408 MB)
    SCSI device sdg: drive cache: write back
    SCSI device sdg: 143374744 512-byte hdwr sectors (73408 MB)
    SCSI device sdg: drive cache: write back
     sdg: sdg1
    Attached scsi disk sdg at scsi2, channel 0, id 1, lun 0
    Attached scsi generic sg7 at scsi2, channel 0, id 1, lun 0,  type 0
      Vendor: JMR ELEC  Model: FORTRA SERIES.    Rev: 1.21
      Type:   Processor                          ANSI SCSI revision: 02
    Attached scsi generic sg8 at scsi2, channel 0, id 5, lun 0,  type 3
    scsi3 : ioc1: LSI53C1030, FwRev=01013d00h, Ports=1, MaxQ=255, IRQ=25
    USB Universal Host Controller Interface driver v2.2
    ACPI: PCI interrupt 0000:00:1d.0[A] -> GSI 16 (level, low) -> IRQ 16
    uhci_hcd 0000:00:1d.0: Intel Corp. 82801CA/CAM USB (Hub #1)
    uhci_hcd 0000:00:1d.0: irq 16, io base 0xec00
    uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 1
    hub 1-0:1.0: USB hub found
    hub 1-0:1.0: 2 ports detected
    usbcore: registered new driver usblp
    drivers/usb/class/usblp.c: v0.13: USB Printer Device Class driver
    Initializing USB Mass Storage driver...
    usbcore: registered new driver usb-storage
    USB Mass Storage support registered.
    usbcore: registered new driver hiddev
    usbcore: registered new driver usbhid
    drivers/usb/input/hid-core.c: v2.0:USB HID core driver
    mice: PS/2 mouse device common for all mice
    input: AT Translated Set 2 keyboard on isa0060/serio0
    md: linear personality registered as nr 1
    md: raid0 personality registered as nr 2
    md: raid1 personality registered as nr 3
    md: raid10 personality registered as nr 9
    md: raid5 personality registered as nr 4
    raid5: automatically using best checksumming function: pIII_sse
       pIII_sse  :  3612.000 MB/sec
    raid5: using function: pIII_sse (3612.000 MB/sec)
    raid6: int32x1    710 MB/s
    raid6: int32x2    953 MB/s
    raid6: int32x4    757 MB/s
    raid6: int32x8    574 MB/s
    raid6: mmxx1     2261 MB/s
    raid6: mmxx2     2871 MB/s
    raid6: sse1x1    1488 MB/s
    raid6: sse1x2    2046 MB/s
    raid6: sse2x1    2429 MB/s
    raid6: sse2x2    3039 MB/s
    raid6: using algorithm sse2x2 (3039 MB/s)
    md: raid6 personality registered as nr 8
    md: multipath personality registered as nr 7
    md: md driver 0.90.1 MAX_MD_DEVS=256, MD_SB_DISKS=27
    device-mapper: 4.3.0-ioctl (2004-09-30) initialised: dm-devel@redhat.com
    NET: Registered protocol family 2
    IP: routing cache hash table of 32768 buckets, 256Kbytes
    TCP: Hash tables configured (established 524288 bind 65536)
    ip_conntrack version 2.1 (8192 buckets, 65536 max) - 304 bytes per conntrack
    ip_tables: (C) 2000-2002 Netfilter core team
    ipt_recent v0.3.1: Stephen Frost .  http://snowman.net/projects/ipt_recent/
    arp_tables: (C) 2002 David S. Miller
    NET: Registered protocol family 1
    NET: Registered protocol family 17
    Starting balanced_irq
    ACPI wakeup devices: 
    PS2K PS2M SMBS AUDI MODM USB0 USB1 USB2 P0P1 
    ACPI: (supports S0 S1 S5)
    md: Autodetecting RAID arrays.
    md: autorun ...
    md: considering sdg1 ...
    md:  adding sdg1 ...
    md:  adding sdf1 ...
    md: sde1 has different UUID to sdg1
    md: sdd1 has different UUID to sdg1
    md: sdc1 has different UUID to sdg1
    md: sdb3 has different UUID to sdg1
    md: sdb1 has different UUID to sdg1
    md: sda3 has different UUID to sdg1
    md: sda1 has different UUID to sdg1
    md: created md3
    md: bind
    md: bind
    md: running: 
    md3: setting max_sectors to 64, segment boundary to 16383
    raid0: looking at sdg1
    raid0:   comparing sdg1(71681920) with sdg1(71681920)
    raid0:   END
    raid0:   ==> UNIQUE
    raid0: 1 zones
    raid0: looking at sdf1
    raid0:   comparing sdf1(71681920) with sdg1(71681920)
    raid0:   EQUAL
    raid0: FINAL 1 zones
    raid0: done.
    raid0 : md_size is 143363840 blocks.
    raid0 : conf->hash_spacing is 143363840 blocks.
    raid0 : nb_zone is 1.
    raid0 : Allocating 4 bytes for hash.
    md: considering sde1 ...
    md:  adding sde1 ...
    md: sdd1 has different UUID to sde1
    md: sdc1 has different UUID to sde1
    md: sdb3 has different UUID to sde1
    md: sdb1 has different UUID to sde1
    md: sda3 has different UUID to sde1
    md: sda1 has different UUID to sde1
    md: created md2
    md: bind
    md: running: 
    raid5: device sde1 operational as raid disk 0
    raid5: not enough operational devices for md2 (5/6 failed)
    RAID5 conf printout:
     --- rd:6 wd:1 fd:5
     disk 0, o:1, dev:sde1
    raid5: failed to run raid set md2
    md: pers->run() failed ...
    md: do_md_run() returned -22
    md: md2 stopped.
    md: unbind
    md: export_rdev(sde1)
    md: considering sdd1 ...
    md:  adding sdd1 ...
    md:  adding sdc1 ...
    md: sdb3 has different UUID to sdd1
    md: sdb1 has different UUID to sdd1
    md: sda3 has different UUID to sdd1
    md: sda1 has different UUID to sdd1
    md: created md2
    md: bind
    md: bind
    md: running: 
    md2: setting max_sectors to 64, segment boundary to 16383
    raid0: looking at sdd1
    raid0:   comparing sdd1(71681920) with sdd1(71681920)
    raid0:   END
    raid0:   ==> UNIQUE
    raid0: 1 zones
    raid0: looking at sdc1
    raid0:   comparing sdc1(71681920) with sdd1(71681920)
    raid0:   EQUAL
    raid0: FINAL 1 zones
    raid0: done.
    raid0 : md_size is 143363840 blocks.
    raid0 : conf->hash_spacing is 143363840 blocks.
    raid0 : nb_zone is 1.
    raid0 : Allocating 4 bytes for hash.
    md: considering sdb3 ...
    md:  adding sdb3 ...
    md: sdb1 has different UUID to sdb3
    md:  adding sda3 ...
    md: sda1 has different UUID to sdb3
    md: created md1
    md: bind
    md: bind
    md: running: 
    raid1: raid set md1 active with 2 out of 2 mirrors
    md: considering sdb1 ...
    md:  adding sdb1 ...
    md:  adding sda1 ...
    md: created md0
    md: bind
    md: bind
    md: running: 
    raid1: raid set md0 active with 2 out of 2 mirrors
    md: ... autorun DONE.
    ReiserFS: md0: found reiserfs format "3.6" with standard journal
    ReiserFS: md0: using ordered data mode
    ReiserFS: md0: journal params: device md0, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30
    ReiserFS: md0: checking transaction log (md0)
    ReiserFS: md0: replayed 1 transactions in 0 seconds
    ReiserFS: md0: Using r5 hash to sort names
    VFS: Mounted root (reiserfs filesystem) readonly.
    Freeing unused kernel memory: 240k freed
    
  13. Kernel Config:
    #
    # Automatically generated make config: don't edit
    # Linux kernel version: 2.6.10
    # Thu Dec 30 13:19:47 2004
    #
    CONFIG_X86=y
    CONFIG_MMU=y
    CONFIG_UID16=y
    CONFIG_GENERIC_ISA_DMA=y
    CONFIG_GENERIC_IOMAP=y
    
    #
    # Code maturity level options
    #
    CONFIG_EXPERIMENTAL=y
    CONFIG_CLEAN_COMPILE=y
    CONFIG_LOCK_KERNEL=y
    
    #
    # General setup
    #
    CONFIG_LOCALVERSION=""
    CONFIG_SWAP=y
    CONFIG_SYSVIPC=y
    CONFIG_POSIX_MQUEUE=y
    # CONFIG_BSD_PROCESS_ACCT is not set
    CONFIG_SYSCTL=y
    CONFIG_AUDIT=y
    CONFIG_AUDITSYSCALL=y
    CONFIG_LOG_BUF_SHIFT=15
    CONFIG_HOTPLUG=y
    # CONFIG_KOBJECT_UEVENT is not set
    CONFIG_IKCONFIG=y
    CONFIG_IKCONFIG_PROC=y
    # CONFIG_EMBEDDED is not set
    CONFIG_KALLSYMS=y
    # CONFIG_KALLSYMS_ALL is not set
    # CONFIG_KALLSYMS_EXTRA_PASS is not set
    CONFIG_FUTEX=y
    CONFIG_EPOLL=y
    # CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
    CONFIG_SHMEM=y
    CONFIG_CC_ALIGN_FUNCTIONS=0
    CONFIG_CC_ALIGN_LABELS=0
    CONFIG_CC_ALIGN_LOOPS=0
    CONFIG_CC_ALIGN_JUMPS=0
    # CONFIG_TINY_SHMEM is not set
    
    #
    # Loadable module support
    #
    CONFIG_MODULES=y
    # CONFIG_MODULE_UNLOAD is not set
    CONFIG_OBSOLETE_MODPARM=y
    # CONFIG_MODVERSIONS is not set
    # CONFIG_MODULE_SRCVERSION_ALL is not set
    CONFIG_KMOD=y
    
    #
    # Processor type and features
    #
    CONFIG_X86_PC=y
    # CONFIG_X86_ELAN is not set
    # CONFIG_X86_VOYAGER is not set
    # CONFIG_X86_NUMAQ is not set
    # CONFIG_X86_SUMMIT is not set
    # CONFIG_X86_BIGSMP is not set
    # CONFIG_X86_VISWS is not set
    # CONFIG_X86_GENERICARCH is not set
    # CONFIG_X86_ES7000 is not set
    # CONFIG_M386 is not set
    # CONFIG_M486 is not set
    # CONFIG_M586 is not set
    # CONFIG_M586TSC is not set
    # CONFIG_M586MMX is not set
    # CONFIG_M686 is not set
    # CONFIG_MPENTIUMII is not set
    # CONFIG_MPENTIUMIII is not set
    # CONFIG_MPENTIUMM is not set
    CONFIG_MPENTIUM4=y
    # CONFIG_MK6 is not set
    # CONFIG_MK7 is not set
    # CONFIG_MK8 is not set
    # CONFIG_MCRUSOE is not set
    # CONFIG_MEFFICEON is not set
    # CONFIG_MWINCHIPC6 is not set
    # CONFIG_MWINCHIP2 is not set
    # CONFIG_MWINCHIP3D is not set
    # CONFIG_MCYRIXIII is not set
    # CONFIG_MVIAC3_2 is not set
    # CONFIG_X86_GENERIC is not set
    CONFIG_X86_CMPXCHG=y
    CONFIG_X86_XADD=y
    CONFIG_X86_L1_CACHE_SHIFT=7
    CONFIG_RWSEM_XCHGADD_ALGORITHM=y
    CONFIG_X86_WP_WORKS_OK=y
    CONFIG_X86_INVLPG=y
    CONFIG_X86_BSWAP=y
    CONFIG_X86_POPAD_OK=y
    CONFIG_X86_GOOD_APIC=y
    CONFIG_X86_INTEL_USERCOPY=y
    CONFIG_X86_USE_PPRO_CHECKSUM=y
    CONFIG_HPET_TIMER=y
    # CONFIG_HPET_EMULATE_RTC is not set
    CONFIG_SMP=y
    CONFIG_NR_CPUS=4
    CONFIG_SCHED_SMT=y
    # CONFIG_PREEMPT is not set
    CONFIG_X86_LOCAL_APIC=y
    CONFIG_X86_IO_APIC=y
    CONFIG_X86_TSC=y
    CONFIG_X86_MCE=y
    CONFIG_X86_MCE_NONFATAL=y
    CONFIG_X86_MCE_P4THERMAL=y
    # CONFIG_TOSHIBA is not set
    # CONFIG_I8K is not set
    # CONFIG_MICROCODE is not set
    # CONFIG_X86_MSR is not set
    # CONFIG_X86_CPUID is not set
    
    #
    # Firmware Drivers
    #
    # CONFIG_EDD is not set
    # CONFIG_NOHIGHMEM is not set
    CONFIG_HIGHMEM4G=y
    # CONFIG_HIGHMEM64G is not set
    CONFIG_HIGHMEM=y
    # CONFIG_HIGHPTE is not set
    # CONFIG_MATH_EMULATION is not set
    CONFIG_MTRR=y
    # CONFIG_EFI is not set
    CONFIG_IRQBALANCE=y
    CONFIG_HAVE_DEC_LOCK=y
    # CONFIG_REGPARM is not set
    
    #
    # Power management options (ACPI, APM)
    #
    CONFIG_PM=y
    # CONFIG_PM_DEBUG is not set
    # CONFIG_SOFTWARE_SUSPEND is not set
    
    #
    # ACPI (Advanced Configuration and Power Interface) Support
    #
    CONFIG_ACPI=y
    CONFIG_ACPI_BOOT=y
    CONFIG_ACPI_INTERPRETER=y
    CONFIG_ACPI_SLEEP=y
    CONFIG_ACPI_SLEEP_PROC_FS=y
    CONFIG_ACPI_AC=y
    CONFIG_ACPI_BATTERY=y
    CONFIG_ACPI_BUTTON=y
    # CONFIG_ACPI_VIDEO is not set
    CONFIG_ACPI_FAN=y
    CONFIG_ACPI_PROCESSOR=y
    CONFIG_ACPI_THERMAL=y
    # CONFIG_ACPI_ASUS is not set
    # CONFIG_ACPI_IBM is not set
    # CONFIG_ACPI_TOSHIBA is not set
    CONFIG_ACPI_BLACKLIST_YEAR=0
    # CONFIG_ACPI_DEBUG is not set
    CONFIG_ACPI_BUS=y
    CONFIG_ACPI_EC=y
    CONFIG_ACPI_POWER=y
    CONFIG_ACPI_PCI=y
    CONFIG_ACPI_SYSTEM=y
    # CONFIG_X86_PM_TIMER is not set
    
    #
    # APM (Advanced Power Management) BIOS Support
    #
    # CONFIG_APM is not set
    
    #
    # CPU Frequency scaling
    #
    # CONFIG_CPU_FREQ is not set
    
    #
    # Bus options (PCI, PCMCIA, EISA, MCA, ISA)
    #
    CONFIG_PCI=y
    # CONFIG_PCI_GOBIOS is not set
    # CONFIG_PCI_GOMMCONFIG is not set
    # CONFIG_PCI_GODIRECT is not set
    CONFIG_PCI_GOANY=y
    CONFIG_PCI_BIOS=y
    CONFIG_PCI_DIRECT=y
    CONFIG_PCI_MMCONFIG=y
    # CONFIG_PCI_MSI is not set
    CONFIG_PCI_LEGACY_PROC=y
    CONFIG_PCI_NAMES=y
    CONFIG_ISA=y
    # CONFIG_EISA is not set
    # CONFIG_MCA is not set
    # CONFIG_SCx200 is not set
    
    #
    # PCCARD (PCMCIA/CardBus) support
    #
    # CONFIG_PCCARD is not set
    
    #
    # PC-card bridges
    #
    CONFIG_PCMCIA_PROBE=y
    
    #
    # PCI Hotplug Support
    #
    # CONFIG_HOTPLUG_PCI is not set
    
    #
    # Executable file formats
    #
    CONFIG_BINFMT_ELF=y
    CONFIG_BINFMT_AOUT=y
    CONFIG_BINFMT_MISC=y
    
    #
    # Device Drivers
    #
    
    #
    # Generic Driver Options
    #
    CONFIG_STANDALONE=y
    CONFIG_PREVENT_FIRMWARE_BUILD=y
    CONFIG_FW_LOADER=m
    # CONFIG_DEBUG_DRIVER is not set
    
    #
    # Memory Technology Devices (MTD)
    #
    # CONFIG_MTD is not set
    
    #
    # Parallel port support
    #
    CONFIG_PARPORT=y
    CONFIG_PARPORT_PC=y
    CONFIG_PARPORT_PC_CML1=y
    # CONFIG_PARPORT_SERIAL is not set
    # CONFIG_PARPORT_PC_FIFO is not set
    # CONFIG_PARPORT_PC_SUPERIO is not set
    # CONFIG_PARPORT_OTHER is not set
    # CONFIG_PARPORT_1284 is not set
    
    #
    # Plug and Play support
    #
    CONFIG_PNP=y
    # CONFIG_PNP_DEBUG is not set
    
    #
    # Protocols
    #
    # CONFIG_ISAPNP is not set
    # CONFIG_PNPBIOS is not set
    # CONFIG_PNPACPI is not set
    
    #
    # Block devices
    #
    CONFIG_BLK_DEV_FD=y
    # CONFIG_BLK_DEV_XD is not set
    # CONFIG_PARIDE is not set
    # CONFIG_BLK_CPQ_DA is not set
    # CONFIG_BLK_CPQ_CISS_DA is not set
    # CONFIG_BLK_DEV_DAC960 is not set
    # CONFIG_BLK_DEV_UMEM is not set
    CONFIG_BLK_DEV_LOOP=y
    # CONFIG_BLK_DEV_CRYPTOLOOP is not set
    # CONFIG_BLK_DEV_NBD is not set
    # CONFIG_BLK_DEV_SX8 is not set
    # CONFIG_BLK_DEV_UB is not set
    # CONFIG_BLK_DEV_RAM is not set
    CONFIG_BLK_DEV_RAM_COUNT=16
    CONFIG_INITRAMFS_SOURCE=""
    CONFIG_LBD=y
    # CONFIG_CDROM_PKTCDVD is not set
    
    #
    # IO Schedulers
    #
    CONFIG_IOSCHED_NOOP=y
    CONFIG_IOSCHED_AS=y
    CONFIG_IOSCHED_DEADLINE=y
    CONFIG_IOSCHED_CFQ=y
    
    #
    # ATA/ATAPI/MFM/RLL support
    #
    CONFIG_IDE=y
    CONFIG_BLK_DEV_IDE=y
    
    #
    # Please see Documentation/ide.txt for help/info on IDE drives
    #
    # CONFIG_BLK_DEV_IDE_SATA is not set
    # CONFIG_BLK_DEV_HD_IDE is not set
    CONFIG_BLK_DEV_IDEDISK=y
    CONFIG_IDEDISK_MULTI_MODE=y
    CONFIG_BLK_DEV_IDECD=y
    # CONFIG_BLK_DEV_IDETAPE is not set
    # CONFIG_BLK_DEV_IDEFLOPPY is not set
    # CONFIG_BLK_DEV_IDESCSI is not set
    # CONFIG_IDE_TASK_IOCTL is not set
    
    #
    # IDE chipset support/bugfixes
    #
    CONFIG_IDE_GENERIC=y
    # CONFIG_BLK_DEV_CMD640 is not set
    # CONFIG_BLK_DEV_IDEPNP is not set
    CONFIG_BLK_DEV_IDEPCI=y
    CONFIG_IDEPCI_SHARE_IRQ=y
    # CONFIG_BLK_DEV_OFFBOARD is not set
    CONFIG_BLK_DEV_GENERIC=y
    # CONFIG_BLK_DEV_OPTI621 is not set
    # CONFIG_BLK_DEV_RZ1000 is not set
    CONFIG_BLK_DEV_IDEDMA_PCI=y
    # CONFIG_BLK_DEV_IDEDMA_FORCED is not set
    CONFIG_IDEDMA_PCI_AUTO=y
    # CONFIG_IDEDMA_ONLYDISK is not set
    # CONFIG_BLK_DEV_AEC62XX is not set
    # CONFIG_BLK_DEV_ALI15X3 is not set
    # CONFIG_BLK_DEV_AMD74XX is not set
    # CONFIG_BLK_DEV_ATIIXP is not set
    # CONFIG_BLK_DEV_CMD64X is not set
    # CONFIG_BLK_DEV_TRIFLEX is not set
    # CONFIG_BLK_DEV_CY82C693 is not set
    # CONFIG_BLK_DEV_CS5520 is not set
    # CONFIG_BLK_DEV_CS5530 is not set
    # CONFIG_BLK_DEV_HPT34X is not set
    # CONFIG_BLK_DEV_HPT366 is not set
    # CONFIG_BLK_DEV_SC1200 is not set
    CONFIG_BLK_DEV_PIIX=y
    # CONFIG_BLK_DEV_NS87415 is not set
    # CONFIG_BLK_DEV_PDC202XX_OLD is not set
    # CONFIG_BLK_DEV_PDC202XX_NEW is not set
    # CONFIG_BLK_DEV_SVWKS is not set
    # CONFIG_BLK_DEV_SIIMAGE is not set
    # CONFIG_BLK_DEV_SIS5513 is not set
    # CONFIG_BLK_DEV_SLC90E66 is not set
    # CONFIG_BLK_DEV_TRM290 is not set
    # CONFIG_BLK_DEV_VIA82CXXX is not set
    # CONFIG_IDE_ARM is not set
    # CONFIG_IDE_CHIPSETS is not set
    CONFIG_BLK_DEV_IDEDMA=y
    # CONFIG_IDEDMA_IVB is not set
    CONFIG_IDEDMA_AUTO=y
    # CONFIG_BLK_DEV_HD is not set
    
    #
    # SCSI device support
    #
    CONFIG_SCSI=y
    CONFIG_SCSI_PROC_FS=y
    
    #
    # SCSI support type (disk, tape, CD-ROM)
    #
    CONFIG_BLK_DEV_SD=y
    # CONFIG_CHR_DEV_ST is not set
    # CONFIG_CHR_DEV_OSST is not set
    # CONFIG_BLK_DEV_SR is not set
    CONFIG_CHR_DEV_SG=y
    
    #
    # Some SCSI devices (e.g. CD jukebox) support multiple LUNs
    #
    # CONFIG_SCSI_MULTI_LUN is not set
    # CONFIG_SCSI_CONSTANTS is not set
    # CONFIG_SCSI_LOGGING is not set
    
    #
    # SCSI Transport Attributes
    #
    # CONFIG_SCSI_SPI_ATTRS is not set
    # CONFIG_SCSI_FC_ATTRS is not set
    
    #
    # SCSI low-level drivers
    #
    # CONFIG_BLK_DEV_3W_XXXX_RAID is not set
    # CONFIG_SCSI_3W_9XXX is not set
    # CONFIG_SCSI_7000FASST is not set
    # CONFIG_SCSI_ACARD is not set
    # CONFIG_SCSI_AHA152X is not set
    # CONFIG_SCSI_AHA1542 is not set
    # CONFIG_SCSI_AACRAID is not set
    # CONFIG_SCSI_AIC7XXX is not set
    # CONFIG_SCSI_AIC7XXX_OLD is not set
    CONFIG_SCSI_AIC79XX=y
    CONFIG_AIC79XX_CMDS_PER_DEVICE=32
    CONFIG_AIC79XX_RESET_DELAY_MS=15000
    # CONFIG_AIC79XX_ENABLE_RD_STRM is not set
    # CONFIG_AIC79XX_DEBUG_ENABLE is not set
    CONFIG_AIC79XX_DEBUG_MASK=0
    # CONFIG_AIC79XX_REG_PRETTY_PRINT is not set
    # CONFIG_SCSI_DPT_I2O is not set
    # CONFIG_SCSI_IN2000 is not set
    # CONFIG_MEGARAID_NEWGEN is not set
    # CONFIG_MEGARAID_LEGACY is not set
    # CONFIG_SCSI_SATA is not set
    # CONFIG_SCSI_BUSLOGIC is not set
    # CONFIG_SCSI_DMX3191D is not set
    # CONFIG_SCSI_DTC3280 is not set
    # CONFIG_SCSI_EATA is not set
    # CONFIG_SCSI_EATA_PIO is not set
    # CONFIG_SCSI_FUTURE_DOMAIN is not set
    # CONFIG_SCSI_GDTH is not set
    # CONFIG_SCSI_GENERIC_NCR5380 is not set
    # CONFIG_SCSI_GENERIC_NCR5380_MMIO is not set
    # CONFIG_SCSI_IPS is not set
    # CONFIG_SCSI_INITIO is not set
    # CONFIG_SCSI_INIA100 is not set
    # CONFIG_SCSI_PPA is not set
    # CONFIG_SCSI_IMM is not set
    # CONFIG_SCSI_NCR53C406A is not set
    # CONFIG_SCSI_SYM53C8XX_2 is not set
    # CONFIG_SCSI_IPR is not set
    # CONFIG_SCSI_PAS16 is not set
    # CONFIG_SCSI_PSI240I is not set
    # CONFIG_SCSI_QLOGIC_FAS is not set
    # CONFIG_SCSI_QLOGIC_ISP is not set
    # CONFIG_SCSI_QLOGIC_FC is not set
    # CONFIG_SCSI_QLOGIC_1280 is not set
    CONFIG_SCSI_QLA2XXX=y
    # CONFIG_SCSI_QLA21XX is not set
    # CONFIG_SCSI_QLA22XX is not set
    # CONFIG_SCSI_QLA2300 is not set
    # CONFIG_SCSI_QLA2322 is not set
    # CONFIG_SCSI_QLA6312 is not set
    # CONFIG_SCSI_QLA6322 is not set
    # CONFIG_SCSI_SYM53C416 is not set
    # CONFIG_SCSI_DC395x is not set
    # CONFIG_SCSI_DC390T is not set
    # CONFIG_SCSI_T128 is not set
    # CONFIG_SCSI_U14_34F is not set
    # CONFIG_SCSI_ULTRASTOR is not set
    # CONFIG_SCSI_NSP32 is not set
    # CONFIG_SCSI_DEBUG is not set
    
    #
    # Old CD-ROM drivers (not SCSI, not IDE)
    #
    # CONFIG_CD_NO_IDESCSI is not set
    
    #
    # Multi-device support (RAID and LVM)
    #
    CONFIG_MD=y
    CONFIG_BLK_DEV_MD=y
    CONFIG_MD_LINEAR=y
    CONFIG_MD_RAID0=y
    CONFIG_MD_RAID1=y
    CONFIG_MD_RAID10=y
    CONFIG_MD_RAID5=y
    CONFIG_MD_RAID6=y
    CONFIG_MD_MULTIPATH=y
    # CONFIG_MD_FAULTY is not set
    CONFIG_BLK_DEV_DM=y
    CONFIG_DM_CRYPT=y
    CONFIG_DM_SNAPSHOT=y
    CONFIG_DM_MIRROR=y
    CONFIG_DM_ZERO=y
    
    #
    # Fusion MPT device support
    #
    CONFIG_FUSION=y
    CONFIG_FUSION_MAX_SGE=40
    # CONFIG_FUSION_CTL is not set
    
    #
    # IEEE 1394 (FireWire) support
    #
    # CONFIG_IEEE1394 is not set
    
    #
    # I2O device support
    #
    # CONFIG_I2O is not set
    
    #
    # Networking support
    #
    CONFIG_NET=y
    
    #
    # Networking options
    #
    CONFIG_PACKET=y
    # CONFIG_PACKET_MMAP is not set
    # CONFIG_NETLINK_DEV is not set
    CONFIG_UNIX=y
    # CONFIG_NET_KEY is not set
    CONFIG_INET=y
    CONFIG_IP_MULTICAST=y
    # CONFIG_IP_ADVANCED_ROUTER is not set
    # CONFIG_IP_PNP is not set
    # CONFIG_NET_IPIP is not set
    # CONFIG_NET_IPGRE is not set
    # CONFIG_IP_MROUTE is not set
    # CONFIG_ARPD is not set
    # CONFIG_SYN_COOKIES is not set
    # CONFIG_INET_AH is not set
    # CONFIG_INET_ESP is not set
    # CONFIG_INET_IPCOMP is not set
    # CONFIG_INET_TUNNEL is not set
    # CONFIG_IP_TCPDIAG is not set
    # CONFIG_IP_TCPDIAG_IPV6 is not set
    
    #
    # IP: Virtual Server Configuration
    #
    # CONFIG_IP_VS is not set
    # CONFIG_IPV6 is not set
    CONFIG_NETFILTER=y
    # CONFIG_NETFILTER_DEBUG is not set
    
    #
    # IP: Netfilter Configuration
    #
    CONFIG_IP_NF_CONNTRACK=y
    # CONFIG_IP_NF_CT_ACCT is not set
    # CONFIG_IP_NF_CONNTRACK_MARK is not set
    # CONFIG_IP_NF_CT_PROTO_SCTP is not set
    # CONFIG_IP_NF_FTP is not set
    # CONFIG_IP_NF_IRC is not set
    # CONFIG_IP_NF_TFTP is not set
    # CONFIG_IP_NF_AMANDA is not set
    CONFIG_IP_NF_QUEUE=y
    CONFIG_IP_NF_IPTABLES=y
    CONFIG_IP_NF_MATCH_LIMIT=y
    CONFIG_IP_NF_MATCH_IPRANGE=y
    CONFIG_IP_NF_MATCH_MAC=y
    CONFIG_IP_NF_MATCH_PKTTYPE=y
    CONFIG_IP_NF_MATCH_MARK=y
    CONFIG_IP_NF_MATCH_MULTIPORT=y
    CONFIG_IP_NF_MATCH_TOS=y
    CONFIG_IP_NF_MATCH_RECENT=y
    CONFIG_IP_NF_MATCH_ECN=y
    CONFIG_IP_NF_MATCH_DSCP=y
    CONFIG_IP_NF_MATCH_AH_ESP=y
    CONFIG_IP_NF_MATCH_LENGTH=y
    CONFIG_IP_NF_MATCH_TTL=y
    CONFIG_IP_NF_MATCH_TCPMSS=y
    CONFIG_IP_NF_MATCH_HELPER=y
    CONFIG_IP_NF_MATCH_STATE=y
    CONFIG_IP_NF_MATCH_CONNTRACK=y
    CONFIG_IP_NF_MATCH_OWNER=y
    # CONFIG_IP_NF_MATCH_ADDRTYPE is not set
    # CONFIG_IP_NF_MATCH_REALM is not set
    # CONFIG_IP_NF_MATCH_SCTP is not set
    # CONFIG_IP_NF_MATCH_COMMENT is not set
    # CONFIG_IP_NF_MATCH_HASHLIMIT is not set
    CONFIG_IP_NF_FILTER=y
    CONFIG_IP_NF_TARGET_REJECT=y
    CONFIG_IP_NF_TARGET_LOG=y
    CONFIG_IP_NF_TARGET_ULOG=y
    CONFIG_IP_NF_TARGET_TCPMSS=y
    CONFIG_IP_NF_NAT=y
    CONFIG_IP_NF_NAT_NEEDED=y
    CONFIG_IP_NF_TARGET_MASQUERADE=y
    CONFIG_IP_NF_TARGET_REDIRECT=y
    CONFIG_IP_NF_TARGET_NETMAP=y
    CONFIG_IP_NF_TARGET_SAME=y
    # CONFIG_IP_NF_NAT_LOCAL is not set
    # CONFIG_IP_NF_NAT_SNMP_BASIC is not set
    CONFIG_IP_NF_MANGLE=y
    CONFIG_IP_NF_TARGET_TOS=y
    CONFIG_IP_NF_TARGET_ECN=y
    CONFIG_IP_NF_TARGET_DSCP=y
    CONFIG_IP_NF_TARGET_MARK=y
    CONFIG_IP_NF_TARGET_CLASSIFY=y
    CONFIG_IP_NF_RAW=m
    CONFIG_IP_NF_TARGET_NOTRACK=m
    CONFIG_IP_NF_ARPTABLES=y
    CONFIG_IP_NF_ARPFILTER=y
    CONFIG_IP_NF_ARP_MANGLE=y
    
    #
    # SCTP Configuration (EXPERIMENTAL)
    #
    # CONFIG_IP_SCTP is not set
    # CONFIG_ATM is not set
    # CONFIG_BRIDGE is not set
    # CONFIG_VLAN_8021Q is not set
    # CONFIG_DECNET is not set
    # CONFIG_LLC2 is not set
    # CONFIG_IPX is not set
    # CONFIG_ATALK is not set
    # CONFIG_X25 is not set
    # CONFIG_LAPB is not set
    # CONFIG_NET_DIVERT is not set
    # CONFIG_ECONET is not set
    # CONFIG_WAN_ROUTER is not set
    
    #
    # QoS and/or fair queueing
    #
    # CONFIG_NET_SCHED is not set
    # CONFIG_NET_CLS_ROUTE is not set
    
    #
    # Network testing
    #
    # CONFIG_NET_PKTGEN is not set
    # CONFIG_NETPOLL is not set
    # CONFIG_NET_POLL_CONTROLLER is not set
    # CONFIG_HAMRADIO is not set
    # CONFIG_IRDA is not set
    # CONFIG_BT is not set
    CONFIG_NETDEVICES=y
    CONFIG_DUMMY=y
    # CONFIG_BONDING is not set
    # CONFIG_EQUALIZER is not set
    # CONFIG_TUN is not set
    # CONFIG_NET_SB1000 is not set
    
    #
    # ARCnet devices
    #
    # CONFIG_ARCNET is not set
    
    #
    # Ethernet (10 or 100Mbit)
    #
    CONFIG_NET_ETHERNET=y
    CONFIG_MII=y
    # CONFIG_HAPPYMEAL is not set
    # CONFIG_SUNGEM is not set
    # CONFIG_NET_VENDOR_3COM is not set
    # CONFIG_LANCE is not set
    # CONFIG_NET_VENDOR_SMC is not set
    # CONFIG_NET_VENDOR_RACAL is not set
    
    #
    # Tulip family network device support
    #
    # CONFIG_NET_TULIP is not set
    # CONFIG_AT1700 is not set
    # CONFIG_DEPCA is not set
    # CONFIG_HP100 is not set
    # CONFIG_NET_ISA is not set
    CONFIG_NET_PCI=y
    # CONFIG_PCNET32 is not set
    # CONFIG_AMD8111_ETH is not set
    # CONFIG_ADAPTEC_STARFIRE is not set
    # CONFIG_AC3200 is not set
    # CONFIG_APRICOT is not set
    # CONFIG_B44 is not set
    # CONFIG_FORCEDETH is not set
    # CONFIG_CS89x0 is not set
    # CONFIG_DGRS is not set
    # CONFIG_EEPRO100 is not set
    CONFIG_E100=y
    # CONFIG_E100_NAPI is not set
    # CONFIG_FEALNX is not set
    # CONFIG_NATSEMI is not set
    # CONFIG_NE2K_PCI is not set
    # CONFIG_8139CP is not set
    # CONFIG_8139TOO is not set
    # CONFIG_SIS900 is not set
    # CONFIG_EPIC100 is not set
    # CONFIG_SUNDANCE is not set
    # CONFIG_TLAN is not set
    # CONFIG_VIA_RHINE is not set
    # CONFIG_NET_POCKET is not set
    
    #
    # Ethernet (1000 Mbit)
    #
    # CONFIG_ACENIC is not set
    # CONFIG_DL2K is not set
    CONFIG_E1000=y
    # CONFIG_E1000_NAPI is not set
    # CONFIG_NS83820 is not set
    # CONFIG_HAMACHI is not set
    # CONFIG_YELLOWFIN is not set
    # CONFIG_R8169 is not set
    # CONFIG_SK98LIN is not set
    # CONFIG_VIA_VELOCITY is not set
    # CONFIG_TIGON3 is not set
    
    #
    # Ethernet (10000 Mbit)
    #
    # CONFIG_IXGB is not set
    CONFIG_S2IO=m
    # CONFIG_S2IO_NAPI is not set
    
    #
    # Token Ring devices
    #
    # CONFIG_TR is not set
    
    #
    # Wireless LAN (non-hamradio)
    #
    # CONFIG_NET_RADIO is not set
    
    #
    # Wan interfaces
    #
    # CONFIG_WAN is not set
    # CONFIG_FDDI is not set
    # CONFIG_HIPPI is not set
    # CONFIG_PLIP is not set
    # CONFIG_PPP is not set
    # CONFIG_SLIP is not set
    # CONFIG_NET_FC is not set
    # CONFIG_SHAPER is not set
    # CONFIG_NETCONSOLE is not set
    
    #
    # ISDN subsystem
    #
    # CONFIG_ISDN is not set
    
    #
    # Telephony Support
    #
    # CONFIG_PHONE is not set
    
    #
    # Input device support
    #
    CONFIG_INPUT=y
    
    #
    # Userland interfaces
    #
    CONFIG_INPUT_MOUSEDEV=y
    CONFIG_INPUT_MOUSEDEV_PSAUX=y
    CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
    CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
    # CONFIG_INPUT_JOYDEV is not set
    # CONFIG_INPUT_TSDEV is not set
    CONFIG_INPUT_EVDEV=y
    # CONFIG_INPUT_EVBUG is not set
    
    #
    # Input I/O drivers
    #
    # CONFIG_GAMEPORT is not set
    CONFIG_SOUND_GAMEPORT=y
    CONFIG_SERIO=y
    CONFIG_SERIO_I8042=y
    # CONFIG_SERIO_SERPORT is not set
    # CONFIG_SERIO_CT82C710 is not set
    # CONFIG_SERIO_PARKBD is not set
    # CONFIG_SERIO_PCIPS2 is not set
    # CONFIG_SERIO_RAW is not set
    
    #
    # Input Device Drivers
    #
    CONFIG_INPUT_KEYBOARD=y
    CONFIG_KEYBOARD_ATKBD=y
    # CONFIG_KEYBOARD_SUNKBD is not set
    # CONFIG_KEYBOARD_LKKBD is not set
    # CONFIG_KEYBOARD_XTKBD is not set
    # CONFIG_KEYBOARD_NEWTON is not set
    CONFIG_INPUT_MOUSE=y
    CONFIG_MOUSE_PS2=y
    # CONFIG_MOUSE_SERIAL is not set
    # CONFIG_MOUSE_INPORT is not set
    # CONFIG_MOUSE_LOGIBM is not set
    # CONFIG_MOUSE_PC110PAD is not set
    # CONFIG_MOUSE_VSXXXAA is not set
    # CONFIG_INPUT_JOYSTICK is not set
    # CONFIG_INPUT_TOUCHSCREEN is not set
    # CONFIG_INPUT_MISC is not set
    
    #
    # Character devices
    #
    CONFIG_VT=y
    CONFIG_VT_CONSOLE=y
    CONFIG_HW_CONSOLE=y
    # CONFIG_SERIAL_NONSTANDARD is not set
    
    #
    # Serial drivers
    #
    CONFIG_SERIAL_8250=y
    CONFIG_SERIAL_8250_CONSOLE=y
    # CONFIG_SERIAL_8250_ACPI is not set
    CONFIG_SERIAL_8250_NR_UARTS=4
    # CONFIG_SERIAL_8250_EXTENDED is not set
    
    #
    # Non-8250 serial port support
    #
    CONFIG_SERIAL_CORE=y
    CONFIG_SERIAL_CORE_CONSOLE=y
    CONFIG_UNIX98_PTYS=y
    CONFIG_LEGACY_PTYS=y
    CONFIG_LEGACY_PTY_COUNT=256
    # CONFIG_PRINTER is not set
    # CONFIG_PPDEV is not set
    # CONFIG_TIPAR is not set
    
    #
    # IPMI
    #
    # CONFIG_IPMI_HANDLER is not set
    
    #
    # Watchdog Cards
    #
    # CONFIG_WATCHDOG is not set
    # CONFIG_HW_RANDOM is not set
    # CONFIG_NVRAM is not set
    CONFIG_RTC=y
    # CONFIG_DTLK is not set
    # CONFIG_R3964 is not set
    # CONFIG_APPLICOM is not set
    # CONFIG_SONYPI is not set
    
    #
    # Ftape, the floppy tape device driver
    #
    # CONFIG_AGP is not set
    # CONFIG_DRM is not set
    # CONFIG_MWAVE is not set
    # CONFIG_RAW_DRIVER is not set
    # CONFIG_HPET is not set
    # CONFIG_HANGCHECK_TIMER is not set
    
    #
    # I2C support
    #
    # CONFIG_I2C is not set
    
    #
    # Dallas's 1-wire bus
    #
    # CONFIG_W1 is not set
    
    #
    # Misc devices
    #
    # CONFIG_IBM_ASM is not set
    
    #
    # Multimedia devices
    #
    # CONFIG_VIDEO_DEV is not set
    
    #
    # Digital Video Broadcasting Devices
    #
    # CONFIG_DVB is not set
    
    #
    # Graphics support
    #
    # CONFIG_FB is not set
    # CONFIG_VIDEO_SELECT is not set
    
    #
    # Console display driver support
    #
    CONFIG_VGA_CONSOLE=y
    # CONFIG_MDA_CONSOLE is not set
    CONFIG_DUMMY_CONSOLE=y
    
    #
    # Sound
    #
    # CONFIG_SOUND is not set
    
    #
    # USB support
    #
    CONFIG_USB=y
    # CONFIG_USB_DEBUG is not set
    
    #
    # Miscellaneous USB options
    #
    CONFIG_USB_DEVICEFS=y
    # CONFIG_USB_BANDWIDTH is not set
    # CONFIG_USB_DYNAMIC_MINORS is not set
    # CONFIG_USB_SUSPEND is not set
    # CONFIG_USB_OTG is not set
    CONFIG_USB_ARCH_HAS_HCD=y
    CONFIG_USB_ARCH_HAS_OHCI=y
    
    #
    # USB Host Controller Drivers
    #
    # CONFIG_USB_EHCI_HCD is not set
    CONFIG_USB_OHCI_HCD=y
    CONFIG_USB_UHCI_HCD=y
    # CONFIG_USB_SL811_HCD is not set
    
    #
    # USB Device Class drivers
    #
    # CONFIG_USB_BLUETOOTH_TTY is not set
    # CONFIG_USB_ACM is not set
    CONFIG_USB_PRINTER=y
    
    #
    # NOTE: USB_STORAGE enables SCSI, and 'SCSI disk support' may also be needed; see USB_STORAGE Help for more information
    #
    CONFIG_USB_STORAGE=y
    # CONFIG_USB_STORAGE_DEBUG is not set
    # CONFIG_USB_STORAGE_RW_DETECT is not set
    CONFIG_USB_STORAGE_DATAFAB=y
    CONFIG_USB_STORAGE_FREECOM=y
    CONFIG_USB_STORAGE_ISD200=y
    CONFIG_USB_STORAGE_DPCM=y
    # CONFIG_USB_STORAGE_HP8200e is not set
    CONFIG_USB_STORAGE_SDDR09=y
    CONFIG_USB_STORAGE_SDDR55=y
    CONFIG_USB_STORAGE_JUMPSHOT=y
    
    #
    # USB Input Devices
    #
    CONFIG_USB_HID=y
    CONFIG_USB_HIDINPUT=y
    # CONFIG_HID_FF is not set
    CONFIG_USB_HIDDEV=y
    # CONFIG_USB_AIPTEK is not set
    # CONFIG_USB_WACOM is not set
    # CONFIG_USB_KBTAB is not set
    # CONFIG_USB_POWERMATE is not set
    # CONFIG_USB_MTOUCH is not set
    # CONFIG_USB_EGALAX is not set
    # CONFIG_USB_XPAD is not set
    # CONFIG_USB_ATI_REMOTE is not set
    
    #
    # USB Imaging devices
    #
    # CONFIG_USB_MDC800 is not set
    # CONFIG_USB_MICROTEK is not set
    # CONFIG_USB_HPUSBSCSI is not set
    
    #
    # USB Multimedia devices
    #
    # CONFIG_USB_DABUSB is not set
    
    #
    # Video4Linux support is needed for USB Multimedia device support
    #
    
    #
    # USB Network Adapters
    #
    # CONFIG_USB_CATC is not set
    # CONFIG_USB_KAWETH is not set
    # CONFIG_USB_PEGASUS is not set
    # CONFIG_USB_RTL8150 is not set
    # CONFIG_USB_USBNET is not set
    
    #
    # USB port drivers
    #
    # CONFIG_USB_USS720 is not set
    
    #
    # USB Serial Converter support
    #
    # CONFIG_USB_SERIAL is not set
    
    #
    # USB Miscellaneous drivers
    #
    # CONFIG_USB_EMI62 is not set
    # CONFIG_USB_EMI26 is not set
    # CONFIG_USB_TIGL is not set
    # CONFIG_USB_AUERSWALD is not set
    # CONFIG_USB_RIO500 is not set
    # CONFIG_USB_LEGOTOWER is not set
    # CONFIG_USB_LCD is not set
    # CONFIG_USB_LED is not set
    # CONFIG_USB_CYTHERM is not set
    # CONFIG_USB_PHIDGETKIT is not set
    # CONFIG_USB_PHIDGETSERVO is not set
    # CONFIG_USB_TEST is not set
    
    #
    # USB ATM/DSL drivers
    #
    
    #
    # USB Gadget Support
    #
    # CONFIG_USB_GADGET is not set
    
    #
    # MMC/SD Card support
    #
    # CONFIG_MMC is not set
    
    #
    # File systems
    #
    CONFIG_EXT2_FS=y
    # CONFIG_EXT2_FS_XATTR is not set
    CONFIG_EXT3_FS=y
    # CONFIG_EXT3_FS_XATTR is not set
    CONFIG_JBD=y
    # CONFIG_JBD_DEBUG is not set
    CONFIG_REISERFS_FS=y
    # CONFIG_REISERFS_CHECK is not set
    # CONFIG_REISERFS_PROC_INFO is not set
    # CONFIG_REISERFS_FS_XATTR is not set
    CONFIG_JFS_FS=y
    # CONFIG_JFS_POSIX_ACL is not set
    # CONFIG_JFS_DEBUG is not set
    # CONFIG_JFS_STATISTICS is not set
    CONFIG_XFS_FS=y
    # CONFIG_XFS_RT is not set
    # CONFIG_XFS_QUOTA is not set
    # CONFIG_XFS_SECURITY is not set
    # CONFIG_XFS_POSIX_ACL is not set
    # CONFIG_MINIX_FS is not set
    # CONFIG_ROMFS_FS is not set
    # CONFIG_QUOTA is not set
    CONFIG_DNOTIFY=y
    # CONFIG_AUTOFS_FS is not set
    # CONFIG_AUTOFS4_FS is not set
    
    #
    # CD-ROM/DVD Filesystems
    #
    CONFIG_ISO9660_FS=y
    CONFIG_JOLIET=y
    # CONFIG_ZISOFS is not set
    CONFIG_UDF_FS=y
    CONFIG_UDF_NLS=y
    
    #
    # DOS/FAT/NT Filesystems
    #
    CONFIG_FAT_FS=y
    CONFIG_MSDOS_FS=y
    CONFIG_VFAT_FS=y
    CONFIG_FAT_DEFAULT_CODEPAGE=437
    CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
    # CONFIG_NTFS_FS is not set
    
    #
    # Pseudo filesystems
    #
    CONFIG_PROC_FS=y
    CONFIG_PROC_KCORE=y
    CONFIG_SYSFS=y
    # CONFIG_DEVFS_FS is not set
    # CONFIG_DEVPTS_FS_XATTR is not set
    CONFIG_TMPFS=y
    # CONFIG_TMPFS_XATTR is not set
    # CONFIG_HUGETLBFS is not set
    # CONFIG_HUGETLB_PAGE is not set
    CONFIG_RAMFS=y
    
    #
    # Miscellaneous filesystems
    #
    # CONFIG_ADFS_FS is not set
    # CONFIG_AFFS_FS is not set
    # CONFIG_HFS_FS is not set
    # CONFIG_HFSPLUS_FS is not set
    # CONFIG_BEFS_FS is not set
    # CONFIG_BFS_FS is not set
    # CONFIG_EFS_FS is not set
    # CONFIG_CRAMFS is not set
    # CONFIG_VXFS_FS is not set
    # CONFIG_HPFS_FS is not set
    # CONFIG_QNX4FS_FS is not set
    # CONFIG_SYSV_FS is not set
    # CONFIG_UFS_FS is not set
    
    #
    # Network File Systems
    #
    CONFIG_NFS_FS=y
    CONFIG_NFS_V3=y
    # CONFIG_NFS_V4 is not set
    # CONFIG_NFS_DIRECTIO is not set
    CONFIG_NFSD=y
    CONFIG_NFSD_V3=y
    # CONFIG_NFSD_V4 is not set
    CONFIG_NFSD_TCP=y
    CONFIG_LOCKD=y
    CONFIG_LOCKD_V4=y
    CONFIG_EXPORTFS=y
    CONFIG_SUNRPC=y
    # CONFIG_RPCSEC_GSS_KRB5 is not set
    # CONFIG_RPCSEC_GSS_SPKM3 is not set
    CONFIG_SMB_FS=y
    # CONFIG_SMB_NLS_DEFAULT is not set
    # CONFIG_CIFS is not set
    # CONFIG_NCP_FS is not set
    # CONFIG_CODA_FS is not set
    # CONFIG_AFS_FS is not set
    
    #
    # Partition Types
    #
    # CONFIG_PARTITION_ADVANCED is not set
    CONFIG_MSDOS_PARTITION=y
    
    #
    # Native Language Support
    #
    CONFIG_NLS=y
    CONFIG_NLS_DEFAULT="iso8859-1"
    CONFIG_NLS_CODEPAGE_437=y
    # CONFIG_NLS_CODEPAGE_737 is not set
    # CONFIG_NLS_CODEPAGE_775 is not set
    # CONFIG_NLS_CODEPAGE_850 is not set
    # CONFIG_NLS_CODEPAGE_852 is not set
    # CONFIG_NLS_CODEPAGE_855 is not set
    # CONFIG_NLS_CODEPAGE_857 is not set
    # CONFIG_NLS_CODEPAGE_860 is not set
    # CONFIG_NLS_CODEPAGE_861 is not set
    # CONFIG_NLS_CODEPAGE_862 is not set
    # CONFIG_NLS_CODEPAGE_863 is not set
    # CONFIG_NLS_CODEPAGE_864 is not set
    # CONFIG_NLS_CODEPAGE_865 is not set
    # CONFIG_NLS_CODEPAGE_866 is not set
    # CONFIG_NLS_CODEPAGE_869 is not set
    # CONFIG_NLS_CODEPAGE_936 is not set
    # CONFIG_NLS_CODEPAGE_950 is not set
    # CONFIG_NLS_CODEPAGE_932 is not set
    # CONFIG_NLS_CODEPAGE_949 is not set
    # CONFIG_NLS_CODEPAGE_874 is not set
    # CONFIG_NLS_ISO8859_8 is not set
    # CONFIG_NLS_CODEPAGE_1250 is not set
    # CONFIG_NLS_CODEPAGE_1251 is not set
    # CONFIG_NLS_ASCII is not set
    CONFIG_NLS_ISO8859_1=y
    # CONFIG_NLS_ISO8859_2 is not set
    # CONFIG_NLS_ISO8859_3 is not set
    # CONFIG_NLS_ISO8859_4 is not set
    # CONFIG_NLS_ISO8859_5 is not set
    # CONFIG_NLS_ISO8859_6 is not set
    # CONFIG_NLS_ISO8859_7 is not set
    # CONFIG_NLS_ISO8859_9 is not set
    # CONFIG_NLS_ISO8859_13 is not set
    # CONFIG_NLS_ISO8859_14 is not set
    # CONFIG_NLS_ISO8859_15 is not set
    # CONFIG_NLS_KOI8_R is not set
    # CONFIG_NLS_KOI8_U is not set
    # CONFIG_NLS_UTF8 is not set
    
    #
    # Profiling support
    #
    # CONFIG_PROFILING is not set
    
    #
    # Kernel hacking
    #
    CONFIG_DEBUG_KERNEL=y
    CONFIG_MAGIC_SYSRQ=y
    # CONFIG_SCHEDSTATS is not set
    # CONFIG_DEBUG_SLAB is not set
    # CONFIG_DEBUG_SPINLOCK is not set
    # CONFIG_DEBUG_SPINLOCK_SLEEP is not set
    # CONFIG_DEBUG_KOBJECT is not set
    # CONFIG_DEBUG_HIGHMEM is not set
    # CONFIG_DEBUG_INFO is not set
    # CONFIG_FRAME_POINTER is not set
    CONFIG_EARLY_PRINTK=y
    # CONFIG_DEBUG_STACKOVERFLOW is not set
    # CONFIG_KPROBES is not set
    # CONFIG_DEBUG_STACK_USAGE is not set
    # CONFIG_DEBUG_PAGEALLOC is not set
    # CONFIG_4KSTACKS is not set
    CONFIG_X86_FIND_SMP_CONFIG=y
    CONFIG_X86_MPPARSE=y
    
    #
    # Security options
    #
    # CONFIG_KEYS is not set
    # CONFIG_SECURITY is not set
    
    #
    # Cryptographic options
    #
    CONFIG_CRYPTO=y
    # CONFIG_CRYPTO_HMAC is not set
    # CONFIG_CRYPTO_NULL is not set
    # CONFIG_CRYPTO_MD4 is not set
    # CONFIG_CRYPTO_MD5 is not set
    # CONFIG_CRYPTO_SHA1 is not set
    # CONFIG_CRYPTO_SHA256 is not set
    # CONFIG_CRYPTO_SHA512 is not set
    # CONFIG_CRYPTO_WP512 is not set
    # CONFIG_CRYPTO_DES is not set
    # CONFIG_CRYPTO_BLOWFISH is not set
    # CONFIG_CRYPTO_TWOFISH is not set
    # CONFIG_CRYPTO_SERPENT is not set
    # CONFIG_CRYPTO_AES_586 is not set
    # CONFIG_CRYPTO_CAST5 is not set
    # CONFIG_CRYPTO_CAST6 is not set
    # CONFIG_CRYPTO_TEA is not set
    # CONFIG_CRYPTO_ARC4 is not set
    # CONFIG_CRYPTO_KHAZAD is not set
    # CONFIG_CRYPTO_ANUBIS is not set
    # CONFIG_CRYPTO_DEFLATE is not set
    # CONFIG_CRYPTO_MICHAEL_MIC is not set
    # CONFIG_CRYPTO_CRC32C is not set
    # CONFIG_CRYPTO_TEST is not set
    
    #
    # Library routines
    #
    # CONFIG_CRC_CCITT is not set
    CONFIG_CRC32=y
    CONFIG_LIBCRC32C=m
    CONFIG_GENERIC_HARDIRQS=y
    CONFIG_GENERIC_IRQ_PROBE=y
    CONFIG_X86_SMP=y
    CONFIG_X86_HT=y
    CONFIG_X86_BIOS_REBOOT=y
    CONFIG_X86_TRAMPOLINE=y
    CONFIG_PC=y