Ultrasonic Non-Destructive Testing (NDT) and imaging systems has been widely used for industrial and medical applications. In NDT system, detection and characterization of target signal can be extremely challenging because of the complex echo scattering environment and the system noise. In this paper, an algorithm based on Neural Network (NN) is presented to explore the possible solutions for ultrasonic target detection. To reduce the computation load and increase the precision of the NN, signal processing algorithms such as Split-Spectrum Processing (SSP), FIR filtering etc. are applied to the signal. In this study, the algorithm is designed to perform target detection on an ultrasonic testing platform based on Zynq System-on-Chip (SoC) in real-time. The speed of computation is crucial for a real-time testing and signal processing, especially when sampling rate is high. The proposed system can generate, capture and process ultrasonic signals. In this design, the FPGA fabric on the Zynq SoC can be used to accelerate the algorithm and to enable real-time split-spectrum processing followed by neural networks.