Pentagon seeks to test & evaluate AI products

The Pentagon is looking to the industry regarding how to better test and evaluate artificial intelligence products in the pipeline to ensure safety and effectiveness.

In a request for information this week, the Pentagon’s Joint Artificial Intelligence Center(JAIC), seeks input on cutting-edge testing and evaluation capabilities to support the “full spectrum” of the Defense Department’s emerging AI technologies including:

1. Machine learning, 
2. Deep learning & 
3. Neural networks 

Read RFI - 832016142.docx :

Stated Objectives:

• The Pentagon wants to augment the JAIC’s Test and Evaluation office, which develops standards and conducts algorithm testing, system testing and operational testing on the military’s many AI initiatives.
• The Pentagon stood up the JAIC in 2018 to centralize coordination and accelerate the adoption of AI and has been building out its ranks in recent months, hiring an official to implement its new AI ethical principles for warfare.
• The JAIC is requesting testing tools and expertise in planning, data management, and analysis of inputs and outputs associated with those tools. 

• The introduction of AI-enabled systems brings changes to the process, metrics, data, and skills necessary to produce the level of testing the military needs, and that is the reason for requesting information.
• Testing and Evaluation provides knowledge of system capabilities and limitations to the acquisition community and to the war-fighter. 
• The JAIC's T&E team will make rigorous and objective assessments of systems under operational conditions and against realistic threats, so that our war fighters ultimately trust the systems they are operating and that the risks associated with operating these systems are well-known to military acquisition decision-makers.

The solicitation indicates it plans to use feedback from the solicitation to guide how it further builds out its capabilities. 

The Pentagon is interested in tech testing tools that focus on:

• Conversational interface applications using voice to text.
• Speech-enabled products and services for DOD applications and systems.
• Image analysis, testing deep learning-based visual search and image classifier.
• Natural Language Processing-enabled products and services.
• Humans augmented by machines, to include human-machine interfaces and improved methods to measure war-fighter cognitive and physical workloads, to include augmented reality and virtual reality test services.
•  Autonomous systems.

The Pentagon also wants feedback regarding evaluation services in five mission areas: 

1. Dateset curation
2. Test harness development
3. Model output analysis
4. Test reporting 
5. Testing services

Finally the pentagon also seeks “other technologies” that  it may not be aware of that “may be beneficial” to testing and evaluation efforts.

~ Jai Krishna Ponnappan

AI-Search & Rescue Defense project at Sea

Australian defense project tests potential of AI at sea

The Australian Department of Defense is conducting trails to test the potential of an artificial intelligence (AI) system in the AI-Search project at sea.

Tests conducted in the search-and-rescue (SAR) trials as part of the project will recognize the potential of AI to augment and enhance SAR and to save lives at sea.

The project is conducted in collaboration with Warfare Innovation Navy Branch, Plan Jericho, the Royal Australian Air Force (RAAF) Air Mobility Group’s No 35 Squadron, and the University of Tasmania’s Australian Maritime College.

Modern AI is used for the detection of small and difficult-to-spot targets, including life rafts and individual survivors.

Plan Jericho AI lead wing commander Michael Gan said: “The idea was to train a machine-learning algorithm and AI sensors to complement existing visual search techniques.

“Our vision was to give any aircraft and other defense platforms, including unmanned aerial systems, a low-cost, improvised SAR capability.”

A series of new machine-learning algorithms were developed for the AI. Deterministic processes to analyse the imagery collected by camera sensors and aid human observers were also used.

Last year saw the first successful trial conducted aboard a RAAF C-27J Spartan. The second trial was performed last month near Stradbroke Island, Queensland.

During these trials, a range of small targets were detected in a wide sea area while training the algorithm as part of the project.

The trials highlighted the feasibility of the technology and its easy integration into other Australian Defense Forces (ADF) airborne platforms.

Warfare Innovation Navy Branch lieutenant Harry Hubbert said: “There is a lot of discussion about AI in Defense but the sheer processing power of machine-learning applied to SAR has the potential to save lives and transform it.”