Fix/docs nav (#624)

* Fixing javadoc generation scripts

* Add docs nav and landing page

* Update JavaCPP notice

* Renamed tools to legacy

* Delete old docs scripts

* Add mention for legacy doc tools

Author: karllessard

.gitignore

@@ -64,3 +64,6 @@ gradleBuild
 # Deployment Files
 settings.xml
 pom.xml.asc
+
+# Docs
+docs/docs/apidocs/

docs/_toc.yaml

@@ -1,14 +1,5 @@
 # TensorFlow for Java
 
-<table class="tfo-notebook-buttons" align="left">
-    <td>
-    <a target="_blank" href="https://www.tensorflow.org/jvm"><img src="https://www.tensorflow.org/images/tf_logo_32px.png" />View on TensorFlow.org</a>
-  </td>
-  <td>
-    <a target="_blank" href="https://github.com/tensorflow/java"><img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png" />View GitHub repository</a>
-  </td>
-</table>
-
 TensorFlow Java can run on any JVM for building, training and running machine learning models. It comes with 
 a series of utilities and frameworks that help achieve most of the tasks common to data scientists 
 and developers working in this domain. Java and other JVM languages, such as Scala or Kotlin, are 
@@ -26,21 +17,19 @@ migrated from Bazel to Maven, which is more familiar for most Java developers.
 
 The following describes the layout of the repository and its different artifacts:
 
-* [tensorflow-core](https://github.com/tensorflow/java/tree/master/tensorflow-core)
-  * All artifacts that build up the core language bindings of TensorFlow for Java
-  * Intended audience: projects that provide their own APIs or frameworks on top of 
-    TensorFlow and just want a thin layer to access the TensorFlow runtime from the JVM 
-
-* [tensorflow-framework](https://github.com/tensorflow/java/tree/master/tensorflow-framework)
-  * Primary API for building and training neural networks with TensorFlow
-  * Intended audience: neural network developers
+### [tensorflow-core](https://github.com/tensorflow/java/tree/master/tensorflow-core)
+  * **Intended audience**: developers who wants to deploy a TensorFlow model on a JVM for inference. Also for projects
+    that provide their own APIs or frameworks on top of TensorFlow and just want a thin layer to access the TensorFlow runtime from the JVM.
+  * All artifacts that make up the core language bindings of TensorFlow for Java.
 
-* [ndarray](https://github.com/tensorflow/java-ndarray)
-  * Generic utility library for n-dimensional data I/O operations
-  * Used by TensorFlow but does not depend on TensorFlow
-  * Intended audience: any developer who needs a Java n-dimensional array implementation, whether or not they
-    use it with TensorFlow
+### [tensorflow-framework](https://github.com/tensorflow/java/tree/master/tensorflow-framework)
+  * **Intended audience**: neural network developers.
+  * Primary API for building and training neural networks with TensorFlow.
 
+### [ndarray](https://github.com/tensorflow/java-ndarray)
+  * **Intended audience**: any developer who needs a Java n-dimensional array implementation, whether or not they use it with TensorFlow.
+  * Generic utility library for n-dimensional data I/O operations.
+  * Used by TensorFlow but does not depend on TensorFlow.
 
 ## Communication
 

docs/docs/assets/tensorflow.svg

@@ -25,8 +25,7 @@ TensorFlow Java 1.0 series and earlier releases also have binaries for:
 
 *   macOS 12 or higher; 64-bit, x86
 
-*Note: To use TensorFlow on Android, see
-[TensorFlow Lite](https://tensorflow.org/lite)*
+*Note: To use TensorFlow on Android, see [LiteRT](https://tensorflow.org/lite)*
 
 ## Versions
 

docs/index.md docs/docs/index.mddocs/index.md renamed to docs/docs/index.md

@@ -0,0 +1,8 @@
+---
+hide:
+  - navigation
+  - toc
+  - title
+---
+#
+<iframe src="https://tensorflow.github.io/java/javadoc/v1.2.0/index.html" style="width:100%; height:800px; border:none;"></iframe>

docs/install.md docs/docs/install.mddocs/install.md renamed to docs/docs/install.md

@@ -0,0 +1,14 @@
+:root  > * {
+    /*--md-primary-fg-color:        #EE782F;*/
+    /*--md-primary-fg-color--light: #455960;*/
+    /*--md-primary-fg-color--dark:  #90030C;*/
+}
+
+.md-typeset h1, .md-typeset h2 {
+    font-weight: 800;
+    letter-spacing: -.01em;
+}
+
+.md-sidebar--primary {
+    display: none;
+}

docs/docs/references.md

@@ -13,6 +13,13 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 # ==============================================================================
+
+######################################################################################################################
+# IMPORTANT: Files in legacy_tools are no longer used to generate the TensorFlow-Java API docs as there are unfixed issues
+# when using DocLava outside of the Google environment. We are keeping these for reference in case they are useful later.
+######################################################################################################################
+
+
 """Generate TensorFlow Java reference docs for TensorFlow.org."""
 from __future__ import absolute_import
 from __future__ import division
@@ -21,6 +28,9 @@
 import pathlib
 import shutil
 import tempfile
+import io
+import requests
+import zipfile
 from git import Repo
 
 from absl import app
@@ -29,14 +39,29 @@
 from tensorflow_docs.api_generator import gen_java
 
 FLAGS = flags.FLAGS
+
 NDARRAY_VERSION = 'v1.0.0'
+JAVACPP_VERSION = '1.5.11'
+PROTOBUF_VERSION = 'v3.21.9'
+
+# __file__ is the path to this file
+TOOLS_DIR = pathlib.Path(__file__).resolve().parent
+DOCS_DIR = TOOLS_DIR.parent
+REPO_ROOT = DOCS_DIR.parent
+DOC_OUTPUT_DIR = DOCS_DIR.joinpath("output")
+
+SECTION_LABELS = {
+  'org.tensorflow': 'Core',
+  'org.tensorflow.ndarray': 'NdArray',
+  'org.tensorflow.framework': 'Framework',
+}
 
 # These flags are required by infrastructure, not all of them are used.
-flags.DEFINE_string('output_dir', '/tmp/java_api/',
+flags.DEFINE_string('output_dir', f"{DOC_OUTPUT_DIR}",
                     ("Use this branch as the root version and don't"
                      ' create in version directory'))
 
-flags.DEFINE_string('site_path', 'java/api_docs/java',
+flags.DEFINE_string('site_path', 'api_docs/',
                     'Path prefix in the _toc.yaml')
 
 flags.DEFINE_string('code_url_prefix', None,
@@ -46,20 +71,15 @@
     'search_hints', True,
     '[UNUSED] Include metadata search hints in the generated files')
 
-# __file__ is the path to this file
-TOOLS_DIR = pathlib.Path(__file__).resolve().parent
-REPO_ROOT = TOOLS_DIR.parent
-
 
-def checkout_ndarray():
-  repo_url = 'https://github.com/tensorflow/java-ndarray'
-  local_repo_path = REPO_ROOT/'ndarray'
+def checkout_repo(repo_url: str, target_dir_name: str, version: str):
+  local_repo_path = f"{REPO_ROOT}/{target_dir_name}"
   if not pathlib.Path(local_repo_path).exists():
     local_repo = Repo.clone_from(repo_url, local_repo_path)
   else:
     local_repo = Repo(local_repo_path)
   local_repo.remotes['origin'].fetch()
-  local_repo.git.checkout(NDARRAY_VERSION)
+  local_repo.git.checkout(version)
 
 
 def overlay(from_root, to_root):
@@ -74,25 +94,36 @@ def overlay(from_root, to_root):
 
 
 def main(unused_argv):
-  checkout_ndarray()
+  checkout_repo('https://github.com/tensorflow/java-ndarray', 'ndarray', NDARRAY_VERSION)
+  checkout_repo('https://github.com/bytedeco/javacpp', 'javacpp', JAVACPP_VERSION)
+  response = requests.get('https://repo1.maven.org/maven2/com/google/protobuf/protobuf-java/3.21.9/protobuf-java-3.21.9-sources.jar')
+  with zipfile.ZipFile(io.BytesIO(response.content)) as z:
+    z.extractall(f"{REPO_ROOT}/protobuf")
+  response = requests.get('https://repo1.maven.org/maven2/org/osgi/osgi.annotation/8.1.0/osgi.annotation-8.1.0-sources.jar')
+  with zipfile.ZipFile(io.BytesIO(response.content)) as z:
+    z.extractall(f"{REPO_ROOT}/osgi")
+
   merged_source = pathlib.Path(tempfile.mkdtemp())
   (merged_source / 'java/org').mkdir(parents=True)
-
   shutil.copytree(REPO_ROOT/'tensorflow-core/tensorflow-core-api/src/main/java/org/tensorflow/', merged_source/'java/org/tensorflow')
   overlay(REPO_ROOT/'tensorflow-core/tensorflow-core-api/src/gen/java/org/tensorflow', merged_source/'java/org/tensorflow')
-  shutil.copytree(REPO_ROOT/'tensorflow-core/tensorflow-core-native/src/gen/java/org/tensorflow/proto', merged_source/'java/org/tensorflow/proto')
-  shutil.copytree(REPO_ROOT/'tensorflow-core/tensorflow-core-native/src/main/java/org/tensorflow/exceptions', merged_source/'java/org/tensorflow/exceptions')
-  shutil.copytree(REPO_ROOT/'tensorflow-core/tensorflow-core-native/src/gen/java/org/tensorflow/internal/c_api', merged_source/'java/org/tensorflow/internal/c_api')
+  overlay(REPO_ROOT/'tensorflow-core/tensorflow-core-api/src/gen/annotations/org/tensorflow', merged_source/'java/org/tensorflow')
+  overlay(REPO_ROOT/'tensorflow-core/tensorflow-core-native/src/gen/java/org/tensorflow/', merged_source/'java/org/tensorflow/')
+  overlay(REPO_ROOT/'tensorflow-core/tensorflow-core-native/src/main/java/org/tensorflow/', merged_source/'java/org/tensorflow/')
   shutil.copytree(REPO_ROOT/'tensorflow-framework/src/main/java/org/tensorflow/framework', merged_source/'java/org/tensorflow/framework')
   shutil.copytree(REPO_ROOT/'ndarray/ndarray/src/main/java/org/tensorflow/ndarray', merged_source/'java/org/tensorflow/ndarray')
+  shutil.copytree(REPO_ROOT/'javacpp/src/main/java/org/bytedeco', merged_source/'java/org/bytedeco')
+  shutil.copytree(REPO_ROOT/'protobuf/com/', merged_source/'java/com')
+  shutil.copytree(REPO_ROOT/'osgi/org/osgi', merged_source/'java/org/osgi')
 
   gen_java.gen_java_docs(
       package='org.tensorflow',
       source_path=merged_source / 'java',
       output_dir=pathlib.Path(FLAGS.output_dir),
       site_path=pathlib.Path(FLAGS.site_path),
+      section_labels=SECTION_LABELS,
       # Uncomment for local testing:
-      # script_path=pathlib.Path(REPO_ROOT/'tools/run-javadoc-for-tf-local.sh'),
+      script_path=pathlib.Path(TOOLS_DIR, 'run-javadoc-for-tf-local.sh'),
   )
 
 

docs/docs/stylesheets/extra.css

@@ -0,0 +1,8 @@
+######################################################################################################################
+# IMPORTANT: Files in legacy_tools are no longer used to generate the TensorFlow-Java API docs as there are unfixed issues
+# when using DocLava outside of the Google environment. We are keeping these for reference in case they are useful later.
+######################################################################################################################
+
+GitPython
+requests
+tensorflow-docs

tools/build_java_api_docs.py docs/legacy_tools/build_java_api_docs.pytools/build_java_api_docs.py renamed to docs/legacy_tools/build_java_api_docs.py

@@ -1,4 +1,10 @@
 #!/bin/bash
+
+######################################################################################################################
+# IMPORTANT: Files in legacy_tools are no longer used to generate the TensorFlow-Java API docs as there are unfixed issues
+# when using DocLava outside of the Google environment. We are keeping these for reference in case they are useful later.
+######################################################################################################################
+
 set -ex
 
 export JAVA_HOME=/Library/Java/JavaVirtualMachines/zulu-11.jdk/Contents/Home # Or change to any JDK 11 home path
@@ -11,12 +17,13 @@ export JAVA_HOME=/Library/Java/JavaVirtualMachines/zulu-11.jdk/Contents/Home # O
 #   $ sudo apt install doclava-aosp #v 6.0.1+r55-1+build1
 #
 # https://unix.stackexchange.com/questions/594841/how-do-i-assign-a-value-to-a-bash-variable-iff-that-variable-is-null-unassigned
-DOCLAVA_JAR=${DOCLAVA_JAR:-'lib/doclava.jar'} # Build lib locally
+DOCLAVA_JAR=${DOCLAVA_JAR:-'tools/lib/doclava.jar'} # Build lib locally
+
 
 # Install java clear silver templates with:
 #
 #   $ sudo apt install libjsilver-aosp-java #v 6.0.1+r55-1+build1
-JSILVER_JAR=${JSILVER_JAR:-'lib/jsilver.jar'} # Build lib locally
+JSILVER_JAR=${JSILVER_JAR:-'tools/lib/jsilver.jar'} # Build lib locally
 
 
 ######### DELETE OUTPUT_DIR #################
@@ -56,16 +63,15 @@ for pkg in "${packages[@]}"; do
   SUBPACKAGES+=" -subpackages ${pkg}"
 done
 ( # Capture the return code. it may be non-zero for minor errors.
-  javadoc \
+  /Library/Java/JavaVirtualMachines/zulu-11.jdk/Contents/Home/bin/javadoc \
   -sourcepath "${SOURCE_PATH}" \
   -docletpath "${DOCLAVA_JAR}:${JSILVER_JAR}" \
   -doclet com.google.doclava.Doclava \
-  -d "${OUTPUT_DIR}" \
   -toroot "${SITE_PATH}"/ \
   -yaml _toc.yaml \
   -templatedir "${TEMPLATES}" \
   -public \
-  -devsite \
+  -d "${OUTPUT_DIR}" \
   ${FEDERATED_PARAMS} \
   ${SUBPACKAGES}
 )