Melhorias no carregamento de Issues para completar corretamente seções dos artigos #1260

robertatakenaka · 2026-01-25T10:21:16Z

Resumo

Este PR implementa melhorias significativas no processo de carregamento de Issues a partir do ArticleMeta, adiciona uma nova task para atualização em lote, e corrige problemas de robustez nos modelos.

Mudanças

Core

BaseLegacyRecord.get: Melhora tratamento de registros duplicados, removendo órfãos e retornando o mais recente em caso de MultipleObjectsReturned

Issue

Models

Issue.short_identification: Gera issue_folder automaticamente se ausente
Issue.get: Trata duplicatas e auto-preenche campos ausentes (issue_folder, issue_pid_suffix, order)
TableOfContents: Adiciona método get_items_by_title para busca padronizada por título

ArticleMeta Loader

Refatora complete_am_issue e get_issue_data_from_am_issue com melhor tratamento de erros e detalhes para debug
Extrai lógica de carregamento para funções dedicadas e reutilizáveis:
- _extract_field: extração genérica com prioridade de fontes
- load_issue_sections: carrega sections com tratamento de erros
- load_issue_titles: carrega títulos com tratamento de erros
- load_bibliographic_strips: carrega strips com tratamento de erros

Issue Utils

Corrige tratamento de valores None em sections_data e bibliographic_strip

Tasks

Nova task task_update_issues_from_amissue: Atualização em lote de Issues com suporte a múltiplos filtros (collection, journal, status, datas, volume, número, etc.)

Article

Sources (xmlsps)

Melhora get_or_create_toc_sections para carregar sections do AMIssue quando necessário
Usa TableOfContents.get_items_by_title para busca padronizada

Dependencies

Limita django-otp < 1.2.0 para compatibilidade com wagtail_2fa.middleware.VerifyUserPermissionsMiddleware

Commits

Limita versão do django-otp para compatibilidade com wagtail_2fa
Melhora tratamento de registros duplicados em BaseLegacyRecord.get
Corrige tratamento de valores None em extract_data_from_harvested_data
Melhora robustez dos modelos Issue e TableOfContents
Refatora loader de issues do ArticleMeta
Adiciona task para atualização em lote de Issues a partir de AMIssue
Melhora busca de seções TOC em get_or_create_toc_sections

Testes

Testar carregamento de issues via ArticleMeta
Testar task_update_issues_from_amissue com diferentes filtros
Verificar busca de seções TOC em artigos

- Restringe django-otp < 1.2.0 - Resolve problema com VerifyUserPermissionsMiddleware

- Remove registros órfãos (sem url e sem data) antes da consulta - Trata MultipleObjectsReturned retornando o registro mais recente

- Adiciona fallback para lista vazia em sections_data - Adiciona fallback para lista vazia em bibliographic_strip

- Restringe django-otp < 1.2.0 - Resolve problema com VerifyUserPermissionsMiddleware

complete_am_issue: - Adiciona validações e detalhes para debug get_issue_data_from_am_issue: - Melhora tratamento de am_issue.data ausente - Extrai journal_pid do pid ao invés de am_data - Adiciona detalhes de debug nas exceções create_issue_from_am_issue: - Extrai lógica de carregamento para funções dedicadas Novas funções: - _extract_field: extração genérica com prioridade de fontes - load_issue_sections: carrega sections com tratamento de erros - load_issue_titles: carrega títulos com tratamento de erros - load_bibliographic_strips: carrega strips com tratamento de erros

task_update_issues_from_amissue: - Suporta filtros por collection, journal_pid, status, datas - Suporta filtros por year, volume, number, supplement - Opção force_update para atualizar Issues existentes - Opção only_without_new_record para processar apenas pendentes - Remove AMIssue órfãos antes do processamento - Retorna estatísticas detalhadas de processamento - Tratamento individual de erros por registro

- Carrega sections do AMIssue se table_of_contents estiver vazio - Usa TableOfContents.get_items_by_title para busca padronizada - Corrige import de load_issue_sections para issue_utils - Move verificação de exists() antes do loop de append

Copilot

Pull request overview

This PR improves the robustness of Issue loading and synchronization from ArticleMeta, adds a batch task to update Issues from AMIssue, and tightens handling of duplicate/legacy records.

Changes:

Adds stricter legacy record retrieval in BaseLegacyRecord.get and Issue.get, including duplicate resolution and auto-filling of computed fields (issue_folder, issue_pid_suffix, order).
Refactors the ArticleMeta Issue loader into reusable helpers (get_issue_data_from_am_issue, load_issue_sections, load_issue_titles, load_bibliographic_strips) and introduces task_update_issues_from_amissue for bulk updates.
Enhances TOC section lookup and ArticleMeta-backed section loading in XML SPS article import, and adds a compatibility pin for django-otp to work with wagtail_2fa.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
requirements/base.txt	Adds `django-otp<1.2.0` constraint to maintain compatibility with `wagtail_2fa`.
core/models.py	Updates `BaseLegacyRecord.get` to clean orphan records and resolve duplicates by returning the most recently updated record.
issue/models.py	Ensures `Issue.short_identification` and `Issue.get` auto-generate missing computed fields and adds `TableOfContents.get_items_by_title` for consistent TOC lookups.
issue/articlemeta/issue_utils.py	Makes `extract_data_from_harvested_data` robust to missing `sections_data` / `bibliographic_strip` by defaulting to empty collections.
issue/articlemeta/loader.py	Strengthens `complete_am_issue` and `get_issue_data_from_am_issue` error reporting and factors out reusable `load_issue_*` helpers for sections, titles, and bibliographic strips.
issue/tasks.py	Introduces `task_update_issues_from_amissue` to bulk-create/update `Issue` records from `AMIssue` with rich filtering and per-item error tracking.
article/sources/xmlsps.py	Integrates AMIssue-backed TOC sections into `get_or_create_toc_sections` and switches to `TableOfContents.get_items_by_title` for normalized TOC section resolution.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-25T10:30:12Z

article/sources/xmlsps.py

+from issue.models import Issue, TableOfContents, AMIssue
+from issue.articlemeta.loader import load_issue_sections


load_issue_sections is imported from issue.articlemeta.issue_utils, but the function is actually defined in issue.articlemeta.loader; as written this import will raise ImportError at module import time and break any code path that loads article.sources.xmlsps.

To fix this, adjust the import to pull load_issue_sections from the correct module (issue.articlemeta.loader).

article/sources/xmlsps.py

Copilot · 2026-01-25T10:30:13Z

issue/models.py

+        if issue.table_of_contents is None:
+            raise cls.DoesNotExist("TableOfContents.get_items_by_title requires valid issue")


if issue.table_of_contents is None: will never trigger because table_of_contents is a reverse related manager on Issue and is never None, even when there are no TOC records; this makes the branch (and its DoesNotExist message about a "valid issue") effectively dead code and may confuse future maintainers.

If you want to guard on the presence of TOC entries, consider checking for emptiness (e.g. if not issue.table_of_contents.exists():) or simply dropping this check and relying on the query/filter itself.

Suggested change

if issue.table_of_contents is None:

raise cls.DoesNotExist("TableOfContents.get_items_by_title requires valid issue")

issue/models.py

Copilot · 2026-01-25T10:30:13Z

core/models.py

+        except Exception:
+            pass


'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

except Exception:

pass

except Exception as exc:

logging.getLogger(__name__).warning(

"Failed to delete legacy records with null url and data in %s.get: %s",

cls.__name__,

exc,

)

Copilot · 2026-01-25T10:30:14Z

issue/tasks.py

+            # Limpar AMIssue órfãos (sem data e sem URL)
+            AMIssue.objects.filter(data__isnull=True, url__isnull=True).delete()
+        except Exception:
+            pass


'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

pass

logging.exception("Failed to delete orphan AMIssue records (data__isnull=True, url__isnull=True)")

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

robertatakenaka added 6 commits January 25, 2026 07:02

Limita versão do django-otp para compatibilidade com wagtail_2fa

b39b81d

- Restringe django-otp < 1.2.0 - Resolve problema com VerifyUserPermissionsMiddleware

Melhora tratamento de registros duplicados em BaseLegacyRecord.get

72af05e

- Remove registros órfãos (sem url e sem data) antes da consulta - Trata MultipleObjectsReturned retornando o registro mais recente

Corrige tratamento de valores None em extract_data_from_harvested_data

bdfd07f

- Adiciona fallback para lista vazia em sections_data - Adiciona fallback para lista vazia em bibliographic_strip

Limita versão do django-otp para compatibilidade com wagtail_2fa

45d604c

- Restringe django-otp < 1.2.0 - Resolve problema com VerifyUserPermissionsMiddleware

Copilot AI review requested due to automatic review settings January 25, 2026 10:21

Copilot started reviewing on behalf of robertatakenaka January 25, 2026 10:21 View session

robertatakenaka force-pushed the fix_article_sections branch from 5bca402 to 32647fa Compare January 25, 2026 10:23

Copilot AI reviewed Jan 25, 2026

View reviewed changes

robertatakenaka and others added 2 commits January 25, 2026 08:26

Update article/sources/xmlsps.py

7530e5c

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update issue/models.py

ccbbc35

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

robertatakenaka merged commit 4a48316 into scieloorg:main Jan 25, 2026
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Melhorias no carregamento de Issues para completar corretamente seções dos artigos #1260

Melhorias no carregamento de Issues para completar corretamente seções dos artigos #1260

Uh oh!

robertatakenaka commented Jan 25, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jan 25, 2026

Uh oh!

Uh oh!

Copilot AI Jan 25, 2026

Uh oh!

Uh oh!

Copilot AI Jan 25, 2026

Uh oh!

Copilot AI Jan 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		from issue.models import Issue, TableOfContents, AMIssue
		from issue.articlemeta.loader import load_issue_sections

		if issue.table_of_contents is None:
		raise cls.DoesNotExist("TableOfContents.get_items_by_title requires valid issue")

-        except Exception:
-            pass
+        except Exception as exc:
+            logging.getLogger(__name__).warning(
+                "Failed to delete legacy records with null url and data in %s.get: %s",
+                cls.__name__,
+                exc,
+            )

	pass
	logging.exception("Failed to delete orphan AMIssue records (data__isnull=True, url__isnull=True)")

Melhorias no carregamento de Issues para completar corretamente seções dos artigos #1260

Melhorias no carregamento de Issues para completar corretamente seções dos artigos #1260

Uh oh!

Conversation

robertatakenaka commented Jan 25, 2026

Resumo

Mudanças

Core

Issue

Models

ArticleMeta Loader

Issue Utils

Tasks

Article

Sources (xmlsps)

Dependencies

Commits

Testes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Jan 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Jan 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant