Skip to content
Toggle navigation
P
Projects
G
Groups
S
Snippets
Help
赵月辉
/
fastgpt-migrated
This project
Loading...
Sign in
Toggle navigation
Go to a project
Project
Repository
Issues
0
Merge Requests
0
Pipelines
Wiki
Snippets
Members
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Unverified
Commit
5ad8c81e
authored
Sep 26, 2024
by
Zhenyi-Wang
Committed by
GitHub
Sep 26, 2024
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
fix: 网页抓取:正确处理//开头的超链接 (#2803)
Co-authored-by: zhenyiwang <zhenyiwang@intl.zju.edu.cn>
parent
cb6fe9d0
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
13 additions
and
4 deletions
+13
-4
packages/service/common/string/cheerio.ts
+13
-4
No files found.
packages/service/common/string/cheerio.ts
View file @
5ad8c81e
...
@@ -14,6 +14,7 @@ export const cheerioToHtml = ({
...
@@ -14,6 +14,7 @@ export const cheerioToHtml = ({
})
=>
{
})
=>
{
// get origin url
// get origin url
const
originUrl
=
new
URL
(
fetchUrl
).
origin
;
const
originUrl
=
new
URL
(
fetchUrl
).
origin
;
const
protocol
=
new
URL
(
fetchUrl
).
protocol
;
// http: or https:
const
usedSelector
=
selector
||
'body'
;
const
usedSelector
=
selector
||
'body'
;
const
selectDom
=
$
(
usedSelector
);
const
selectDom
=
$
(
usedSelector
);
...
@@ -32,14 +33,22 @@ export const cheerioToHtml = ({
...
@@ -32,14 +33,22 @@ export const cheerioToHtml = ({
// if link,img startWith /, add origin url
// if link,img startWith /, add origin url
selectDom
.
find
(
'a'
).
each
((
i
,
el
)
=>
{
selectDom
.
find
(
'a'
).
each
((
i
,
el
)
=>
{
const
href
=
$
(
el
).
attr
(
'href'
);
const
href
=
$
(
el
).
attr
(
'href'
);
if
(
href
&&
href
.
startsWith
(
'/'
))
{
if
(
href
)
{
$
(
el
).
attr
(
'href'
,
originUrl
+
href
);
if
(
href
.
startsWith
(
'//'
))
{
$
(
el
).
attr
(
'href'
,
protocol
+
href
);
}
else
if
(
href
.
startsWith
(
'/'
))
{
$
(
el
).
attr
(
'href'
,
originUrl
+
href
);
}
}
}
});
});
selectDom
.
find
(
'img'
).
each
((
i
,
el
)
=>
{
selectDom
.
find
(
'img'
).
each
((
i
,
el
)
=>
{
const
src
=
$
(
el
).
attr
(
'src'
);
const
src
=
$
(
el
).
attr
(
'src'
);
if
(
src
&&
src
.
startsWith
(
'/'
))
{
if
(
src
)
{
$
(
el
).
attr
(
'src'
,
originUrl
+
src
);
if
(
src
.
startsWith
(
'//'
))
{
$
(
el
).
attr
(
'src'
,
protocol
+
src
);
}
else
if
(
src
.
startsWith
(
'/'
))
{
$
(
el
).
attr
(
'src'
,
originUrl
+
src
);
}
}
}
});
});
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment