Managing large data files, vector and raster, in QGIS 3How to repair and clean contours from high data resolution (lidar)?Per-pixel (statistical) calculations on a raster stack using GDALPoor performance with storing large rasters in PostGIS and visualising in QGISExport vector layer to raster in QGISPacking tiled rasters into a Geopackage in QGIS 3.0.3 whilst maintaining the colourWhat does the vector to raster option in QGIS do, exactly?
Will Curiosity and the Mars 2020 rover be able to communicate with each other via a Mars orbiter?
Designing Borders with QGIS
I need to draw this picture in LaTeX. How do I do that? Boxes with arrows
Who are the strongest non-professional GMs?
What antenna is this in an Apollo 15 LM photo?
Hough transform algorithm - Idiomatic c++
Can a Barbarian/Wizard multiclass trigger a magic item while raging?
Difference between apply and funcall
Wifi near the speed of light
Large products with glass doors
Strange labels on our servers
What is the contemporary meaning of primary storage?
Patent Agreement in Order to Graduate
Is 'lay lady lay' grammatical?
Could dinosaurs breathe modern air?
Were there ever 12-, 24-, 48-, etc bit processors?
What are these color strips for?
Is military allowed to wear civilian clothes when testifying in Congress?
Lost Time at Motel?
How to save current font so that it can be used again later?
Rats biting off fuel line ( again and again and again )!
Did any 360-compatible machine implement registers in core?
How to handle a colleague who appears helpful in front of manager but doesn't help in private?
Is it possible to protect saved games on Nintendo Switch from being played by other users?
Managing large data files, vector and raster, in QGIS 3
How to repair and clean contours from high data resolution (lidar)?Per-pixel (statistical) calculations on a raster stack using GDALPoor performance with storing large rasters in PostGIS and visualising in QGISExport vector layer to raster in QGISPacking tiled rasters into a Geopackage in QGIS 3.0.3 whilst maintaining the colourWhat does the vector to raster option in QGIS do, exactly?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty
margin-bottom:0;
I work for a local authority so use data covering a large area. Therefore some datasets I use become large and cumbersome.
For example I have a vector layer for contours and a raster layer for hillshade. As I am relatively inexperienced I created Booth of these of a series of tiled points and created a single file. I am happy with the results but performance is slower.
What are the main options available to improve the performance of the vector layers for a single user on one PC and for a team? This should allow me to both display and analyse the data efficiently.
What are the main options available to improve the performance of the raster layer?
qgis data-management
add a comment
|
I work for a local authority so use data covering a large area. Therefore some datasets I use become large and cumbersome.
For example I have a vector layer for contours and a raster layer for hillshade. As I am relatively inexperienced I created Booth of these of a series of tiled points and created a single file. I am happy with the results but performance is slower.
What are the main options available to improve the performance of the vector layers for a single user on one PC and for a team? This should allow me to both display and analyse the data efficiently.
What are the main options available to improve the performance of the raster layer?
qgis data-management
Do you have access to a server, so you could host your own WMS? Could you reduce the resolution of the DEM/countour layer? What do you mean by "I created both of these of a series of tiled points and created a single file"? Aren't they two files? Why do you actually need hillshade? Would it be feasible to turn on these layers only when you really need them?
– Erik
Sep 12 at 9:32
To improve performance of a raster layer in qgis, convert it to a single GeoTiff and add external overviews (fastest) or a virtual raster with external overviews (generally fast). You can create a virtual raster then save it as a GeoTiff. Overviews are created in the properties view (right click the layer).
– vinh
Sep 12 at 15:52
You could try creating a geopackage. This uses a sqlite database and can include optimisations for both vector (spatial index) and raster (overviews and internal tiles).
– vinh
Sep 12 at 15:54
In addition to the suggestions to build a spatial index for vectors and pyramids for rasters , you might check out Settings - Options - Rendering for some settings that affect rendering speed. You can generalize/simplify vectors like contours (those from LiDAR are often quite noisy) as well. If you have rasters with complex symbology, especially 32 bit floating point, converting them to rendered 8 bit can greatly improve speed. If you have complex labeling expressions, masking or locations, or complex symbology expressions or symbols, those can slow drawing.
– johns
Sep 12 at 17:12
add a comment
|
I work for a local authority so use data covering a large area. Therefore some datasets I use become large and cumbersome.
For example I have a vector layer for contours and a raster layer for hillshade. As I am relatively inexperienced I created Booth of these of a series of tiled points and created a single file. I am happy with the results but performance is slower.
What are the main options available to improve the performance of the vector layers for a single user on one PC and for a team? This should allow me to both display and analyse the data efficiently.
What are the main options available to improve the performance of the raster layer?
qgis data-management
I work for a local authority so use data covering a large area. Therefore some datasets I use become large and cumbersome.
For example I have a vector layer for contours and a raster layer for hillshade. As I am relatively inexperienced I created Booth of these of a series of tiled points and created a single file. I am happy with the results but performance is slower.
What are the main options available to improve the performance of the vector layers for a single user on one PC and for a team? This should allow me to both display and analyse the data efficiently.
What are the main options available to improve the performance of the raster layer?
qgis data-management
qgis data-management
edited Sep 12 at 20:49
csk
17.1k1 gold badge15 silver badges42 bronze badges
17.1k1 gold badge15 silver badges42 bronze badges
asked Sep 11 at 20:31
boberdorfboberdorf
8919 silver badges14 bronze badges
8919 silver badges14 bronze badges
Do you have access to a server, so you could host your own WMS? Could you reduce the resolution of the DEM/countour layer? What do you mean by "I created both of these of a series of tiled points and created a single file"? Aren't they two files? Why do you actually need hillshade? Would it be feasible to turn on these layers only when you really need them?
– Erik
Sep 12 at 9:32
To improve performance of a raster layer in qgis, convert it to a single GeoTiff and add external overviews (fastest) or a virtual raster with external overviews (generally fast). You can create a virtual raster then save it as a GeoTiff. Overviews are created in the properties view (right click the layer).
– vinh
Sep 12 at 15:52
You could try creating a geopackage. This uses a sqlite database and can include optimisations for both vector (spatial index) and raster (overviews and internal tiles).
– vinh
Sep 12 at 15:54
In addition to the suggestions to build a spatial index for vectors and pyramids for rasters , you might check out Settings - Options - Rendering for some settings that affect rendering speed. You can generalize/simplify vectors like contours (those from LiDAR are often quite noisy) as well. If you have rasters with complex symbology, especially 32 bit floating point, converting them to rendered 8 bit can greatly improve speed. If you have complex labeling expressions, masking or locations, or complex symbology expressions or symbols, those can slow drawing.
– johns
Sep 12 at 17:12
add a comment
|
Do you have access to a server, so you could host your own WMS? Could you reduce the resolution of the DEM/countour layer? What do you mean by "I created both of these of a series of tiled points and created a single file"? Aren't they two files? Why do you actually need hillshade? Would it be feasible to turn on these layers only when you really need them?
– Erik
Sep 12 at 9:32
To improve performance of a raster layer in qgis, convert it to a single GeoTiff and add external overviews (fastest) or a virtual raster with external overviews (generally fast). You can create a virtual raster then save it as a GeoTiff. Overviews are created in the properties view (right click the layer).
– vinh
Sep 12 at 15:52
You could try creating a geopackage. This uses a sqlite database and can include optimisations for both vector (spatial index) and raster (overviews and internal tiles).
– vinh
Sep 12 at 15:54
In addition to the suggestions to build a spatial index for vectors and pyramids for rasters , you might check out Settings - Options - Rendering for some settings that affect rendering speed. You can generalize/simplify vectors like contours (those from LiDAR are often quite noisy) as well. If you have rasters with complex symbology, especially 32 bit floating point, converting them to rendered 8 bit can greatly improve speed. If you have complex labeling expressions, masking or locations, or complex symbology expressions or symbols, those can slow drawing.
– johns
Sep 12 at 17:12
Do you have access to a server, so you could host your own WMS? Could you reduce the resolution of the DEM/countour layer? What do you mean by "I created both of these of a series of tiled points and created a single file"? Aren't they two files? Why do you actually need hillshade? Would it be feasible to turn on these layers only when you really need them?
– Erik
Sep 12 at 9:32
Do you have access to a server, so you could host your own WMS? Could you reduce the resolution of the DEM/countour layer? What do you mean by "I created both of these of a series of tiled points and created a single file"? Aren't they two files? Why do you actually need hillshade? Would it be feasible to turn on these layers only when you really need them?
– Erik
Sep 12 at 9:32
To improve performance of a raster layer in qgis, convert it to a single GeoTiff and add external overviews (fastest) or a virtual raster with external overviews (generally fast). You can create a virtual raster then save it as a GeoTiff. Overviews are created in the properties view (right click the layer).
– vinh
Sep 12 at 15:52
To improve performance of a raster layer in qgis, convert it to a single GeoTiff and add external overviews (fastest) or a virtual raster with external overviews (generally fast). You can create a virtual raster then save it as a GeoTiff. Overviews are created in the properties view (right click the layer).
– vinh
Sep 12 at 15:52
You could try creating a geopackage. This uses a sqlite database and can include optimisations for both vector (spatial index) and raster (overviews and internal tiles).
– vinh
Sep 12 at 15:54
You could try creating a geopackage. This uses a sqlite database and can include optimisations for both vector (spatial index) and raster (overviews and internal tiles).
– vinh
Sep 12 at 15:54
In addition to the suggestions to build a spatial index for vectors and pyramids for rasters , you might check out Settings - Options - Rendering for some settings that affect rendering speed. You can generalize/simplify vectors like contours (those from LiDAR are often quite noisy) as well. If you have rasters with complex symbology, especially 32 bit floating point, converting them to rendered 8 bit can greatly improve speed. If you have complex labeling expressions, masking or locations, or complex symbology expressions or symbols, those can slow drawing.
– johns
Sep 12 at 17:12
In addition to the suggestions to build a spatial index for vectors and pyramids for rasters , you might check out Settings - Options - Rendering for some settings that affect rendering speed. You can generalize/simplify vectors like contours (those from LiDAR are often quite noisy) as well. If you have rasters with complex symbology, especially 32 bit floating point, converting them to rendered 8 bit can greatly improve speed. If you have complex labeling expressions, masking or locations, or complex symbology expressions or symbols, those can slow drawing.
– johns
Sep 12 at 17:12
add a comment
|
3 Answers
3
active
oldest
votes
For your contours, you can
- use simple scale dependent visibility to turn the layer off when zoomed out too far
- build a spatial index
- you could also try a rule-based symbology to limit the contour interval shown as you zoom out
- you may need to build an attribute index on the elevation field to help speed up any labelling and rule-based symbology
For your raster, you can
- build raster overviews (pyramids)
add a comment
|
The best way to manage your large datasets is to store them in a spatial database like PostGIS. The loading speed will be twice faster.
1
PostGIS will also allow you to do very fast spatial operations.
– ragnvald
Sep 12 at 5:09
Thanks Simon, I don't know much about databases, so will check it out.
– boberdorf
Sep 12 at 10:11
1
@boberdorf This answer is a bit of an over generalisation. Where I work, our very large datasets are stored in a remote spatial database (which is tuned for speed) and it's still way quicker to pull the data down to a local filesystem format to do anything with it. A local database may be quicker, but will have overhead in setting up and tuning.
– user2856
Sep 13 at 0:44
Geoserver is not a spatial database.
– user2856
Sep 14 at 2:49
add a comment
|
Please check out PostGIS
- https://postgis.net/
Also you can access Postgres databases in QGIS
https://www.bostongis.com/blog/index.php?/archives/271-New-in-QGIS-3.2-Save-Project-to-PostgreSQL.html
https://docs.qgis.org/2.8/en/docs/training_manual/databases/db_browser.html
add a comment
|
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "79"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/4.0/"u003ecc by-sa 4.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fgis.stackexchange.com%2fquestions%2f335621%2fmanaging-large-data-files-vector-and-raster-in-qgis-3%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
For your contours, you can
- use simple scale dependent visibility to turn the layer off when zoomed out too far
- build a spatial index
- you could also try a rule-based symbology to limit the contour interval shown as you zoom out
- you may need to build an attribute index on the elevation field to help speed up any labelling and rule-based symbology
For your raster, you can
- build raster overviews (pyramids)
add a comment
|
For your contours, you can
- use simple scale dependent visibility to turn the layer off when zoomed out too far
- build a spatial index
- you could also try a rule-based symbology to limit the contour interval shown as you zoom out
- you may need to build an attribute index on the elevation field to help speed up any labelling and rule-based symbology
For your raster, you can
- build raster overviews (pyramids)
add a comment
|
For your contours, you can
- use simple scale dependent visibility to turn the layer off when zoomed out too far
- build a spatial index
- you could also try a rule-based symbology to limit the contour interval shown as you zoom out
- you may need to build an attribute index on the elevation field to help speed up any labelling and rule-based symbology
For your raster, you can
- build raster overviews (pyramids)
For your contours, you can
- use simple scale dependent visibility to turn the layer off when zoomed out too far
- build a spatial index
- you could also try a rule-based symbology to limit the contour interval shown as you zoom out
- you may need to build an attribute index on the elevation field to help speed up any labelling and rule-based symbology
For your raster, you can
- build raster overviews (pyramids)
answered Sep 11 at 23:32
user2856user2856
35.3k2 gold badges65 silver badges117 bronze badges
35.3k2 gold badges65 silver badges117 bronze badges
add a comment
|
add a comment
|
The best way to manage your large datasets is to store them in a spatial database like PostGIS. The loading speed will be twice faster.
1
PostGIS will also allow you to do very fast spatial operations.
– ragnvald
Sep 12 at 5:09
Thanks Simon, I don't know much about databases, so will check it out.
– boberdorf
Sep 12 at 10:11
1
@boberdorf This answer is a bit of an over generalisation. Where I work, our very large datasets are stored in a remote spatial database (which is tuned for speed) and it's still way quicker to pull the data down to a local filesystem format to do anything with it. A local database may be quicker, but will have overhead in setting up and tuning.
– user2856
Sep 13 at 0:44
Geoserver is not a spatial database.
– user2856
Sep 14 at 2:49
add a comment
|
The best way to manage your large datasets is to store them in a spatial database like PostGIS. The loading speed will be twice faster.
1
PostGIS will also allow you to do very fast spatial operations.
– ragnvald
Sep 12 at 5:09
Thanks Simon, I don't know much about databases, so will check it out.
– boberdorf
Sep 12 at 10:11
1
@boberdorf This answer is a bit of an over generalisation. Where I work, our very large datasets are stored in a remote spatial database (which is tuned for speed) and it's still way quicker to pull the data down to a local filesystem format to do anything with it. A local database may be quicker, but will have overhead in setting up and tuning.
– user2856
Sep 13 at 0:44
Geoserver is not a spatial database.
– user2856
Sep 14 at 2:49
add a comment
|
The best way to manage your large datasets is to store them in a spatial database like PostGIS. The loading speed will be twice faster.
The best way to manage your large datasets is to store them in a spatial database like PostGIS. The loading speed will be twice faster.
edited Sep 16 at 1:58
answered Sep 12 at 4:14
Simon GISSimon GIS
2631 silver badge9 bronze badges
2631 silver badge9 bronze badges
1
PostGIS will also allow you to do very fast spatial operations.
– ragnvald
Sep 12 at 5:09
Thanks Simon, I don't know much about databases, so will check it out.
– boberdorf
Sep 12 at 10:11
1
@boberdorf This answer is a bit of an over generalisation. Where I work, our very large datasets are stored in a remote spatial database (which is tuned for speed) and it's still way quicker to pull the data down to a local filesystem format to do anything with it. A local database may be quicker, but will have overhead in setting up and tuning.
– user2856
Sep 13 at 0:44
Geoserver is not a spatial database.
– user2856
Sep 14 at 2:49
add a comment
|
1
PostGIS will also allow you to do very fast spatial operations.
– ragnvald
Sep 12 at 5:09
Thanks Simon, I don't know much about databases, so will check it out.
– boberdorf
Sep 12 at 10:11
1
@boberdorf This answer is a bit of an over generalisation. Where I work, our very large datasets are stored in a remote spatial database (which is tuned for speed) and it's still way quicker to pull the data down to a local filesystem format to do anything with it. A local database may be quicker, but will have overhead in setting up and tuning.
– user2856
Sep 13 at 0:44
Geoserver is not a spatial database.
– user2856
Sep 14 at 2:49
1
1
PostGIS will also allow you to do very fast spatial operations.
– ragnvald
Sep 12 at 5:09
PostGIS will also allow you to do very fast spatial operations.
– ragnvald
Sep 12 at 5:09
Thanks Simon, I don't know much about databases, so will check it out.
– boberdorf
Sep 12 at 10:11
Thanks Simon, I don't know much about databases, so will check it out.
– boberdorf
Sep 12 at 10:11
1
1
@boberdorf This answer is a bit of an over generalisation. Where I work, our very large datasets are stored in a remote spatial database (which is tuned for speed) and it's still way quicker to pull the data down to a local filesystem format to do anything with it. A local database may be quicker, but will have overhead in setting up and tuning.
– user2856
Sep 13 at 0:44
@boberdorf This answer is a bit of an over generalisation. Where I work, our very large datasets are stored in a remote spatial database (which is tuned for speed) and it's still way quicker to pull the data down to a local filesystem format to do anything with it. A local database may be quicker, but will have overhead in setting up and tuning.
– user2856
Sep 13 at 0:44
Geoserver is not a spatial database.
– user2856
Sep 14 at 2:49
Geoserver is not a spatial database.
– user2856
Sep 14 at 2:49
add a comment
|
Please check out PostGIS
- https://postgis.net/
Also you can access Postgres databases in QGIS
https://www.bostongis.com/blog/index.php?/archives/271-New-in-QGIS-3.2-Save-Project-to-PostgreSQL.html
https://docs.qgis.org/2.8/en/docs/training_manual/databases/db_browser.html
add a comment
|
Please check out PostGIS
- https://postgis.net/
Also you can access Postgres databases in QGIS
https://www.bostongis.com/blog/index.php?/archives/271-New-in-QGIS-3.2-Save-Project-to-PostgreSQL.html
https://docs.qgis.org/2.8/en/docs/training_manual/databases/db_browser.html
add a comment
|
Please check out PostGIS
- https://postgis.net/
Also you can access Postgres databases in QGIS
https://www.bostongis.com/blog/index.php?/archives/271-New-in-QGIS-3.2-Save-Project-to-PostgreSQL.html
https://docs.qgis.org/2.8/en/docs/training_manual/databases/db_browser.html
Please check out PostGIS
- https://postgis.net/
Also you can access Postgres databases in QGIS
https://www.bostongis.com/blog/index.php?/archives/271-New-in-QGIS-3.2-Save-Project-to-PostgreSQL.html
https://docs.qgis.org/2.8/en/docs/training_manual/databases/db_browser.html
answered Sep 12 at 10:41
raajraaj
1297 bronze badges
1297 bronze badges
add a comment
|
add a comment
|
Thanks for contributing an answer to Geographic Information Systems Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fgis.stackexchange.com%2fquestions%2f335621%2fmanaging-large-data-files-vector-and-raster-in-qgis-3%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Do you have access to a server, so you could host your own WMS? Could you reduce the resolution of the DEM/countour layer? What do you mean by "I created both of these of a series of tiled points and created a single file"? Aren't they two files? Why do you actually need hillshade? Would it be feasible to turn on these layers only when you really need them?
– Erik
Sep 12 at 9:32
To improve performance of a raster layer in qgis, convert it to a single GeoTiff and add external overviews (fastest) or a virtual raster with external overviews (generally fast). You can create a virtual raster then save it as a GeoTiff. Overviews are created in the properties view (right click the layer).
– vinh
Sep 12 at 15:52
You could try creating a geopackage. This uses a sqlite database and can include optimisations for both vector (spatial index) and raster (overviews and internal tiles).
– vinh
Sep 12 at 15:54
In addition to the suggestions to build a spatial index for vectors and pyramids for rasters , you might check out Settings - Options - Rendering for some settings that affect rendering speed. You can generalize/simplify vectors like contours (those from LiDAR are often quite noisy) as well. If you have rasters with complex symbology, especially 32 bit floating point, converting them to rendered 8 bit can greatly improve speed. If you have complex labeling expressions, masking or locations, or complex symbology expressions or symbols, those can slow drawing.
– johns
Sep 12 at 17:12